Multicast Events - the finale

In my previous two posts I presented a technique using the new generics language feature of Delphi 2009 to create a typesafe multicast event. In the previous post, I showed how you can create a TMulticastEvent<T> instance and assign it to an event handler for an existing event on a TComponent derived type. Using the existing FreeNotification mechanism, you didn't need to worry about explicitly freeing the multicast event object. What if one of the components in the sink event handlers in the multicast event list was freed? The good thing is that the FreeNotification mechanism works both ways. We can leverage this functionality again to handle cleanup from the other direction.

In order to implement the complete cleanup for the TComponentMulticastEvent<T>, we need to know when an event handler was added and when one was removed. To do this I added these two virtual methods to the base TMulticastEvent class (the base non-generic version). There are also helper functions, RemoveInstanceReferences() and IndexOfInstance() that can be used in descendants to remove all event handlers that refer to a specific object instance and check if a specific instance is being referenced within the list.

  TMulticastEvent = class
    ...
  strict protected
    procedure EventAdded(const AMethod: TMethod); virtual;
    procedure EventRemoved(const AMethod: TMethod); virtual;
  protected
    procedure RemoveInstanceReferences(const Instance: TObject);
    function IndexOfInstance(const Instance: TObject): Integer;
    ...
  end;

They're not marked abstract because the immediate descendant, TMulticastEvent<T> doesn't need to and should not be forced to override them. They just do nothing in the base class. In the corresponding Add and Remove methods on TMulticastEvent, these virtual methods are then called with event just added or just removed. Now we override the EventAdded and EventRemoved methods in the TComponentMulticastEvent<T> class:

  TComponentMulticastEvent<T> = class(TMulticastEvent<T>)
    ...
  private
    FSink: TNotificationSink;
  strict protected
    procedure EventAdded(const AMethod: TMethod); override;
    procedure EventRemoved(const AMethod: TMethod); override;
    ...
  end;

We also need to hold a reference to the internal notification sink class in order to use its FreeNotification mechanism. Here's the implementation of these methods:

procedure TComponentMulticastEvent<T>.EventAdded(const AMethod: TMethod);
begin
  inherited;
  if TObject(AMethod.Data) is TComponent then
    FSink.FreeNotification(TComponent(AMethod.Data));
end;

procedure TComponentMulticastEvent<T>.EventRemoved(const AMethod: TMethod);
begin
  inherited;
  if (TObject(AMethod.Data) is TComponent) and (IndexOfInstance(TObject(AMethod.Data)) < 0) then
    FSink.RemoveFreeNotification(TComponent(AMethod.Data));
end;

And then the Notification on the private TNotificationSink class:

procedure TComponentMulticastEvent<T>.TNotificationSink.Notification(AComponent: TComponent;
  Operation: TOperation);
begin
  inherited;
  if Operation = opRemove then
    if AComponent = FOwnerComp then
      Free
    else
      FEvent.RemoveInstanceReferences(AComponent);
end;

In the EventRemoved method we call IndexOfInstance() to ensure that there aren't multiple references to the same instance in the list before we remove the free notification hook. This is because FreeNotification will add the instance to its internal list only once.

So there you go, a multicast event that also performs full auto-cleanup for both the source and the sink instances. If the source instance goes away, the multicast event instance is automatically cleaned up. Likewise, if one of the sink event handlers' instances go away, it will automatically be removed from the list so there are no stale references. Of course, I'll remind the reader that with this implementation, it only works for TComponent derived instances. For non-TComponent derived instances, you can still use a descendant of TMulticastEvent<T> in a manner similar to TComponentMulticastEvent<T> mixed with, for instance, a technique described here.

Multicast Events - the cleanup

In my last post, I introduced a multicast event that uses generics to address the problem of needing to manually declare and implement a new multicaster for each unique event type. If you remember, I referred to some other posts that presented a technique for doing automatic cleanup of both the multicast event object itself and automatic removal of listeners. While the technique presented is indeed very generic and will work for nearly all instances, my only critique is that it carries a fairly heavy "contract" in order to fully realize its potential. This set all the creaky wheels moving as I tried to come up with something a little less "contract" heavy. This solution has an assumption that most events and event handlers are on TComponent derived classes. Yes, there are plenty of folks out there that use method pointers on object instances of types other than TComponent. I'm using the 80/20 rule. This will be highly useful to 80% of you out there and not as useful for the remaining 20%.

One of the main problems of a "bolt-on" multicast event class in Delphi for Win32 is proper cleanup. With normal single-cast events there is but one sender and one listener. The end-points are already presumed to be lifetime managed. For multicast events, no longer is there a point-to-point connection, but rather an intermediate "agent" that serves to translate a single event into "n" events for "n" listeners. In a nice garbage collected run-time environment, this isn't a problem because this intermediary will be properly cleanup in due time. This is not meant to be a ding against the inherent non-GC Delphi/Win32 environment but merely that we need to be a little more creative. The good thing here is that once we've established this solution, you can return to your "regularly scheduled programming" and no longer worry about this cleanup.

Let's start with a little review. TComponent and its derivatives carry the notion of "ownership" for the exact reason of not being able to rely on a GC. This model was established from the very first introduction of Delphi and the VCL. Any component that has an "owner" will be automatically cleaned up when its owner was cleaned up. In order for this to operate effectively, there was a need for some mechanism to let everyone involved know when an instance was going away. This is the reason for the Notification virtual method on TComponent. Every component that is being cleaned up, or more accurately, being removed from the list of "owned" components is sent broadcast to all the other owned components through this Notification method with the "opRemove" operation enumeration value. Initially, this notification only worked for components that shared a common "owner," which worked well for Delphi 1 and 2. In Delphi 3, form-inheritance and form-linking where introduced which allowed cross-form/datamodule references between components. For instance, the TTable component on a datamodule could now be referenced from a TDataSource component on a form. This presented a problem because the datamodule and form could have very different lifetimes and the previously mentioned notification mechanism just doesn't work. This is when the FreeNotification() method was introduced on TComponent. This allows any component to insert themselves into the "free notification" list of another component in order to "link" them together and know when each other is going away. Now proper cleanup (such as setting references to nil) can be done even for components that do not share the same owner. We can use this knowledge to devise a solution for properly cleaning up a multicast event object.

I created the following generic descendant of TMulticastEvent<T>:

type
  TComponentMulticastEvent<T> = class(TMulticastEvent<T>)
  private type
    TNotificationSink = class(TComponent)
    private
      FEvent: TMulticastEvent;
      FOwnerComp: TComponent;
    protected
      procedure Notification(AComponent: TComponent; Operation: TOperation); override;
    public
      constructor Create(AOwner: TComponent; AEvent: TMulticastEvent); reintroduce;
      destructor Destroy; override;
    end;
  public
    constructor Create(AOwner: TComponent);
  end;

Since the TMulticastEvent ancestor isn't a TComponent derivative I needed a way to "listen in" on the notifications sent between TComponents. This is why there is a private component derived from TComponent which provides the link between the multicast event instance and the component notifications. I could have simply made this private component owned by the component and not even worried about the Notification method because all owned components are properly cleaned up. However, the intent is that this internal component needs to be invisible (relatively speaking) and not show up in the list of owned components. A lot of code out there iterates through the list of owned components and performs some operation on each one. Don't want to risk breaking that code. So, when this private component is instantiated, it isn't "owned" by any component but by using FreeNotification, it will still know when the "owner" component is freed.

In order to make it a little easier to instantiate these multicast events I added the following public class static methods to the base non-generic TMulticastEvent class:

type
  TMulticastEvent = class
    ...
  public
    ...
    class function MulticastEvent<T>(const AMethod: T): TMulticastEvent<T>; static;
    class function Create<T>(AComponent: TComponent): T; overload; static;
    class function Create<T>: T; overload; static;
  end;

The first one we'll come back to in a moment. The two overloaded Create methods will create a TComponentMulticastEvent<T> or simply a TMulticastEvent<T>, respectively. If you'll notice that the multicast event instance is not being returned, but rather a value of the method pointer type itself is returned. This is the Invoke property. You can now simply do this:

procedure TForm1.Form1Create(Sender: TObject);
begin
  Button1.OnClick := TMulticastEvent.Create<TNotifyEvent>(Button1);
end;

Notice how that looks remarkably similar to a normal object instantiation? Cool. Uh... waitaminnit. How do I add listeners? I cannot call Add, or Remove now. One solution would be to first create the event, add the listeners, then assign the Invoke property to the OnClick event like this:

procedure TForm1.Form1Create(Sender: TObject);
var
  Click: TMulticastEvent<TNotifyEvent>;
begin
  Click := TComponentMulticastEvent<TNotifyEvent>.Create(Button1);
  Click.Add(Button1Click);
  Click.Add(Button2Click);
  Button1.OnClick := Click.Invoke;
end;

Yes, that would work, but only if you know up front what the listeners would be and that the list won't ever change once it is setup. The beauty of multicast events is that they're dynamic and can change at run-time. Remember the first class static method listed above? This is where that cute little method comes into play. Let's redo the above code using that method:

procedure TForm1.Form1Create(Sender: TObject);
begin
  Button1.OnClick := TMulticastEvent.Create<TNotifyEvent>(Button1);
  TMulticastEvent.MulticastEvent<TNotifyEvent>(Button1.OnClick).Add(Button1Click);
  TMulticastEvent.MulticastEvent<TNotifyEvent>(Button1.OnClick).Add(Button2Click);
end;

While the above techniques are equivalent in this instance, what if the addition or removal of the listeners occurred elsewhere? Instead of keeping a separate reference to the multicast event instance, it is simply fished out of the method pointer value itself. Also, because we know it too will be cleaned up, we can now simply concentrate on using it.

On a final note, the above implementation doesn't currently support automatic removal of listeners when they go away, but that can certainly be done. We'll look at that in the next installment. Your homework is this; using the class static methods above to in-turn implement class static Include() and Exclude() methods that will mimic the Include/Exclude standard functions in Delphi for .NET.

Multicast events using generics

Ever since before Delphi 1, the (then Delphi only) RAD Studio IDE has been full of home-grown multicast event class types. I usually refer to these as an "event bus." This is from my hardware days when I designed microcontroller based security/access control equipment. A CPU has an "address bus" and a "data bus" which can have one sender and any number of "listeners," not unlike a multicast event.

I've been following with interest these series of posts about creating a multicast event class in native Delphi. What struck me was that the class being presented was remarkably similar to the class and its descendants that we've used in the RAD Studio IDE since before Delphi 1. However the one presented in those posts does take it to another level and adds some extra housekeeping to notify the listener and/or the multicast event when each other is being freed. While that's a nice addition, it does tend to complicate its use. From my point of view, regardless of whether you have a single-cast or multicast event, you should always balance assigning the event with clearing the event. IOW, make sure the event isn't pointing to a dead instance. Overall, what was presented is a pretty decent design given what the Delphi language has to offer today.

One of the key new Delphi language features for Tiburón is the introduction of generics. During the development cycle, I began thinking if there was a way to create a multicast event class using generics. At first glance this seemed like an easy thing to do. After all, a multi cast event class is just a wrapper around an array of method pointers. To send out the event you iterate through the array of method pointers invoking each event in turn. So surely it's as simple as this:

type
  TMulticastEvent<T> = class
  private
    FHandlers: array of T;
  public
    procedure Add(Handler: T);
    procedure Remove(Handler: T);
    procedure Invoke(<uh oh>);
  end;

Hmm... Do you see the problem? How can I actually invoke the event? Adding and removing handlers looks straight forward, but invoking events is a problem. The multicast event class inside the Delphi IDE starts with a base class that handles the array of method pointers and it is up to the manually created descendants to add the Add and Remove methods and the specific Invoke (or Send in our case) method that provides the right parameter list and does the job of iterating through the assigned method pointers. Lots of ugly typecasting was happening in addition to having to manually create the descendants.

Ok, now I have a challenge. The gauntlet has been thrown. I'm going to design a generic multicast event using Delphi native code, dagnabbit! The end result still looked very similar to the internal IDE multicast event classes. The way I solved the invoke problem was to declare a property of type T.

type
  TMulticastEvent<T> = class
  private
    FInvoke: T;
    <same above as>
    property Invoke: T read FInvoke;
  end;

Syntactically you just call Invoke like any old method:

type
  TMyEvent = procedure (Sender: TObject; IntValue: Integer; FloatValue: Double) of object;
  MulticastMyEvent: TMulticastEvent<TMyEvent>
begin
  MulticastMyEvent := TMulticastEvent<TMyEvent>.Create;
  MulticastMyEvent.Invoke(Obj, 10, 3.14);
end.

There is still an interesting problem here. Do you see it? So I can use the event property "simulate" a method, but what is assigned to this method pointer? Where does control go when you call through this method pointer? When is is assigned? This is when things started getting a little interesting.

The problem with all the other multicast event class implementations I've seen (including our own internal one) is that for type-safety they require you to manually create a descendant class with the actual type of the event. It's tedious and error prone, and promotes gross duplication of code. These are the problems that generics are suppose to solve, right? However in this instance, it was almost like I needed to instantiate code at run-time (something that .NET actually does which solves many of these problems). Barring that, I had to reach a little deeper into my bag-o-tricks. All the multicast event classes share a couple of things; maintain a list or array of method pointers and iterate over that list and invoke each method. All method pointers are the same size (8 bytes consisting of a pointer to an object instance and a pointer to the method to invoke). I created a non-generic base class much like the existing base class we've had for years:

type
  TMulticastEvent = class
  strict protected type TEvent = procedure of object;
  strict private
    FHandlers: array of TMethod;
    procedure Add(const AMethod: TEvent); overload;
    prodedure Remove(const AMethod: TEvent); overload;
    function IndexOf(const AMethod: TEvent): Integer; overload;
  end;

Hmmmm... Why is the array declared here instead of the descendant? How will this be "typesafe?" You can't cast an arbitrary "T" to a TMethod. Oh and waidaminnit... the Add, Remove and IndexOf methods are private! To address these issues, I pulled out some assembler tricks out of my proverbial bag and I added the following protected methods.

type
  TMulticastEvent = class
    <stuff from above>
  protected
    procedure InternalAdd;
    procedure InternalRemove;
    procedure InternalIndexOf;
  end;

Uh, dude, there are no parameters on those methods. What do they do? Here's the method body from the InternalAdd method. The others are very similar.

procedure TMulticastEvent.InternalAdd;
asm
       XCHG  EAX,[ESP]
       POP   EAX
       POP   EBP
       JMP   Add
end;

What this function does is removes itself and the immediate caller from the call chain and directly transfers control to the corresponding "unsafe" method while retaining the passed in parameter(s). As we'll see later on, we're going to create a generic descendant class from this one where the method bodies of the actual typesafe versions of Add, Remove and IndexOf merely call the corresponding "InternalXXX' versions from this base class. Here's some of what that descendant will look like and the body of the Add method:

type
  TMulticastEvent<T> = class(TMulticastEvent)
  public
    procedure Add(const AMethod: T); overload;
    procedure Remove(const AMethod: T); overload;
    function IndexOf(const AMethod: T): Integer; overload;
  end;

procedure TMulticastEvent.Add(const AMethod: T);
begin
  InternalAdd;
end;

Ah, there's the "T"... It's beginning to look pretty generic. Why do it this way? The primary reason is that you cannot have any assembly code in the method body of any generic type or method, but you can refer to other methods that are written in assembly. So in this case the ancestor class merely provides those "non-generic-able" assembler functions. Another reason, which we'll get to, is that I also want to make sure that iterating through the array of event handlers is also done in some common code, which will also need to be in assembly due to some more stack tricks that must be done.

When this generic class is instantiated with an event type, the Add, Remove, and IndexOf methods will always get a stack frame. Even though the method body doesn't touch the parameter, the frame is still there and needs to be cleaned up because the size of the event type is > 4 bytes. Since it wont fit in a register, it is always passed by value on the stack. This is also why I declared a private TEvent type up in the base class so that the stack pattern will match. If I had merely declared the parameters as a TMethod, which is a record that allows you to "crack open" the event type, it would not have worked. This is because method pointers are passed differently from records because you can directly refer to a method and instance in code as the parameter when calling a method that takes a method pointer:

begin
  ...
  MC.Add(Obj.EventHandler);
  ...
end;

In this case the compiler simply pushes the instance reference from the variable Obj onto the stack, followed by the address of the EventHandler method. However for a record, the compiler passes a pointer to the record, which may even be in a register. Then, depending upon whether or not the parameter is declared as "const" or not, the called method will use the pointer directly or make a local copy of the record on the stack.

I won't bore you with the details of how the Add, Remove, and IndexOf methods actually interact with the array as that really isn't the interesting part and is merely boilerplate code anyway. Let's instead turn our attention to triggering or invoking the event. Let's add back in the FInvoke field and the Invoke property:

type
  TMulticastEvent<T> = class(TMulticastEvent)
  strict private
    FInvoke: T;
  public
    procedure Add(const AMethod: T); overload;
    procedure Remove(const AMethod: T); overload;
    function IndexOf(const AMethod: T): Integer; overload;

    property Invoke: T read FInvoke;
  end;

So here we are again. What is assigned to the FInvoke field? This is where yet another trick from my bag comes out. We're going to leverage some of the dynamic dispatching code that has been laying somewhat dormant down in the ObjAuto.pas unit for many Delphi releases. If you haven't had the chance to look at some of the little gems that live in that unit and the associated ObjComAuto.pas, there is some pretty powerful bits of code in there. ObjAuto's primary purpose is to allow late-bound calling of object methods. It does this through some rather rich RTTI that the compiler has generated for several releases. A lot of this extra information is generated for public and published methods on classes and descendants that have been declared within a {$METHODINFO ON} block. However, for method pointer types, this information is always available regardless of that setting. While I'm not going to be generating code at runtime, I will be calculating some information about call signature of the method at runtime from this extra RTTI.

In ObjAuto, there is an interesting global function called CreateMethodPointer(). This function takes a pointer to a type data record (the compiler RTTI) and an interface. It returns a TMethod record that points to a stub object and method that knows how to "pick apart" the passed in parameters convert them to Variants and then call some methods on the interface. Don't worry, we're not going down that (using Variants) road! Down in the implementation of this unit was something much closer to what I wanted. I just need raw data structure that was an opaque representation of the Invoke call. I don't care what the data is, only how large it is and where it is located. IOW, what data is in a register and what data is on the stack and how much. I won't go into the raw details of the changes made down in ObjAuto since you'll be able to see that once Tiburón ships. I added another overloaded CreateMethodPointer() function, that takes a method pointer callback instead of an interface and a pointer to a type data record. Here's what the declarations look like:

type
  ...
  PParameters = ^TParameters;
  TParameters = packed record
    Registers: array[paEDX..paECX] of Cardinal;
    Stack: array[0..1023] of Byte;
  end;
  TDynamicInvokeEvent = procedure (Params: PParameters; StackSize: Integer) of object;

function CreateMethodPointer(const ADynamicInvokeEvent: TDynamicInvokeEvent; TypeData: PTypeData): TMethod; overload;
procedure ReleaseMethodPointer(MethodPointer: TMethod);

Call CreateMethodPointer with a reference to a method of type TDynamicInvokeEvent and the type data for the method pointer (T). It will return a TMethod record, which is essentially a raw method pointer. We'll use this value to assign to FInvoke. The trick is to get around the fact that the compiler won't allow this cast:

  FInvoke := T(CreateMethodPointer(...)); // compiler chokes on this code

We have to resort to another assembler trick. Remember how I said that passing a method pointer as a parameter is a little different that with records? In this case, it's not going to matter because we're going to declare a method in the generic class that takes a "var" parameter. In this case the compiler passes the address of the method pointer variable rather than the value. Let's add this method:

type
  TMulticastEvent<T> = class(TMulticastEvent)
  strict private
    FInvoke: T;
    procedure SetEventDispatcher(var ADispatcher: T; ATypeData: PTypeData);
  public
    < same as above >
  end;

procedure TMulticastEvent.SetEventDispatcher(var ADispatcher: T; ATypeData: PTypeData);
begin
  InternalSetDispatcher;
end;

There's another "InternalXXXX" call! That one is only slightly different than the others, but the effect is the same. InternalSetDispatcher simply jumps over to the SetDispatcher method on the TMulticastEvent type, which is declared and implemented on the base TMulticastEvent class:

type
  TMulticastEvent = class
  strict protected type TEvent = procedure of object;
  strict private
    FHandlers: array of TMethod;
    FInternalDispatcher: TMethod; // this class needs to keep it's own reference for cleanup later
    ...
    procedure InternalInvoke(Params: PParameters; StackSize: Integer);
    procedure SetDispatcher(var AMethod: TMethod; ATypeData: PTypeData);
  protected
    ...
    procedure InternalSetDispatcher;
  end;

procedure TMulticastEvent.SetDispatcher(var AMethod: TMethod; ATypeData: PTypeData);
begin
  if Assigned(FInternalDispatcher.Code) and Assigned(FInternalDispatcher.Data) then
    ReleaseMethodPointer(FInternalDispatcher);
  FInternalDispatcher := CreateMethodPointer(InternalInvoke, ATypeData);
  AMethod := FInternalDispatcher;
end;

We're almost there. One of the last bits to cover is the InternalInvoke method and the constructor of the generic TMulticastEvent class. InternalInvoke is called from the little stub class instance created by the CreateMethodPointer() call. It passes in the pointer to the TParameters record and the space taken up on the stack by the parameters that "spill" from the CPU registers. As you may have already guessed, we're going to drop down to some more assembly:

procedure TMulticastEvent.InternalInvoke(Params: PParameters; StackSize: Integer);
var
  LMethod: TMethod;
begin
  for LMethod in FEvents do 
  begin 
    // Check to see if there is anything on the stack. 
    if StackSize > 0 then 
      asm
        // if there are items on the stack, allocate the space there and 
        // move that data over. 
        MOV ECX,StackSize
        SUB ESP,ECX 
        MOV EDX,ESP
        MOV EAX,Params 
        LEA EAX,[EAX].TParameters.Stack[8]
        CALL System.Move 
      end; 
    asm
      // Now we need to load up the registers. EDX and ECX may have some data 
      // so load them on up.
      MOV EAX,Params 
      MOV EDX,[EAX].TParameters.Registers.DWORD[0] 
      MOV ECX,[EAX].TParameters.Registers.DWORD[4]
      // EAX is always "Self" and it changes on a per method pointer instance, so 
      // grab it out of the method data. 
      MOV EAX,LMethod.Data 
      // Now we call the method. This depends on the fact that the called method 
      // will clean up the stack if we did any manipulations above. 
      CALL LMethod.Code 
    end; 
  end; 
end;

Now the constructor for the generic version of TMulticastEvent<T>:

constructor TMulticastEvent<T>.Create;
var
  MethInfo: PTypeInfo;
  TypeData: PTypeData;
begin
  MethInfo := TypeInfo(T);
  TypeData := GetTypeData(MethInfo);
  inherited Create;
  Assert(MethInfo.Kind = tkMethod, 'T must be a method pointer type');
  SetEventDispatcher(FInvoke, TypeData);
end;

Notice the Assert, that is there because we cannot "constrain" the generic type at compile-time to only accept method pointer types for "T". So we have to do that check at run-time. This is something we'll look at for future releases. We can safely call SetEventDispatcher because FInvoke will be of type "T".

There you have it, a generic multicast event. At some point, I'll present some extensions that will help make using this easier by adding some automatic cleanup of the multicast event instance itself and a technique for using the source event itself to be the only instance reference. For now here's one potential use scenario:

procedure TForm1.FormCreate(Sender: TObject);
var
  E: TMulticastEvent<TNotifyEvent>;
begin
  E := TMulticastEvent<TNotifyEvent>.Create;
  // Since the Invoke property is of type "T" and T = TNotifyEvent, you can
  // Assign it directly to any event of type TNotifyEvent, such as OnClick.
  Button1.OnClick := E.Invoke;  
  // Now you can add as many "listeners" as you want.
  E.Add(Button1Click);
  E.Add(AnotherButtonClick);
end;

I will try to get the full code posted within the next few weeks or so. I still have Tiburón to get out the door and take a few days off.

There are also a lot of caveats and cases where a multicast event can really mess things up. For instance, what if the method pointer had a result type? What if it had one or more "var" or and "out" parameters? Which method invocation in the list of listening handlers decides what the result is, or assigns to an "out" parameter? What if an exception is raised? Are there any ordering dependencies or expectations? From a framework designer standpoint, these are some real questions one has to ask. Making all events multicast is akin to the shotgun approach of making all methods virtual. It doesn't really foster extensibility in the way one would think. As a matter of fact it can serve to severely limit extensibility and evolve-ability of a given framework.

There are many types of events that lend themselves very well to supporting multicasting, and there are plenty of other cases where multicasting only creates more problems than it solves. Having multicast events as the default behavior for any event type is certainly nice from a consistency and overall availability standpoint, but it just makes it way too easy to hang yourself inadvertently. Supporting multicasting for an event source should be a decision left to the designer of the class on which the event is placed.

Tiburón - String Theory

No, not that String Theory, or even this one. What this is about is an interesting extension to AnsiString. During the field test cycle of Tiburón and our own internal porting of the IDE code (which was accomplished in about 1.5 months, by 2-3 folks, with > 2 million LOC), it became clear that there was a need for easily encoding UTF16 character data as UTF8. For the astute among you, you probably already know about the RTL defined UTF8String which is really just an alias to AnsiString. The problem is that it is UTF8 in name only. Unless you explicitly ensured that only UTF8 data was placed into the payload, it could just as easily hold normal Ansi character data. We needed to make the use of UTF8String easier. As we looked at how AnsiString worked, it was clear that AnsiString always had this "affinity" to carry it's data payload encoded as whatever the RTL had determined was the active code page, at runtime. So we wondered, "what if we could create an AnsiString type where the programmer determined at compile time, the code page affinity for AnsiString?"

It turns out that, to Windows, UTF8 encoding is just another code page. Down in the RTL, the conversions to/from UnicodeString or WideString use the Windows API functions WideCharToMultiByte() and MultiByteToWideChar(). One of the parameters is the code page identifier to or from which the data is converted. If you pass CP_UTF8 (65001) to those functions, they'll convert between UTF16 and UTF8. This is a lossless conversion. In Tiburón, we're introducing an enhancement to declaring your own "typed" AnsiString. You have always been able to create a unique type based on any intrinsic type by declaring the new type with the "typed type" syntax:

type
  MyString = type AnsiString;

This would create a new type that is assignment compatible* with AnsiString, but with a unique type name and a unique RTTI structure. We've used this in VCL to distinguish normal "strings" from special strings such as TFileName or TCaption. By creating these unique string types, it was possible to create property editors that would be associated with a specific type of property, regardless of which component it used on or what the property name was. This is how only the Caption property on many components will automatically update the live design-time component as you type in the Caption value.

The thing is, with the above declaration, MyString will continue to always have an affinity for whatever the current runtime code page was. So, we introduced the following syntax for AnsiString only:

type
  MyString = type AnsiString(<1..65534>);

You can now control the code page affinity of any "typed type" AnsiString at compile time. The "parameter" to AnsiString must be a word constant expression. The values 0 and 65535 have special meanings. 0 is a normal AnsiString, and 65535 ($FFFF) means "no affinity." $FFFF is worth noting here as already being declared as a RawByteString. When assigning between AnsiStrings or passing them as parameters, if the code page affinity of the source and destination strings are different, an automatic conversion is done. In order to minimize potential dataloss during the conversion, all conversions go "through" UTF16. However, a string with an affinity of $FFFF tells both the compiler and the RTL, that none of these conversions should be done and to just move over the payload. In practice, however, there would be only a few instances of needing to use RawByteString, but it is there for your use.

So we now have the following declarations in the System unit:

type
  UTF8String = type AnsiString(CP_UTF8);
  RawByteString = type AnsiString($FFFF);

Like I stated previously, any assignment (or passing as a parameter to procedure or function) where the code page affinity between the source and destination are different, the payload will automatically be converted. Say you have a function that must only take UTF8 data. You can declare it like this:

procedure WriteUTF8Data(const Data: UTF8String);
begin
  // write UTF8 data to stream, file, socket, etc...
end;

Now, no matter what type of string you pass to this procedure, you know that the payload will have been coerced into UTF8. Pass a normal AnsiString, and the data arrives as the UTF8 version of that AnsiString converted from whatever the active codepage was or whatever that AnsiString's affinity was set to. Pass a UnicodeString or WideString to the function and it too will be converted to UTF8. Pretty cool, no?

With the title to this post... All those physicists out there are going to hate me and Google.. hehe ;-)

 

*"Assignment compatible" means that you can assign or pass as a parameter from one "typed type" to another "typed type" or the intrinsic type on which it was based. They are not "var/out parameter" compatible. This means you can pass them by value to a function but not by reference.

Brand New Day...

Well, my access card worked.  I guess I still have a job :-).

I just finished listening to a company (Embarcadero not Borland) wide conference call announcing to the whole company the closure of the Embarcadero+CodeGear deal. Wayne Williams, our new boss, made some very encouraging statements. Most notably was that just like ER/Studio, RapidSQL, and DBArtisan, Delphi/C++Builder are core Embarcadero product offerings. This means that these products are key to the business. Yes, there are many other products being sold, incubated and introduced, but it is the aforementioned products that form the pillars on which the company is based. Without them, many of the other products would not be possible. Like the foundation of a home, you just don't take a metaphoric jackhammer to them and expect the structure (the company) to remain sound.

Another encouraging (or maybe scary, depending upon your perspective) point was that we are the last independent tools vendor with the breadth of offerings we have out there. This means no vendor or stack lock-in. We have tools for nearly every database and OS platform out there. We are also one the of the very few software companies that offer very strong non-Open Source tools right along side tools built either on or for Open Source stacks. JBuilder and 3rdRail are built on top of the very popular Eclipse framework. Delphi for PHP and 3rdRail are built for the very popular and widely used Open Source PHP and Ruby/Rails environments, respectively.

How things will change or even stay the same is still being planned and scoped. A lot of work had been done between the announcement of this deal and its close, but now is where most of the work can actually take place. Now that we're no longer joined to Borland, we can now chart a new course under a new captain. I wish all the best for Borland as I've seen many happy and exciting days while there.

An interesting anomaly is that many of the CodeGear folks have had their service bridged, which means that some of us have, on paper, now worked for Embarcadero longer then they've existed :-). This is now day 6022 for me.

The Last Day.

Today is my last official day as a Borland/CodeGear employee. After today, I will have been a Borland/CodeGear employee for 6021 days, or 16 years, 5 months, and 25 days. What is interesting about this is that tomorrow, I will continue to drive to the same building, ride the same elevator, and unlock the same office door. The only difference will be that I will be employed by Embarcadero. It has certainly been an interesting ride in my many years at Borland/CodeGear.

For some perspective, I started when the Borland stock was around $80 a share (ouch). The new Borland Campus wasn't completed. Borland occupied a reasonably large number of buildings throughout Scotts Valley. If Borland didn't occupy some or all of a building, chances are Seagate did. To move between the various buildings, there was a continuously running shuttle bus service for Borland employees. I also remember the clout that Borland had locally. When we (my family and myself) first arrived and were looking for someplace to rent, when asked where I worked and I said Borland, suddenly we were at the top of the list without any kind of other checks. It was pretty nice.

After today, there will be no official presence of Borland in Scotts Valley. The only thing linked to Borland will be name on the lease they maintain on part of this building that was once the Borland Corporate HQ and campus. Embarcadero will sub-lease the space we currently occupy from Borland.

Even though there have been many things Borland has done over the years that I firmly disagreed with, they've also done some pretty good stuff too. I've learned so much more than I ever could have imagined about software, the software industry, developer tools, frameworks, and compilers. I've also had the pleasure of knowing and meeting many folks in the industry from all around the world.

Tomorrow will start a new chapter with new challenges and opportunities. I'm sure it will take some time until the dust settles, the cultures merge, and things settle down to the right groove. Don't expect any earth-shattering announcements or radical changes immediately out of the gate. This is a new experience for nearly all involved, so some period of acclimation is bound to take place.

>10 years in the making...

Sure, the latest news of Embarcadero Technologies signing an definitive agreement to acquire the CodeGear portion of Borland's business is a recent development. However, looking back on my years here at Borland, I have to chuckle a little bit. I remember sitting with other Delphi team members many years ago on more than one occasion waxing poetically about how great it would be to spin out Delphi and the other dev tools products into either a stand-alone company or into the arms of someone who will nurture and cultivate their value. While at the time it was idle chit-chat, who would have thought that this day would actually come?

Throughout the day, yesterday, I got into several IM conversations from former co-workers all asking how I felt about the news. I honestly told them that this is the best outcome for the future of the developer tools products anyone of us here could have imagined. As I was on my way from work last night, I got a phone call on my mobile phone. I was wondering when I would get this call. The voice on the other end said, "Hi Allen, this is Chuck." Yep, I was totally expecting a direct phone call sometime from Chuck. If you've been following any of my posts over the last few years, you know that Chuck Jazdzewski was very instrumental in my getting hired at Borland. Chuck served as my mentor for many years.

Chuck's first question was one of concern and wanting to know how I felt about these developments. Thanks, Chuck, that meant a lot. We talked for a while as I was driving. Don't worry, I use a bluetooth hands free headset :-). Part of the conversation quickly drifted to our many conversations in the past about how cool it would be to take the dev tools out of Borland. Unfortunately, I arrived at my destination and had to cut the conversation short. I would like to have continued. Chuck agreed that if he had any other questions that he'd be sure to give me a call.

If there is one thing about my tenure with the Borland developer tools group, it has rarely ever been a dull moment. I doubt that is going to be changing any time soon. Change is something the CodeGear teams have become accustomed to. It's great that this change is one of the most positive ones :-).

What a day...

Needless to say, it's been a crazy day here at CodeGear. I'm sure the same can be said of the fine folks at Embarcadero. Rather than saying the same things you can read about in all the various outlets and blogs, I figured I'd give folks my impression of how this is affecting the teams here at CodeGear.

At 9am PDT, we held a company "all-hands" meeting here in Scotts Valley, CA. Since this meeting coincided with Borland's earnings announcement, this was not an unusual event. Oh sure, the internal rumors had been flying for months. Folks kept saying that "something is happening." After the usual presentation and outline of the deal, there was an opportunity for a Q&A session with Jim Douglas, the CodeGear CEO. Jim was candid and frank. Like anything that is a major change, there is bound to be some level of nervousness. In this case, the employees were far more engaged in the meeting that I've seen in the past. It seemed that there was this collective sigh of relief among the whole crowd. We finally made it!

Michael Swindell presented information about how this is a great fit for both companies and began outlining a lot of the immediate and long term opportunities. When something fits this well, it's easy for people to just "get it." It's not a tough thing to "sell." If you're a CodeGear customer, I would encourage you to head on over to Embarcadero's site and just browse around and look at what they're offering. You should also take a moment to read this blog post from Greg Keller.

Throughout most of today, intermixed with doing a little bit of coding, I've also been cruising the newsgroups, blogosphere, and other message boards. So far the overall tone is very positive and upbeat. There are lots of questions and concerns. It's only the first day, so I'm sure as folks have a chance to digest what this all means, the real questions will start flowing. Off the top, right now there are no plans to veer from our current roadmaps or to interrupt any current schedules. As a matter of fact, this was the directive given from the Embarcadero folks to make sure we're still on track to deliver what we've committed to.

I also spent some time roaming through the halls here in Scotts Valley.  After the 9am meeting, there was a lot of hallway conversations about what this all means. There was no "hand-wringing" and "angst" among the team that I could see. In fact, most folks were already talking about what Embarcadero does, the products they offer, and how they fit with what we're doing. Those are exactly the conversations we should be having internally. By the afternoon, most of the team had settled back into their routine and the development engine was back running on all cylinders.

So if you have any questions or comments, feel free to let me know and post through the comment system here. As I already stated above, it is business as usual in terms of products, schedules and features. As we approach the final close of this deal, more and more information will become available. So if I cannot answer your question right away, I hope to be able to do so in the coming weeks.

UPDATE: It seems that many of you are already hitting Embarcadero's site.  It appears to be suffering from a mild case of "slashdot effect". This is a good thing. Let's make sure they know how engaged, passionate, and committed the CodeGear customers are.

Fly! Be free! (this time for sure!).

Just in case you've not heard the "Good News", you should read it here... go ahead, I'll wait.

 

Well? Is that not some interesting news? If you've been wondering why this blog has been somewhat quiet lately, it's been due not only to being "heads-down" on Tiburón, but also because of this news. It's been a long-time in coming, that's for certain. In the coming days, you're sure to hear a lot more about what this means and comments from various sources.  Once I've had a chance to gather together some more information, I'll be sure to post it here.

When being "objective" is being "biased"

I did the most unconscionable thing a few weeks ago. Something I've resisted for a very long time. I bought a Mac. For the first time. Ever. I'm not against Macs or Apple in general (I own an iPod and a Zune), in fact I find them very quaint and highly capable. Ever since they succumbed to the Intel juggernaut, my interest was piqued. They are no longer tied to what had become a "boutique" CPU, the PowerPC, but rather to a tried-and-true, mainstream, continuously developed and advancing architecture.  Say what you will about all the flaws (and there are many) with whole x86 architecture, it is here to stay, is getting some serious industry investment from both Intel, AMD, and even VIA.  Remember the whole IA64 (Itanium) phase?  This just proves the point that even Intel cannot stop the force they created.  AMD, recognized this early on and created the AMD64 extensions to the existing x86 architecture.  Eventually, Intel realized that they're only reasonable course of action was to jump onto that bandwagon.  It is the biggest example of "if you can't beat 'em, join 'em."  Apple finally realized this when the powerPC just wasn't able to keep up in terms of raw CPU power and the onslaught of the multi-core CPU. Apple had to cut costs, and the easiest way to do it was to join the commoditized-to-the-hilt x86 industry.  Yes, this is old news.

I'll get to the real motivation for swallowing my pride and buying the Mac in a moment. This article popped up on my RSS feed and it really resonated with me.  Regardless of the comments, which all too painfully only served to prove the author's point (and which I'm sure there will be comments to this post which will do the same thing), it was interesting to see something I've long suspected, actually put into words.  It was also so apropos since I've, sort of, joined the ranks of Mac owners. The gist of this article and the book from which it is excerpted (from what I can gather), is that we're all very passionate, like to "belong," and want to feel validated and sometimes even lauded for our choice in car, home, clothes, spouse, occupation, religion, computer, etc... Yes, I admit that I sometimes feel much the same way as the stereotypical Apple-fanboy when confronted with valid, well-reasoned, unbiased analysis.

This hit home all too well lately because, well, this new Mac is not really mine.  Yes, I was involved with its purchase, but I don't actually use it.  See, my wife has decided to return to school and formally pursue something she's come to really enjoy over the past few years, graphic design and layout.  It started with doing monthly newsletters for the local Mothers of Twins club (my daughters are twins), then on to doing other various projects, and finally culminated in doing the entire Scotts Valley High School annual Football program.  She's already committed to do it again for the Fall-2008 season. She has used PCs (natch) for many years and has a very nice 17" notebook. Now that she's back in school, which is where she's at while I write this post, it became painfully obvious that there is a very clear bias in this particular field toward the mac.  Sure, the same applications are available for the PC, but to be truly "accepted" shouldering a mac really helps. Most printers happily work with raw files from these specific applications and will actually charge hefty fee if you submit work using certain PC-only applications. It also makes the professor actually want to talk with you.

Here is where this whole Mac-fanboy, "Apple can do no harm," attitude really hit me.  As relayed by my wife, this professor unequivocally stated on the first day of class that you must use the Mac version of the applications.  The pure shock and horror on the professor's face when asked if one could use the PC versions had to be priceless. A similar attitude was expressed when my wife submitted some homework which had a few things marked off.  The professor actually told her that she must not have used a laser printer because a laser printer would have printed correctly.  Well, that chapped my hide to no end... I wanted to load up my nice COLOR laser printer into the truck, drag it up to her class and plop it into the middle of the professor's classroom and insist that they prove to me that this is not a stinkin' laser printer! (no, it is not one of those dye-sublimation or "melted crayon" printers) That lasted for only a few moments as I stood there speechless and dumbfounded.  Upon further examination and obtaining more information, it turns out that it wasn't printed on the same brand and model laser printer that the professor used.  See this is a design and layout class for printed medium, and to grade the work, it is not about the content (which is usually that funky fake Latin text), but about its layout and presentation on the page.  So where things are is more important that whether or not you spelled some word correctly.  Ok, that makes sense, she'll have to use the school's laser printers instead of ours.

The whole purchasing experience was just too surreal.  You walk into an Apple store, and it is literally like being a voyeur to this whole cult-like Apple... thing...  I just cannot explain it.  We'd already researched exactly which Mac we're going to get and we'll use my wife's student ID for a discount.  It was simply a matter of walking up to the machine we wanted, point to it and grunt a few syllables about how we're going to pay for it and instruct the resident drone to walk back through the stainless steel door at the back of the store to obtain said box of Jobs' magic.  The top-of-the-line 17" MacBook Pro is certainly a sexy bit of hardware. You just cannot deny that. That was nearly four weeks ago. At about two weeks into this, things started going a little south on us.

Once some of the applications needed for the class arrived (after getting some nice deep student discounts), it was time to install them.  I get this frantic message from my wife saying that the machine won't recognize the disk, and now it won't even boot!  Hmmm, ok...  not. good.  Oh, yeah, the whole "see I knew it!" flags started popping up in my head all over the place.  They do have flaws!  Eventually, she was able to eject the disk, after making some really horrendous noises and still never recognizing the disk.  Less than two weeks old, and the Super-drive is dead.

You gotta love this, you can make an appointment at the "Genius bar" online.  So we did.  Drove back to the store and met with the "Genius" (yes, I'm snickering here...).  We explained that we've had the machine for barely two weeks and only now had an occasion to use the drive and this is what happened.   They tested it and, sure enough, no workee.  Just nasty noises.  I was ready for what happened next.  I asked if they're going to "send it in" for repair.  I was all ready to explain that if they went down that road, I'd be a little irritated because, to me this was a failure that left the factory in that condition and why should we be without the machine for however long it takes to fix it.  Also, I'd just be inclined to return it for a refund.  Luckily, they saw the logic agreed to simply replace the machine.

Yeah!  They're going to replace it... not so fast, there buckaroo. It turns out that we'd bought the machine only a few days before Apple decided to announce some minor changes to the whole MacBook line.  New things like LED backlit LCD, multi-touch track pad, newer Penryn-based CPU.  That all meant they were out of stock.  Rats.  They said they'd call once the new machines were in.  (I've heard that song and dance all too often) For the next week, we called the store several times and still no shipments. We contemplated simply getting a refund and ordering online, but that would mean not having the machine for school.  Finally, yesterday, they actually called. A new machine had been set aside and to come in anytime.  After arriving, we explained why we're there and they walked back through those sterile stainless steel doors, and returned with a box to which was affixed a post-it note with my wife's name on it.  Nice.

After about another hour as they transferred the documents, settings and applications, we were on our way.  And, yes, I did have them test the drive to make sure it worked :-).  All-in-all, Apple's service was pleasant and understanding.  I'd have to say that the hardware is no higher or lower quality than a solid ThinkPad or some of the newer Dells (older Dell notebooks had, ahem, some real issues).  They're certainly more "artsy" looking than a pure utilitarian PC notebook. And OS X? It's just different. Some things I like about it, and others just make me want to scream. I can say the same things about XP or Vista.

I'm sure many will see through my "unbiased" view straight to the obvious bias in this post.  And for those Apple-haters, I hope this redeems me; sitting next to me here in my home office, is a new, home-built, overclocked quad-core, 4GB, Vista x64, 512MB nVidia 8800, 1TB raid, PC rig that I had to build to seek forgiveness and cleansing from committing  Apple-adultery ;-).