Whither go the RTL

There seems to be some confusion cropping up with Delphi 8 for .NET. It seems that many folks are somewhat befuddled by the difference between a "library" assembly and a "package" assembly. It's really quite simple, but I can certainly understand the confusion. An assembly in .NET/CLR has the .dll extension. Well, for the Delphi programmer that means a "library" project, right? Well... almost... it seems that we inadvertently threw you a curve. An "assembly" in .NET/CLR is a very richly adorned .dll... not unlike a traditional Delphi "package." When we began digging under the hood of the CLR, we discovered that an assembly contains classes, metadata and code that are publically available to other assemblies and applications... Well... a Delphi "package" contains classes, metadata (RTTI) and code that are publically available to other packages or applications. I'm sure you see where I'm going with this. Yep, assembly=package.

What about "library?" It works too... Yes it does, but for different reasons and different purposes. You see, a CLR assembly can normally be called only from other CLR assemblies (or through COM-Interop, for the pedantic ones out there)... or so we thought. When we were looking at the whole of the managed code space, this included educating ourselves about CIL (Common Intermediate Language), CLR, C#, VB.NET and C++ with Managed extensions. It was that last one that sparked our curiosity. It seems that in C++ with managed extensions, you can create an assembly (you know; a .dll) that is directly callable from unmanged code! It can be called using the standard LoadLibrary, GetProcAddress pattern. No, you can't directly construct a managed CLR class, or call instance methods of that class, but you can call static methods. To the caller they simply appear as a flat "C"-style API. Of course you have to tell the compiler/linker which ones to "export" from the assembly and under what name. You also have to make sure the caller passes in the parameters correctly. But the interesing thing here is that all that work that the CLR does to allow P/Invoke (Platform Invoke) calls out to unmanaged APIs from managed code also works in reverse! This is how the whole thing operates. The final kicker in all of this is C# cannot do this.

What does that have to do with Delphi? Originally, the library syntax was used simply as an internal stop-gap measure until we had finished the compiler work to support "packages." The "library" syntax was earmarked for destruction. When we discovered that little tidbit above, the trusty ole' "library" syntax gained a new lease on life. We figured out that by pulling the same little linker tricks that the Managed/C++ compiler/linker do to export flat functions from an assembly, we can do the same thing. Hey, it's the little things that excite us sometimes... So now you can create a managed code assembly using the "library" syntax and then define exports via the tried and true "exports" clause. There is one little caveat to all these machinations. You must mark the library as "unsafe." This means that your managed code assembly no longer will pass the PEVerify test (a utility that is part of the .NET SDK determines if an assembly or managed application violates several key tests). This is due to the fact that the assembly now has true Win32/ia32 entry points. Since unmanged code can only call a dll via a "CALL" machine instruction, there needs to be native machine instructions at that entry point in order to properly transfer control.

So the general guide lines here are you should rarely, if ever use a "library" when you want to build an assembly. Use "package" for that purpose.

Now on to another bit of confusion. What is the deal with Borland.Delphi.dll? If I don't link against that assembly, things don't seem to function correctly. Lets first step back and get a little perspective. CLR/.NET defines a common set of rules, metadata formats, file formats, and data-types designed to allow language interoperability. This allows one to declare and implement a class type in one language then create a derived class type in another language. The CLR only defines those things nessesary for two types defined by different languages to play nice with one another. Languages are different for a reason. Syntax is the most obvious difference one thinks of when comparing two languages. The other differences that actually affect the run-time behavior of language are a little harder to characterize. For, instance CLR/.NET has no notion of a class method. They has static methods, but that is different than a Delphi class method. This is because a CLR static method really has no direct knowleged of the class in which it is executing. A Delphi class method has direct knowledge of the class in which it belongs. It has an implicit Self parameter that is actually a class type reference. This brings up the another difference, class references. Yes, CLR does has a similar entity called the System.Type type. Close, but no cigar. In Delphi you can also declare a class method as virtual and call it polymorphically through a class reference variable.

All these differences require some level of run-time support. There are also certain global variables that the Delphi programmer has come to expect from the RTL. This is the reason there is still the trusty old System unit, which in CLR is renamed to Borland.Delphi.System. This is where all the compiler "magic" functions reside, and all the classic base Delphi types are declared. If you look at other non-C# CLR languages, such as Visual Basic.NET, Visual J#, they also have a language-specific runtime like microsoft.visualbasic.dll. Once you understand that there are types and global data in that unit (and thus in Borland.Delphi.dll), you can see the importance of using packages instead of libraries and also make sure all your packages ultimately link to a common Borland.Delphi.dll. For an explination of what can happen when you mix seemingly identical types from two separate assemblies, check out this blog entry.

ATOM Feed is now available.

I've just added an Atom feed to this blog. You can find it here.
Want to know what an Atom feed is? Look here: atomenabled.org

Menu items, Toolbars, glyphs... etc..

One area that has always been a common source of questions with the Open Tools API is "how do I add a menu item and tool button to the IDE." Well... starting with C#Builder new methods have been added to INTAServices that make adding items such as these much simpler. If you look in ToolsAPI.pas at INTAServices:

  INTAServices = interface(INTAServices70)
    ['{89160C3A-8EF4-4D2E-8FD5-D8492F61DB3E}']
    {! AddImages takes all the images from the given image list and adds them to the
       main application imagelist.  It also creates an internal mapping array from the
       original image indices to the new indices in the main imagelist.  This
       mapping is used by AddActionItem to remap the ImageIndex property of the
       action object to the new ImageIndex.  This should be the first method
       called when adding actions and menu items to the main application window.
       The return value is the first index in the main application image list of
       the first image in the source list. Call this function with an nil
       image list to clear the internal mapping array. }
    function AddImages(AImages: TCustomImageList): Integer;
    {! AddActionMenu takes an action item, a menu item, and a menu item component name
       and inserts the action in to the main action list and the menu item into the
       menu either preceding or following the named menu component.  If the action
       component has an ImageIndex > -1, then the mapping table created by the
       previous call to AddImages above is used to determine the new value for
       ImageIndex.  NewAction can be nil, in which case only the menu item is
       added. Likewise, NewMenu can be nil, in which case the Name param is
       ignored and only the action is added to the main action list. If Name
       cannot be found, an exception is raised.  If the ImageIndex of NewAction
       is out of range, then it is set to -1. }
    procedure AddActionMenu(const Name: string; NewAction: TCustomAction;
      NewItem: TMenuItem; InsertAfter: Boolean = True; InsertAsChild: Boolean = False);
    {! NewToolBar creates a new toolbar with the given name and caption.
       If the ReferenceToolBar parameter is specified, it is used as a reference point
       for insertion based on the InsertBefore parameter.  If InsertBefore is True, then
       the new toolbar is inserted physically before the reference, else it is after.
       if ReferenceToolBar is not specified, then the toolbar is inserted into a
       position determined by the IDE. }
    function NewToolbar(const Name, Caption: string;
      const ReferenceToolBar: string = '';
      InsertBefore: Boolean = False): TToolbar;
    {! AddToolButton creates a new toolbutton on the named toolbar using the given
       Action component.  In order for the user to be able to add and remove this
       toolbutton, an Action *must* be specified.  otherwise the user may remove
       the button, never to return until the toolbar config entries in the registry
       are deleted and the toolbars are reset to the original configuration. If
       IsDivider is True, then Action is ignored since a divider toolbutton is
       created. If you wish the toolbutton to have a dropdown menu, then owner-
       ship of that menu *must* be transferred to the owner of the toolbutton. }
    function AddToolButton(const ToolBarName, ButtonName: string;
      AAction: TCustomAction; const IsDivider: Boolean = False;
      const ReferenceButton: string = ''; InsertBefore: Boolean = False): TControl;
    {! UpdateMenuAccelerators causes the IDE to reset all the assigned accelerator
       keys to the associated menu items.  This is the accelerators as defined in
       the current keymap }
    procedure UpdateMenuAccelerators(Menu: TMenu);
    {! ReadToolbar reads the configuration of the given toolbar from the
       registry and recreates it if necessary. If SubKey is specified, it will
       attempt to obtain a stream from that key first and if not found, then
       will get the stream from the main toolbar key.  Use Subkey to read
       view-specific version of a toolbar. You can also optionally pass in a
       TStream object if you wish to control the actual storage of the stream
       itself. Set DefaultToolbar to true in order to read one of the global
       toolbars from the "toolbar reset" storage.  This is the default
       configuration of the toolbar. }
    procedure ReadToolbar(AOwner: TComponent; AParent: TWinControl; const AName: string;
      var AToolBar: TWinControl; const ASubKey: string = ''; AStream: TStream = nil;
      DefaultToolbar: Boolean = False);
    {! WriteToolbar will take the given toolbar and write it out to the registry
       under the subkey name if specified.  Use SubKey to write view-specific
       versions of a toolbar. }
    procedure WriteToolbar(AToolbar: TWinControl; const AName: string = '';
      const ASubkey: string = ''; AStream: TStream = nil);
    {! CustomizeToolbars will open the toolbar customize dialog and set the given
       toolbars into customize mode.  The INTACustomizeToolbarNotifier interface
       is used to handler certain events during the customizing process.  If
       ActionList is specified, then only the actions in that action list can
       be used to customize the toolbar.  If not specified, =nil, then the IDE's
       global action list is used.  The return value is the customize dialog
       component.  You can add a FreeNotification in order to know when the user
       closes the dialog and customization is complete. }
    function CustomizeToolbar(const AToolbars: array of TWinControl;
      const ANotifier: INTACustomizeToolbarNotifier; AButtonOwner: TComponent = nil;
      AActionList: TCustomActionList = nil; AButtonsOnly: Boolean = True): TComponent;
    {! Call CloseCustomize when it is needed to forcibly terminate the customize
       mode.  For instance if the view being customized is destroyed or hidden,
       this procedure may be called to terminate customization. }
    procedure CloseCustomize;
    {! Call ToolbarModified if you wish to notify all interested parties that
       a toolbar was modified by means other than calling CustomizeToolbar.  For
       instance, when the toolbar is on a TControlBar and the toolbar band was
       repositioned.  This will cause all registered INTACustomizeToolbarNotifier.ToolbarModified
       events to be called. }
    procedure ToolbarModified(AToolbar: TWinControl);
    {! RegisterToolbarNotifier registers the given INTACustomizeToolbarNotifier
       in order to receive certain events related to toolbars and customizing
       them.  The most often used event will probably be the ToolbarModified
       event since that is how other views can know when a particular named
       toolbar is customized. }
    function RegisterToolbarNotifier(const ANotifier: IOTANotifier): Integer;
    procedure UnregisterToolbarNotifier(Index: Integer);
    {! MenuBegin/EndUpdate allows the caller to control how often the main
       menu will be updated.  If many changes to the main menu are made at
       one time performance can be improved by using these methods. }
    procedure MenuBeginUpdate;
    procedure MenuEndUpdate;
  end;

...

This interface is designed to be used when initializing an IDE addin. What many of the internally developed IDE addins do is to create Data Module. On that data module, drop a TImageList, TActionList, and TPopupMenu. Add the images you want to use with the toolbar buttons and menu items to the image list. Add the action items, reference the image list and the index of the specific images (these will be fixed up later). Now add a menu item to the popup menu and assign the specific action item to the Action property of the menu item. This is all done via the designer (Delphi 7 can be used to do this). Now, when you want to initialize your add-in, you need to create an instance of the data module then call in this order:

  var
    NTAServices: INTAServices;
    ToolBar: TToolBar;
    ...
  begin
    ...
    NTAServices := BorlandIDEServices as INTAServices;
    ...
    NTAServices.AddImages(ImageList1);
    // Inserts the new menu item just before the File|Exit... menu item
    NTAServices.AddActionMenu('FileExitItem', ActionItem1, MenuItem1, False); 
    NTAServices.AddActionMenu(...);// add any other menu items.
    ...
    ToolBar := NTAServices.NewToolBar('NewCustomToolbar', 'Allen''s Custom Toolbar');
    NTAServices.AddToolButton('NewCustomToolbar', 'ActionButton1', ActionItem1);
    ...
  end;

...

One item of note is that currently, if you attempt to query BorlandIDEServices from within IDERegister, it will fail. This is due to the fact that IDERegister is call very early in the initialization cycle. One workaround is to set a timer and try it again periodically. Another, more accurate technique is to create a hidden window (See AllocateHWnd and DeallocateHWnd in Classes.pas), and post a message with PostMessage to that window handle. This will ensure that you know the initialization is complete because the message loop has started (TApplication.Run).

The above technique is how all internally developed IDE personalities and extensions add their items to the main menu. The benefit of following these procedures is that your newly added action items will now participate in toolbar customization and persistence. Some of the more advanced uses of this interface and the attending INTAToolbarNotifier interface you can implement, is the ability to have local-view toolbars that can be persisted and designed just like the main toolbars (see the toolbar on the WinForm Designer view).

Object Inspector API

Starting in C#Builder, the ability to provide an arbitrary selection to the Object Inspector was added to the native Delphi Win32 Open Tools API. There are two mechanisms you can use to get a selection into the OI. The first, and most complicated is to implement the IOTAPropInspSelection interface and several other optional interfaces. The idea is to provide a list of IProperty interfaces (defined in DesignIntf.pas). There are also several new interfaces you can implement in addition to IProperty to get additional features like more customized control over painting a particular item, custom drop-down editors, etc.. but that is a subject for another blog entry. The second mechanism simply allows a list of native Delphi Win32 object instances to be selected.

IOTAPropInspSelection is a very rich and complicated interface to implement, but gives the most level of control over virtually all aspects of the OI, including the hiding and showing of the description and hot commands panes. The description pane is a way to provide a simple help text that describes the currently selected property. The hot commands pane is an area where an arbitrary number of "hot-links" are displayed that the user can click in order to invoke functionality related to the entire selection.

INTAPropInspServices is a shortcut. Using this interface (which is queried for from the IOTAPropInspServices interface), you can simply provide an array of TPersistent object instances, and the OI will then select and display the property list of published properties. You can even call RegisterPropertyEditor in DesignIntf to register specific property editors in order to customize the behaviour in much the same way a Delphi component writer would register custom property editors for their design-time components. In fact, this is the exact mechanism that is used to display information about the specific items in the project manager. When the user selects an item in the PM, that item is asked for a TPersistent that represents its specific properties and is selected into the OI by calling INTAPropInspServices.SelectObjects(). One "gotcha" is that as long as that item is selected in the OI, it must remain a live instance. If you free the instance too early, the OI and the property editors will continue to attempt access to the now dead instance, which is a recipe for disaster.

There are some differences in the API between C#Builder and Delphi 8, but a simple diff of the PropInspAPI.pas files will reveal them. Also there is a reasonably detailed reference comments in that file that describes what each method/property does on the interfaces.

IDE Integration pack for Delphi 8 and C#Builder

As promised, I have uploaded to CodeCentral two zip files that contain the necessary files to compile Win32 VCL based IDE extensions.

For Delphi 8:
http://codecentral.borland.com/codecentral/ccweb.exe/listing?id=21333

For C#Builder 1.0:
http://codecentral.borland.com/codecentral/ccWeb.exe/listing?id=21334

Please note that the declaration for TDesktopForm.LoadWindowState and TDesktopForm.SaveWindowState have changed:


TDesktopForm = class(TForm, IEditHandler)
...
public
...
procedure SaveWindowState(Desktop: TCustomIniFile; isProject: Boolean); virtual;
procedure LoadWindowState(Desktop: TCustomIniFile); virtual;
...
end;


There are also two new files in addition to the changes made to ToolsAPI.pas; PaletteAPI.pas and PropInspAPI.pas. These two files describe the APIs to the new Tool Palette and the Object Inspector, respectively.

Please note that these are provided as-is and are unsupported. You will be required to click through a license stating this in order to download the files. Make sure you do not mix the C#Builder version with the Delphi 8 version and vice-versa. Also, an IDE add-in built with the files from the C#Builder integration pack will not work on Delphi 8. The same holds for IDE add-ins built with the Delphi 8 version cannot be used with C#Builder.

The version of the Win32 VCL shipped with Delphi 8 is different that the Delphi 7 and may contain some subtle differences in behavior and properties.

Any bug reports should be made through Quality Central.

Commenting changes.

Sorry for being a little flakey here, but it turns out that I can use php pages on http://homepages.codegear.com. So I've downloaded and implemented comments locally instead of using the HaloScan. This way I'm less at the whim of a separate hosting company and closer to it all being integrated into the Borland community site.

More IDE secrets - UTF8 and the Editor

In C#Builder and Delphi8, the code editor operates internally on UTF8 characters only. This requires a "filtering" mechanism to translate from/to various other text file formats when the file is loaded or saved. These are the "filters" you can select by right-clicking in the editor and selecting the encoding from the menu. By default, the editor will store the file as locale-specific ANSI encoding which can lead to a potential loss of character data if a file from another locale using a different code-page were loaded and saved. You can change the default encoding of the files to always be UTF8 by setting the following key:

C#Builder: HKCUSoftwareBorlandBDS1.0Editor
Delphi 8: HKCUSoftwareBorlandBDS2.0Editor

DefaultFileFilter="Borland.FileFilter.UTF8ToUTF8"

Using UTF8 encoding when operating on a file in memory instead of straight UCS-2 (Unicode) was done for efficiency, not only in implementation time, but also in terms of memory usage. The editor kernel already knew how to manage middle- and far-eastern multibyte codepages, so extending the kernel to simply treat UTF8 as simply another multi-byte encoding was a relatively trivial excercise. Also, since the vast majority of source files contain only characters from the 0-127 ASCII range, each character will remain a single byte. Only embedded strings and comments would typically have extended (>128) characters. Also, UTF8 conversion is a very fast bit-level transform without the need for look-up tables. This allows the editor painting code to do a simple quick transform into UCS-2 and then use the Unicode APIs for painting the text. This way, a file created from one locale will render correctly when opened and edited in another locale since the UTF8/Unicode character space includes encodings for all languages.

Commenting..

I've gone ahead and added commenting to this blog. It is currently hosted by HaloScan, which is a free comment hosting site. Eventually, this whole sha-bang, will move to using a Borland provided blogging and commenting system. For now I'll stick with what is already out there and available.

Where's Waldo... er... Allen?

It's been close to a month since my last blog entry and a *lot* of stuff has happened since then, both personal and corporate. On the corporate front, Delphi 8 for .NET was released to manufacturing on Dec 19, 2003. This is great news because it meant we were able to take advantage of the two-week corporate shut-down.

On the personal front, I lost a very dear person in my life. On December 30th, my mother succumbed to lung cancer. This was quite a shock to my family as she has never smoked a day in her life. However, I am truly thankful that I was able to be with her for her last few days and was by her side the moment she passed along with my wife, sister and brother-in-law. She will be sorely missed. So, until our spirits meet again, bye Mom...

The next few weeks are going to be interesting as we begin execution and planning for the next product releases. I will try and keep things going here, but there will certainly be many distractions as the meeting counts increase and while we slog through the details of planning product cycles.