One namespace to rule them all...

Right before Danny left on his trip to Amsterdam, London and the grandparents, I presented a potential solution to the multi-unit namespace issues. Danny was quite excited about the idea. I've been discussing it with others internally and so far haven't been able to shoot too many holes in it (at least none that we can't easily deal with). Since Danny is due to be back in the office tomorrow, I'm anxious to hear if he's been able to poke any more holes in the idea. I'm sure you'll be able to read about it from either one or both of us within the next couple of weeks.

2rss.com ad free..

Apparently Anders Ohlsson struck a deal with the owners of 2rss.com so that all the Borland bloggers now have ad-free atom->rss feeds. Good job Anders.

2RSS.com & adverts..

Anders Ohlsson just posted this regarding the fact that 2rss.com is now injecting ads into my rss feed. This blog entry on their site explains why they started doing this. Unlike Anders, however, I cannot actually fault them for this since, as they explained, they have to justify providing a free service. Also, the $14.95 annual fee seems like a reasonable charge in order to use the service ad-free. On the other hand, it would have been nice if they had injected an entry into the feed that merely points out that they would start injecting ads and that you can sign up for the service to support their efforts.

So far the ads seem innocuous and unobtrusive... On the other hand, this may be a catalyst to finally get a blogging solution up and going on the Borland Developer Network site.

Delphi 7.1 and persistent fields

The following is a post to the borland.public.delphi.non-technical newsgroup from Mark Edington, a senior engineer on the Delphi development team:

Sorry about the delay in commenting here. My newsreader was giving me fits
trying to post yesterday and I've been running around like a headless
chicken dealing with this issue.

Clearly many folks (and I'll put myself at the top of the list),
underestimated the ramifications of the change that was made to to enforce
the size matching. I was a little hesitent at the time I made the change,
but I really had no idea how common it it is for folks to have this
situation in their applications.

As it stands, the fix is not helping anyone because it has adversely
affected so many applications (and clearly that was not the intention of the
change), so the arguments about whether or not the DB code should enforce
the size matching are somewhat moot. We don't want to be introducing
changes in a patch (or even a point release) that have this effect. Period.

The original TClientDataset defect that prompted the change (QC#3874) can
and will be fixed in a different way.

We are are urgently working on a remedy for this situation and I will be
posting more information regarding that as soon as it is available.

For folks who installed the patch and are affected by this it is my
recomendation that you simply revert the related files from the original
Delphi 7 CD. There are only a handful of fixes in those files anyway. I'll
outline what to copy from the CD below.

By way of defense for the change, I would point out that the size checking
code introduced is actually a slightly more relaxed version of the checking
that was done in Delphi 4 and prior releases. So it's not something
completely new. On the other hand, it is certainly valid for the size of a
persistent field to be larger than the size of the underlying field in the
case where the data is read-only and at the very least that provision should
have been coded in as well. In any case, it is my opinion that the "fix"
should be completely removed and the code restored to the way it was in
Delphi 7.0.

Finally, I would like to offer my most sincere apology to you and the other
folks who were impacted by this issue.

Mark

I couldn't have stated it better myself. I was involved with a rather lengthy meeting regarding this very issue and can certainly attest to the fact that we are working as quickly as possible to get a resolution delivered as soon as possible. Please bear with us while we try and shift gears...

What is a "Breaking interface change?"

There seems to be a level of confusion regarding what constitutes a source code change that would result in symbol version mismatch errors. The rules are actually rather simple, but somewhat obscure and steeped in various levels of voodoo.

The "unit" structure of Delphi (or Borland Pascal) determines what other units can "use" from a particular unit. By placing declarations in the interface section of a unit, the user is publicizing the existence of a particular type, constant, variable, or function for external use. This declaration forms a sort of "contract" between the user of that symbol and the implementation of that symbol. For instance, a class declaration describes general structure of a particular class in addition to the actual physical layout of an instance of that class. When another unit (or program) references that class symbol and then it is compiled, certain bits of information are persisted to the binary .dcu/.dcuil file that describes in very fine detail the actual physical layout of an instance of that class, the class vmt (virtual method table), the RTTI, etc.. For instance, given a class TFoo with a virtual method Bar, a copy of the vmt is placed into the .dcu file with certain offsets. For a descendant of this TFoo, any virtuals it may introduce, will now be appended to the end of ancestor's vmt and the offsets to those entries are recorded. The same thing happens for fields as they have the offsets into the class instance recorded as well.

Now, say you decided to change the declaration of TFoo and wanted to add a new virtual method Baz. When the unit is recompiled, this method would now be inserted into the vmt and all the offsets are recalculated. This represents a classic "breaking" change. In most cases the user is totally unaware that this happened because the compiler is smart enough to recognize that the symbol's version (an internal compiler generated signature that allows the compiler to quickly determine that a symbol has changed), has changed it knows that all other units that refer to that symbol must now be recompiled.

Problems now arise from the fact that if you had a .dcu file that didn't include the original source, there is no way for the compiler to resolve the changes with the users of that symbol. All the compiler has is the final binary representation of that source with all the various offsets and fixups resolved in an intermediate format. This is what causes the symbol version mismatch errors folks may have seen.

All the above also applies to Delphi packages and their corresponding .dcp files as well. .dcp files are simply a concatenation of all the containing .dcu files sans any code blocks since that data is persisted into the .bpl file.

So what can you change that won't break .dcu compatibility? The rules are as follows:

You cannot:

• change the ancestor type of an existing class


• add/remove a field to/from a class or record


• add/remove a new method, virtual, dynamic or otherwise to a class


• add/remove a property decl to a class, including a property redecl


• remove a constant, variable, type, or function from the interface


• change the value of a constant


• change the calling convention of a method or function


• change the visibility of a field, method or property


• change a method to/from virtual or dynamic

You can:

• add a new global constant


• add a new class type


• add a new global variable


• add a new global procedure or function

This list should cover most situations but there may be some cases that don't quite fit the above list, or at least appear to not fit. For instance, if you had unit A; unit B; and unit C;. Now if the "uses" order were C uses B uses C and a breaking change were made to C, that change may cause a corresponding breaking change in B, and so on. If you didn't have the source to B, there'd be no way to resolve that breaking change.

This is exactly the situation that Borland strives to prevent when we publish a patch. We cannot afford to cause this kind of grief for all the third-parties and ultimately their customers that so diligently support Delphi. Were we to be careless when providing fixes and constantly make breaking changes, the pain and frustration level would rise far above the pain we tried to relieve. Kinda like, "So we cured the patient of that deadly disease, however the problem is that he didn't survive the treatment!"

Now before you skeptics and "flame wolves" pounce and start off with things like, "Hey C++ code doesn't ever have this problem!" Au Contraire! C++ has this same problem in spades. The problem is that you get far less compiler help. Say you had a C++ DLL that exported some classes, or even a .LIB that contained some classes. If you simply replaced the DLL, and thought.. "I only added a private field to my base class," you have just set yourself up for a long debugging session of wondering why some of the fields in your base class are now being overwritten. The same holds true if you had a LIB B for which there were no source (headers don't count here folks), and it linked to a common LIB A. Then LIB A changed the declaration of a class which was used or descended from in LIB B. LIB B will contain instructions, and field offsets based solely on the old version of LIB A. The problem is with this scenario is that the compiler/linker won't tell you that there even is a problem. As long as you didn't change any the method declarations in LIB A, it will link just fine (because the mangled names haven't changed). That is purely insane!

For that other crowd out there that would like to point out that C# isn't afflicted with these problems, I'll have to say for the most part you're right. Since classes in .NET (CLR) are purely late-bound by virtue of the JIT compiler, the runtime execution engine can figure out what the class layout actually is. So you can certainly add virtuals, statics, and fields to an existing class without necessarily breaking existing code... but what if I needed to add a parameter to a method, or I wanted to remove a param? What if that old property or method is obsolete, so I removed it? Same issue folks. Murphy (of Murphy's law) would step in and promptly find that one user that called that changed/removed method and cause their app to break. The interesting thing is that even if that method were referenced in some code, the CLR, and thus the user, would not find out about the problem until the JIT'er actually had to compile the calling method. Microsoft knows all of this and has taken steps to assist the user in solving these issues. The first solution is the late-binding nature of the CLR, the other solutions involve strong names and side-by-side execution, which are out of the scope of this post.

Oh and finally, yes there have been a couple of cases where we did introduce a breaking change in a patch. These situations are met with very internal resistance while the costs and benefits are carefully weighed. These are generally cases where the change was deemed to affect a relatively small number of users, and the likelihood of there being a third-party component that depends on the unit being changed is small. A breaking change to TButton would certainly not likely even be considered. However a change to DDEMan, would certainly be entertained... if only to see who would even notice.

New Title, New Look

I've just updated the title and overall look of this blog. Since Blogger.com has vastly updated their site and features, I figured it was time to do the same. The title change merely reflects the fact that this blog is more about Delphi related things than being a pure vanity thing. Although I do tend to mix in a few personal tidbits now and again.

Michael Swindell now blogging

http://homepages.codegear.com/mswindell

This is a good thing. Welcome aboard Michael!

Files, Units and namespaces.. oh my!

Arbitrary and structural... these are words I would use describe the relationship between how C# contends with files and how Delphi (nay, Turbo/Borland Pascal) contends with them. As I thought more about this it became more clear about how this subtle difference in how Delphi and C# view a physical source file caused some of the misunderstandings, assumptions, and outright disappointments regarding Delphi's handling of namespaces. I may be completely out in left field here... or I may just be stating the obvious... and that may be the issue... duh! that's obvious!

In C#, the notion of a physical source file is almost purely arbitrary and abstract. To the compiler, there are no file boundaries. They carry no structural information to the compiler. You have to tell the compiler about every source file you intend to build into the assembly. That list of files also must be referentially consistent, or every type reference must be resolvable either by the list of source files or by external explicit assembly references. You see, that's a key difference between Delphi and C#'s view of files, structure. In Delphi, a physical file is a language construct in and of itself. It is the boundaries of the actual file that define the "namespace." This is why Delphi can keep .dcu/.dcuil files in order to decrease compile and link times. It can easily tell if a .dcu/.dcuil (which are essentially a raw dump of the compiler's internal symbol tables) is out of date by checking both file dates and symbol versions. It is also this same file-based structure that gives serves to seemingly limit the programmer's ability to define exactly how a set of components or classes appear when they are exported from an assembly.

What about Java? Java is actually closer to the Delphi way than the C# way when dealing with files. To Java, the physical file is also a language construct. You can't have more than one class in a .java or ultimately a .class file, except inner classes, of course. To Java, the folder hierarchy is also a language construct. It serves as the definition of a namespace. Java then goes and throws in another twist... the imports clause can contain the '*' character to tell the compiler that you want to reference all classes from a given namespace (or folder hierarchy) without explicitly specifying every class you intend to reference. This allows you to add classes into a specific namespace and have those new classes appear without updating the dependent source files' imports clauses.

So, what does this all lead up to? Nothing really. You don't even have to think this is particularly insightful. This was just a random thought... You can also see that this is going to be a very tough nut to crack. Delphi's notion of files=units=structure is fundamental to the overall character of the Delphi language. Maybe we should begin to look at other languages' playbooks for ways of dealing with namespaces.

One Bacon Double Research, Hold the Labels

One Bacon Double Research, Hold the Labels

Danny has posted this great piece on why we blog. It's worth a read... Don't worry, I'm sure there are plenty of other "Hornet's nest" out their for either one or both of us to go kicking...

Delphi 7.1 is now available

Whew! I was beginning to worry that I had lied when I posted that the Delphi 7 update would be available this week... Well, it is now available. Here's the BDN article with the appropriate links.