Charlie clarifies

Here's my response to Charlie's response that I posted as a comment:

Charlie,
As always, your wordsmithing abilities have far outstripped my feeble attempts at articulating my thoughts. Upon re-examination I can certainly see where my interpretation may have come from. In the third paragraph you mentioned how the election officials were told by the manufacturer that their machines could tally 10,000 votes. I can see how the officials probably looked at this and thought.. "we only have about 8,000 registered voters in our precinct, so 10,000 should be plenty." However when the machine was delivered, it could only tally 3,005 votes. In light of that, the last sentence seemed to convey, to me at least, that he manufacturer *intentionally* delivered the less capable machine. I was only scoffing at the appearance that your statements seemed to place some level of hubris on the part of the manufacturer. I understand now that you were merely highlighting the incompetence of the folks involved.

Now, as far as open-source being the "solution" to this problem, let me toss in a few more kinks. I don't think that this is only a software issue. I think that it comes down to a hardware *and* a software issue. For instance, in this case, suppose *all* voting machines from this manufacturer contain the same exact core software (or firmware). Then, based on what physical hardware is installed, the system knows how many votes it can store. What if the manufacturer simply grabbed the wrong machine (ie. one without the correct number of flash memory devices) from the assembly-line and shipped it out. Incompitence one. Now when the machine was delivered to the voting authority, only a few test votes were cast to make sure the system functioned. Also nobody double-checked that the machine "sub-model" was in fact the correct one.

Most of these ideas about how all the various ways in which the machine could have failed have little to do with the software. In fact, prior to working for Borland, I used to design, build, and program magnetic stripe encoding and access control equipment. In order to save significant manufacturing costs, the embedded software (which I wrote in all 6800 assembly) contained *all* features that the customer could possibly order. Also, they could order varying levels of card storage and "store-and-forward" buffer. By simply using a single master ROM, we'd burn several hundred copies. By default all but only the core functionality was enabled. By placing the chips into a specialized EPROM burner, the tech could burn certain memory addresses in order to enable all the extended functionality. If the customer ordered more than a few units all with the same features, then a new temp master is created and then it is off to the gang programmer and the custom ROMs are burned.

I would imagine that most of these voting machines are in fact embedded systems, that in order to keep the costs way down, they are using much older technology. In fact they are probably not running any version of Windows, Linux, or whatever... Since they are single purpose systems, you can do a whole lot to cut costs, increase manufacturing efficiencies, all without compromising the core functionality of the machine.

Yes, there is room for programmer error. There is also room for manufacturing defects. And finally there is room for configuration errors. The latter two items have little to nothing to do with the embedded firmware. Also, since these hardware platforms are themselves proprietary, unless you have a full understanding of the environment under which the actual voting code is running, having the code may simply not be enough. About all you can hope to get from being open source is simple code reviews and find some of the more blatant errors. Most of which should be caught by simple in-house peer reviews and fundamental QA testing.

Finally, your ideas for making things verifiable and trackable are pretty good. However I'd take it a step further. There should be hardware keys that are burned into the actual device that uniquely identifies that machine. These keys should be similar to those funky paralell or USB port dongles that folks tend to loath. What is interesting about these devices is that they contain some "write-only" memory. Huh? What good is "write-only" memory? Well what you do is write to this memory a private key so that when you send data through the device (or onboard chip), it encrypts this information with this key. Since *only* that machine has that particular private key *and* there is no way ever retrieve it without destroying the device itself, you can be certain that a particular vote is from a valid machine. This is because the public key is well known and available to anyone. So basically as long as you can decrypt the data with specific public keys you know that the vote could have only come from a valid machine. Not even the manufacturer or the election officials should ever have this private key. That is just off the top of my head...

I better stop, or I'll start being accused of being a too "wordy." ;-)...

Open source and e-voting.

Charlie Calvert has an interesing bit on CodeFez about electronic voting machine failure.  I haven't really formed any opinion regarding all the bruhaha surrounding proprietary vs. open source e-voting solutions.  I think both sides have very valid arguments.  However, I do take some issue with the assumptions that Charlie has taken in this editorial piece.  I had to read the following quote several times to believe that it had just been said.  Especially that last sentence!

 What do they mean the machine could only handle 3,005 votes? In this day of 32 bit operating systems, where the standard limit for an Integer value is over 2 billion, exactly how did they manage to create a limit of 3,005 votes? A failure on this magnitude takes real work to achieve! It is something only a proprietary software company, intentionally trying to cripple their software, would be likely to achieve. [emphasis mine]

Wow.  Unless I totally misinterpreted this, it certainly looks like Charlie has flat accused the voting machine vendor of intentional voter fraud!  There are way too many variables to simply make that kind of judgement.  For instance, according to Charlie, the vendor stated that the machine “had the capacity to record 10,000 votes.”  What if that machine could be configured in various ways to store more or less verification data with each vote? Each different configuration would affect the total vote storage capacity.  Voting machines are more than simple counters that accumulate a tally.  They have to store transactional data, timestamps, and other bits of verification data (obviously short of associating a particular voter with a specific vote!).  I would imagine that each precinct would be able to dial up or down the level of verification data stored depending upong their state or precinct's rules regarding election verification. 

I would be more quick to pin this problem on the sales and support teams, rather than the programmers!  Either the salesman didn't properly convey that as they dial up the verification data, the total vote storage capacity will decrease or the folks charged with setting up the machines didn't RTFM!  Open source would not fix that problem one bit! 

What about the software that runs the Space Shuttle?  Should that be “Open Sourced“ as well.  It would have made little difference to the Columbia crew.  In fact, all the reports I read or heard, talk about how the Shuttle kept correcting the yaw introduced by the extra drag created on by that gaping hole in the RCC panel.  Even to the point of firing attitude thrusters.  The software that runs the flight control systems has been through rigorous testing and performed flawlessly.  It is proprietary.  Sure, you can argue that that software is running on mission critical systems.  What about the software that runs the resporation machine that is helping keep your relative alive while the doctors are performing a triple bypass?  Yep.  Proprietary.  Sure, no-one died when the voting machines failed, but it *does* attack the very core foundation of what this country was built on.  I just don't see how open-source would have been the “magic-bullet“ to solve all these problems?  You can apply that same argument to all the other cases where software is a cricital component, but I don't see an outcry from the “open-source“ proponents to have GE Medical Systems open source their defibrillator firmware.  I admit that is a bit of a hyperbole, but I just want to point out that closed-source systems do work and do provide significant value to our society.

Regarding Charlie's statement about intentionally crippling the software, I have to wonder what that company's motivation would be?  Have criminal charges been filed against the voting machine company?  A company is in business to make money, not make a few quick bucks by defrauding the voters in some North Carolina county, then go to jail for voting fraud.  Some grand conspiricy is a little far-fetched.  Almost to the level of Roswell cover-ups and alien autopsies. 

According to all the articles I've read regarding the machine failure, I find nothing about the failure being software or hardware.  They simply state that it was a “voter machine failure.“  It very well could have been a bad bank of flash memory where the software thought it was writing the proper tracking data, but it just flew out the bit-bucket.  Sure, the software should properly verifiy that it was writing the data correctly, and if an error is detected it should block all futher voting and alert the poling place staff.  I'd be interested in seeing a reference to some article that outlines the specifics of the machine failure.  I couldn't find any in Charlie's piece.

Finally, I like Charlie.  I have a lot of respect for him.  He's certainly a better writer than I'll ever be.  But, we don't have to agree on everything ;-)...  Besides, it appears that the courts have finally decided the race.

 

Danny is now finally on blogs.borland.com

Yep... He's finally joined all us crazies over here on blogs.borland.com.  You can get all the good stuff here from now on.

Free lunches, memory, and Moore's law..

I've been reading a lot of the various comments and observing the overall gasps and guffaws surrounding Herb Sutter's DDJ and C/C++ User's Journal articles regarding the end of the line for Moore's Law.  Granted, this is actually the first time in nearly 30 years that processor speeds haven't grown at the “normal“ meteoric rate.  However, I don't think this should be cause for immediate concern.  In fact this should be seen as a good thing.  As the old proverb goes, “neccessity is the mother of invention,” so too do I think that this may be a good time for the PC vendors to focus on other aspects of the whole system.  Rather than focusing on the core processor for all your speed gains, they should turn their attention to the memory bus.  This is the number one bottle-neck of a system.  What if all the memory (RAM that is..) ran on the same clock as the core CPU?  What if a new value from the memory could be fetched in a single clock cycle?  Sure, there are small amounts of memory that do use the internal CPU clock, the level 1 cache.  But those are just cheats and tricks.  In practice, the cache does a fairly good job of keeping the CPU monster fed with new instructions and data to push around, but as soon as the level 1 cache became exhausted, they added a cache for the cache, the level 2 cache. This cache is usually 2-4 times the size of the level 1 cache and runs approximately that much slower.  Some systems even add a level 3 cache.  What's next?  A level 4 cache... oh wait.. that's the main system memory ;-)..

Now there are some new architectures that are being used.  For instance, Non-Uniform Memory Access or NUMA, is an interesting technique for multiple CPU systems to reduce cross CPU contention.  By giving each CPU its own chunk of system memory that only it can access, each CPU can now run without too much worry that the other CPU may be accessing the memory at the same time.  Mainly because the CPUs have to negotiate with each other in order to get access to the memory that they don't control.

In a nutshell that is the state of the hardware.  What about the software that runs on these systems?  I think Julian Bucknall has covered that detail quite well.  Basically, it would be a good idea to get used to writing code to take advantage of multi-processor architectures.  This will present a whole new raft of problems, for sure, but there are a multitude of techniques for solving these problems.

We, on the Delphi team, are looking into various things we can do to help the users write better code for these architectures.  Everything from RTL/VCL support to language enhancements are all on the plate for us to look into.  Again, “neccessity is the mother of invention,” and this current stalling of Moore's Law may just be the catalyst that is needed for the software tools industry to step in and lend a hand.  This is an opportunity.

Happy 2005!

That's the year 2005.. not Delphi 2005..  Well... Borland is slowly coming back online after the Christmas (or however you wish to refer it) Holidays... as am I ;-)  We had miserable weather throughout most of the break.  It rained pretty much the whole time.  Of course it is the “rainy season” here in San Francisco/Monterey Bay area.  Other than that, we had a very relaxing time...  Didn't interact with any extended family since we ended up staying local.  This meant that we didn't have to rush from house to house on Christmas day.  No schedules to maintain in order to spend equal time with all the various family factions :-)...

Now if only my brain will come back online...