November 15, 2007
The parallel election

Okay, this is disturbing.

Johnnie German admitted he was nervous as he used high-security codes to tap into the Harris County elections computer system last week and change some of the results manually.

The system was in good hands as the votes were counted from the sprawling Nov. 6 contests. German is the county's respected administrator of elections, and there were witnesses present as he corrected the vote totals on a sales tax referendum for a fire/ambulance district in the Cypress-Fairbanks area of northwest Harris County.

But German's late-night deed, said by officials to be a first-time event in the six years Harris County has used the eSlate voting system, has rekindled the debate about whether the newest electronic methods for counting votes should be trusted.

What German graphically demonstrated was that with the proper physical and informational access, one person can alter the results of an election in a county of 1.8 million registered voters.

The adjustments also highlighted the fact that, with multiple election boundaries snaking through precincts to separate city voters from county voters and municipal utility districts from emergency services districts, there usually are flaws that put voters in front of the wrong ballot screens.


The omission of the tax proposal on ballots in parts of three precincts was discovered thanks to an alert from a voter, and Harris County Clerk Beverly Kaufman's staff was able to get the tax question on the right ballots for Election Day -- but it was too late to have those votes recorded on the main computer.

Instead, they were recorded separately and later added to the totals.


The county Web site already showed that all precinct totals had been counted; three sheriff's deputies who guarded the counting process on the fourth floor of the County Administration Building in downtown Houston had been sent home.

Also in the locked, glass-walled room were Republican Kaufman and John R. Behrman, a computer expert and longtime election observer representing the Democratic Party. He said he considers Kaufman's staff the most knowledgeable election computer administrators on the continent and does not question their motives.

But Behrman said he was shocked when he saw German use a series of passwords and an "encryption key" -- a series of numbers on a nail file-size computer memory storage device -- to reach a computer program that said "Adjustment."

"A hundred percent of precincts reporting, and everything had been distributed to the press," he said. "Then and only then did I see how they were going to do this, and frankly I never thought it was possible.

"Basically it turns out, without regard to any ballots that have been cast, you can enter arbitrary numbers in there and report them out in such a way that, unless you go back to these giant (computer) logs and interpret the logs, you wouldn't know it has been done."

The fact that this is possible isn't actually shocking to me. Pretty much any distributed computer system is going to have an Administrator account on it, which has the rights to do anything it wants to do. You can't install or configure software without that level of access. Frankly, it would have been shocking - and inexcusable - if it hadn't been possible to do this sort of thing, to handle the event of an incorrect ballot.

The question then becomes one of procedure - how do you ensure that the person or persons who have access to the Admin account (and other aspects of the system) don't abuse their position? That's something Dan Wallach harped on when I interviewed him about voting machine security, and it remains a big deal. Wallach brings up other issues with the way these corrections were made as well, though you really can't tell that from the article:

Computer scientist Daniel Wallach, who started Rice University's Computer Security Lab and was on the task force that recently studied California's electronic voting systems, is skeptical about the eSlate system supplied to Harris County at a cost of $12 million by Austin-based Hart InterCivic.

The "encryption key" code could be extracted from voting equipment at each precinct, according to Wallach, who studied the company's systems in California.

County officials and Hart InterCivic, which also provides its state-certified voting equipment in Fort Bend County and Austin and Fort Worth, said the system merits public confidence because it has multiple layers of secret access codes.

I emailed Dan to ask him for a comment. This is what he sent me:

Neither their quote from me nor their responses from the county officials really explains enough for the paper's readers to understand what is actually going on here. For readers who want the gory details, I would recommend they read California's "top to bottom" analysis of the Hart InterCivic voting system. I co-authored the source code analysis of Hart's systems. The relevant section that describes the "vote adjustment" feature is on pages 49-50, labeled "Issue 17: The Tally interface allows a Tally administrator to `adjust vote totals.' This can create inconsistencies in the reported vote totals."

To summarize, Hart's tabulation system, "Tally" supports a feature that allows an election administrator (i.e., somebody who knows the special administrator password, has the appropriate USB key token, and has access to the Tally machine) to make pretty much arbitrary changes to the election totals. This functionality operates by directly editing the totals, which goes entirely against standard bookkeeping practices (where you never, ever overwrite a number in the books; you instead add a line to the books that states what the correction is and where the error occurred). Hart's basic design allows for innocent mistakes to go uncorrected, since there is no easy way to audit any corrections that may have been made. Corrections do not show up on official election reports.

As a secondary matter, the security features, intended to prevent unauthorized users from accessing this feature, are similarly inadequate. The password necessary to interact with the database is stored on the disk where any user of the machine can easily access it (see our report, pages 48-49, "Issue 15: Database passwords are stored insecurely"). Similarly, the USB tokens, used to manage cryptographic keys, turn out to all contain precisely the same key, which is used throughout the county. The very same key is stored inside machines in every precinct and can be easily extracted (see our report, pages 55-57, Section 6.7, "Cryptographic Key Management").

So, indeed, Hart has multiple lines of defense. Unfortunately, every one of them is incorrectly engineered, rendering the system entirely vulnerable to compromise. Of course, I am not stating that any such compromise has ever happened in Harris County. What I am saying is that the design of the Hart system is entirely insufficient to prevent such attacks, should a competent attacker wish to make them.

I've said before that I think the only way we're going to see a change in attitude towards our inadequate voting machines is if we suffer a catastrophic failure. What happened last week wasn't catastrophic, but it should serve as a wakeup call anyway. I hope we pay heed to what we've been told.

Posted by Charles Kuffner on November 15, 2007 to Election 2008

I have been harping on this for years. The weakest part of the voting system is at the county tabulator levels where anyone with the proper code and access can make changes to vote totals that are not detectable or reversible.

Do you want the security of your ballot to be the trustworthiness of partisan county officials?

Something that hasn't been mentioned here is that the numbers are not stored encrypted. If you have access to a database program on the machine you can bypass the vote counting program entirely and just change the vote totals without no record of anyone entering the voting software.

Posted by: Gary Denton on November 15, 2007 12:14 PM

Thanks for your interview with Dan Wallach and for posting his analysis of design/accounting flaws in our current e-slate system.
I appreciate the link to California's "top to bottom" analysis and the work that Dan Wallach and others are doing to analyze electronic voting systems to assure their integrity.
Does Dan Wallach and his fellow researchers believe that, if their recommendations are followed, e-slate voting can be used with confidence or do these specialists recommend another choice?

Posted by: Evelyn Chorush on November 15, 2007 1:10 PM

Our California report does not make recommendations on how Hart systems should or should not be used. It points out the problems, and then the California Secretary of State imposed a variety of conditions on the use of Hart systems. Those conditions are also likely to change and evolve.

If Texas were to adopt all of the conditions of how voting systems are used in California (including parallel testing, mandatory paper trails, mandatory audits of the paper trails, limits on the number of DREs per precinct with most voters casting optical scan paper ballots, and so forth), that would be a great start. Current procedures in Texas are far behind the California standards and, in my own opinion, are unacceptably error-prone and insecure.

Posted by: Dan Wallach on November 15, 2007 9:36 PM

Can someone "outside" hack "inside" and make these same adjustments?

It is scary to realize, whether they're inside or outside, how easily the totals can be manipulated. Particularly if indeed there is no record of someone having done so.

Posted by: Baby Snooks on November 16, 2007 9:27 AM

The answer is "maybe". Harris County uses a system called "Rally" in combination with "Tally". The broad idea is that they have regional processing centers, running Rally, which are connected via modem to the central tabulation center, running Tally. This use of the phones is a potential vulnerability that could allow an "outside" attacker.

Separately, I have no knowledge of how the County's network is configured. Proper procedure would say there is no network path from the outside world to the tabulation system (sometimes called an "airgap"). If the County had the tabulation system connected via a firewall or something, that would violate the requirement and could represent a source of attack.

Posted by: Dan Wallach on November 16, 2007 6:01 PM

Don't we need to do better to prevent climate collapse?

Election Defense Alliance
Nov. 17, 2006

Landslide Denied!
Major Miscount in 2006 Election: Were 4% of Votes "Misplaced"?

Read the Full Press Release
Read the Report
Study the Exit Poll Data

Election Defense Alliance, a national election integrity organization, issued an urgent call today for an investigation into the 2006 election results and a moratorium on deployment of all electronic voting equipment after analysis of national exit polling data indicated a major undercount of Democratic votes and an overcount of Republican votes in congressional races across the country. These findings are an alarming indictment of the American election system in which 80% of voters used electronic voting equipment.

As in 2004, the Exit Poll and the reported election results do not add up. But this time there is an objective yardstick in the methodology that establishes the validity of the Exit Poll and exposes the inaccuracy of the election returns. These findings are detailed in a paper published today on the EDA website.

The Edison-Mitofsky media Exit Poll, posted Election Night on, had a sample base of more than 10,000 voters, and showed Democratic House candidates winning over Republicans by an 11.5 percent margin.

The reported vote count showed Democrats winning by a 7.6 margin, 3.9 percent less than the Exit Poll and far outside the poll's +/-1-percent margin of error. This discrepancy entailed at least 3,000,000 votes.

The Exit Poll was then adjusted, by a process known as "forcing," to match reported election vote totals. The final result, posted at 1:00 p.m. November 8, showed Democrats winning by a 7.6 percent margin, exactly mirroring the reported vote totals.

The objective yardstick was the proportion of respondents who indicated they had voted for Bush or Kerry in 2004. The sample in the already weighted Election Night Exit Poll had 47 percent Bush voters and 45 percent Kerry voters, a valid sample given the very conservative assumption that Republicans and Democrats turned out with equal enthusiasm in 2006. However, after the forcing process, the sample contained 49 percent Bush voters and only 43 percent Kerry supporters. This 6 percent gap is more than twice the size of the 2004 Bush win of 2.8 percent. It indicates a significant over-sampling of Republican voters in the adjusted 2006 Exit Poll.

Such a gross oversample of Republicans was necessary to match the actual vote counts, which therefore could not have been an accurate count of the actual electorate. Had the intended votes been accurately tallied, this election would have produced a Democratic landslide of epic proportions.

Read the Full Report: Election Defense Alliance


* The purpose of EDA is to help build and coordinate a comprehensive, cohesive national strategy for the election integrity movement, in order to regain public control of the voting process in the United States, and to ensure that the process is honest, transparent, secure, subject to unambiguous verification, and worthy of the public trust.

* To accomplish this purpose, EDA will provide resources, strategic planning and coordination opportunities for a nationwide network of citizen electoral integrity groups and individuals already working at the national, state, and local levels.


Update: In the following article Holt's HR 550 is now HR 811 and Diebold has since changed the name of its voting equipment company.

Pull the Plug on E-Voting
By Bruce O'Dell

OpEd News
Wednesday 25 October 2006

The FBI is investigating the "possible theft" of the Diebold touch screen voting software in Maryland. Excuse me ... but I fail to see what all the fuss is about. I certainly don't condone theft; it's just that I don't understand why anyone would bother with stealing the Diebold source code - or why anyone would take the time to read it.

Don't get me wrong: I've spent twenty five years in the financial services industry helping to protect billions of dollars of other people's money. I designed internet security services as an employee of American Express to protect the online financial identities of hundreds of thousands of people, and recently spent a year at one of the twenty largest companies in America as chief architect of a project to replace the foundation of all their internal and external security systems. I understand risks from thieves and embezzlers - I've designed financial audit and control systems. In the world I work in, there's no room for excuses.

Source Code Is Irrelevant

I'll let you in on a dirty little secret of the computing profession: in the real world, there's simply no way to ensure that any program alleged to be written by Programmer Bob on June 24th bears any relationship whatsoever to what actually runs on computer "X" thousands of miles away on November 7th. Even if Programmer Bob's corporate public relations and sales reps swear up and down that it must be so.

When it comes to security, source code is irrelevant. The actual behavior of a computer at point of use is the only thing that matters. Yet many of my IT colleagues continue to believe that it is somehow possible to look at a vendor's source code and determine what a particular voting computer will actually do in a precinct or county election office during an election. This seems to be the rationale behind "open source voting": if I can see the program is benevolent, then must be safe to use. Sounds plausible. But in reality any computer academic or professional practitioner who tells you that anyone on earth can determine whether a vote tabulation system is secure and accurate simply by looking at a source code document ... is either ill-informed or lying.

Consider Microsoft's Windows XP operating system. As a critically-important widely-used program nevertheless riddled with bugs and security holes, this is a particularly apt comparison to voting software. Even if I could obtain a copy of the current Windows XP source code and read its millions of lines of text in its entirety with perfect comprehension, the act of reading the program text tells me precisely nothing at all about the integrity and security of any of the hundreds of millions of computers running Windows XP all around the world.

Think about it. Some surveys indicate 70% or more of Windows PCs are infested with viruses, spyware or, worst of all, rootkits. Rootkits hijack precisely those portions of the operating system that are used to detect the presence of malicious software and in so doing so become effectively undetectable. Can looking at the source code version of Windows XP tell me whether your particular PC is echoing all your keystrokes to a server owner by the Russian mob while you're innocently doing your online banking?

Software Is Inherently Untrustworthy ...

How do so many of my colleagues get such a fundamental issue so wrong? Although computer technology can seem endlessly complex, the fundamental issues are simple enough.

Computer program "source code" is just a text document. It's written using a word processor in a highly specialized dialect that is a shorthand mishmash of English words and math symbols. In order to get a computer to do my bidding, I first edit and save a text file, then run other programs (called "compilers" or "interpreters") to convert my human-readable text into the binary electrical impulses that a computer can understand and execute.

Here's where it becomes one twisty hall of mirrors. All means of verifying the version and features of any program as it is running in a computer require use of other software, the version and features of which in turn are verified by use of other software, the version and features of which in turn is verified by other software ... and so on. Software alone can't vouch for software. It is a very well-known maxim in my profession that the only way to truly know what is running in a computer at any given time is to present all the inputs, record all the outputs, and verify that the two match up as expected.

All computer systems which process high-value transactions include audit mechanisms that monitor the advertised features of the system to enable an independent means of detecting flawed or fraudulent program logic ... uh, everywhere that is except for voting systems, which arguably process the most important transactions of all. Go figure.

I'm so tired of hearing e-voting compared to using an Automated Teller Machine. Voting could not be more different than using an ATM. ATMs ask for not one but two forms of identification - a bank card and a PIN. Whereas the act of voting is private and anonymous. "Private, anonymous banking" is just another way to say "robbery in progress" - as in sawing open the ATM and taking its cash. ATMs exchange transaction and audit records with multiple counterparties and offer the user a receipt. Some but not all e-Voting systems may create or scan a paper vote record, but the voter surely can't keep it, or votes could be coerced or sold. e-Voting machines and ATMs are truly "apples and bicycles".

When it comes to electronic voting, we can't use any of the techniques we apply to securing electronic financial transactions all of which are predicated on the strong proofs of identity and exchange of transaction data with multiple counterparties that are rightfully banned in voting systems. Voting systems are national security systems demanding a much higher standard of protection than mere financial systems.

... Yet the Behavior of Voting Software Is Allowed to Go Unaudited

Many voting systems provide only an internal electronic audit trail of electronic vote tallies. What foolishness to allow programs to vouch for programs in such a way; as if it is somehow impossible to make two programs lie consistently!

Rep. Rush Holt's HR 550 legislation and its supporters in the academic computer science community are trying to salvage computerized voting by requiring that e-Voting touch-screen equipment always produce a "voter-verified paper audit trail" (VVPAT). This is a kind of receipt which in theory could be audited sometime after an election if the official results were contested. Setting aside the chain of custody problem - as soon as paper leaves the room, it is potentially compromised - when it comes to observing voters actually verifying their paper audit trail, the results are startling.

A 2005 study by the Caltech-MIT Voting Project concluded the following: " no errors were reported in our post-survey data ... and over 60 percent of participants indicated that they were not sure if the paper trail contained errors." That's right: in test elections full of deliberately engineered VVPAT errors - including swapped votes and even missing races - no one reported a VVPAT error while voting, a majority were unsure wtether there were any errors or not, and almost a third of the participants continued to insist that there no errors at all even after they were told otherwise by those who switched the votes!

But even that subset of touch screen voting systems with some kind of voter-verified paper trail, and optical scan systems that could in theory be audited ... in practice, are not. Certainly not by the standards of the financial services industry.

HR 550 was regarded as something of a revolutionary breakthrough in voting accountability simply by requiring a random audit of 2% of precincts after the fact. Under the Sarbanes-Oxley financial accountability law passed in the wake of the Enron scandal in 2002, the board of directors of any public company foolish enough to apply the same standard of auditability to their own books now have personal criminal liability for their decisions and so would face prison time for approving such a threadbare scheme.

But apparently when it comes to elections, no standard of protection is too lax.

Voting by Computer Considered Harmful

There was a remarkable article published by the Computer Professionals for Social Responsibility in 2001, citing work by the Caltech-MIT Voting Project:

... our best efforts applying computer technology have decreased the accuracy of elections, to the point where the true outcomes of many races are unknowable. Many technologists and technology enthusiasts will read the above words and refuse to believe them. 'There must be some other explanation,' they will say. 'Nothing has been proven,' they will say. 'Future technology will be better,' they will say. But there is no other plausible explanation: new technology may have reduced the cost of elections, and certainly has increased counting speed, but the above results show no statistically significant progress in elections accuracy over people counting paper ballots, one at a time, by hand.

Let me recap: voting by computer may be inherently untrustworthy and in practice poorly crafted, overpriced, prone to breakdowns and wide open to subversion - but at least it's less accurate than counting by hand.

Here's an indictment of the IT profession, and a fine irony: the degree of independent hand-auditing of paper ballot records sufficient to verify the corresponding computerized vote tallies is comparable to the effort required to more accurately count all the ballots by hand in the first place, dispensing with the machines. But until that day arrives, the programs that the voting vendors actually distribute - as opposed to the software they may say they distribute - will continue to determine who takes power after the votes are tallied.

To add further:
(From BBV):

Love this quote from Avi Rubin, about what we should do when we dump all these security-defective voting machines:

I recommended to them [state officials] that they give these thirty, forty thousand machines that they have to the schools, attach a mouse and a keyboard, they're Windows machines, let the kids use them, said Avi Rubin, who votes in Maryland. Or give them to a country whose government we want to control.

Posted by: Prove Our Democracy with Paper Ballots on November 17, 2007 3:33 AM