But the most startling information in the state's 13-page report (.pdf) is not about why the system lost votes, which Threat Level previously covered in detail, but that some versions of Diebold's vote tabulation system, known as the Global Election Management System (GEMS), include a button that allows someone to delete audit logs from the system.
Auditing logs are required under the federal voting system guidelines, which are used to test and qualify voting systems for use in elections. The logs record changes and other events that occur on voting systems to ensure the integrity of elections and help determine what occurred in a system when something goes wrong.
"Deleting a log is something that you would only do in de-commissioning a system you're no longer using or perhaps in a testing scenario," says Princeton University computer scientist Ed Felten, who has studied voting systems extensively. "But in normal operation, the log should always be kept."
Yet the Diebold system in Humboldt County, which uses version 1.18.19 of GEMS, has a button labeled "clear," that "permits deletion of certain audit logs that contain – or should contain – records that would be essential to reconstruct operator actions during the vote tallying process," according to the California report.
The button is positioned next to the "print" and "save as" buttons (see image above), making it easy for an election official to click on it by mistake and erase crucial logs.
In fact, the report says, this occurred recently in a California county when an official, while attempting to print out a copy of a so-called "poster log," inadvertently deleted it instead.
The system provides no warning to the operator that clicking on the button will result in permanent deletion of records in the log, nor does it require the operator to confirm the action before executing it.
Apparently Premier/Diebold was aware that having a "clear" button on its system was a bad idea. According to California's report, one of the system's developers wrote in an e-mail in 2001: “[a]dding a 'clear' button is easy, but there are too many reasons why doing that is a bad idea.” Yet the company included the button in its system anyway.
The button was removed from version 1.18.20 of the software and following, but Premier/Diebold never went back to jurisdictions using previous versions to upgrade them, and version 1.18.19 is still used in three California counties as well as in other states. It's unclear how many previous versions of the software had the button, or why it was included in the first place.
According to the report:

The “Clear” buttons . . . allow inadvertent or malicious destruction of critical audit trail records in all GEMS version 1.18.19 jurisdictions, risking the accuracy and integrity of elections conducted using this voting system. Five years after the company recognized the need to remove the “Clear” buttons from the GEMS audit log screens, not only Humboldt, San Luis Obispo and Santa Barbara Counties in California but jurisdictions in other parts of the country, including several counties in Texas and Florida, continue to use GEMS version 1.18.19. . . .

The report states that the inclusion of the button violated the federal voting system standards under which the Premier/Diebold system was qualified to be used in elections. The standards require that voting system software automatically creates and permanently retains electronic audit logs of important system events that occur on the machine.
Premier/Diebold did not respond to a request for comment.
The "clear" button isn't the only problem with the auditing log in the Premier/Diebold system. I wrote previously about other issues with the logs -- for example, they don't record significant events that occur in the tabulation system, such as when someone deletes votes from the software.
The California report states that the "clear" button, along with other problems with the auditing logs as well as the software flaw that caused the system to lose votes in Humboldt County (see below for more information on that flaw), should have been red flags to the testing laboratories that certified the system and should have been sufficient to "fail" the system and prevent it from being used in any federal election.
As the report points out, under the voting system standards (VSS) "each of the errors and deficiencies in the GEMS version 1.18.19 software described in this report standing alone would warrant a finding by an Independent Testing Authority (ITA) of 'Total Failure' (indicated by a score of 1.0) had the flaw been detected. Under the 1990 VSS, a finding of 'Total Failure' required failure of the voting system."
"Presumably some organization, some lab, looked at this system and decided they thought it complies with the standard," says Felten. "And, obviously, they were wrong. Any state that uses GEMS should be looking at this seriously."
The findings raise questions about the auditing logs on voting systems made by other vendors and about what states that use the Premier/Diebold system will do now that they know their voting software does not create an adequate audit trail to ensure the integrity of an election.