Analysis of the 2002 VSS
Alan Goldfine
Final Draft, April 6, 2005
1Introduction
This document presents the results of an analysis of the requirements contained in the "Voting Systems Performance and Test Standards" ("VSS"), released by the Federal Election Commission in 2002. The analysis was performed by the NIST voting team in support of the Election Assistance Commission’s Technical Development Guideline Committee (TGDC) resolution 25-05. The objectives of the analysis were to:
•thoroughly review the VSS to determine which parts of the VSS can be extracted or recast into a new voting system standard, called the Voluntary Voting System Guidelines (VVSG)
•identify which parts of the VSS require substantial changes or replacement
•identify the VSS requirements that are and are not testable
•help identify omissions in the VSS that have resulted from changes in technology or other reasons.
This analysis will be a starting point for the development of the VVSG version 2.
2Approach
The VSS comprises two volumes: "Volume I, Voting System Performance Standards" and "Volume II, Voting System Test Standards." The normative text in each volume (sections 2 through 9 of Volume I and sections 2 through 7 of Volume II) was carefully examined and the individual requirements identified. Each requirement was then evaluated as to its disposition in the VVSG: whether it should be retained as is, revised, deleted, or moved. This information was then placed in Table 1, corresponding to Volume I, or Table 2, corresponding to Volume II, as appropriate. Each requirement in the tables was assigned to the TGDC subgroup that was considered to be the subgroup most responsible for that requirement. Each requirement was also associated with the Organizing Principle that was considered to be the one most closely supported by that requirement. Finally, any observations and open questions were recorded.
3Summary of the Analysis
Much of the VSS is valid and can be retained in the VVSG, either as-is or rewritten to be more precise. Most of the current functional and hardware requirements can stay largely intact, but some need to be made more precise and testable, and others need to be updated to correspond to current technology. The principal areas that require more major changes are the VSS requirements dealing with accessibility and usability, security, software design and coding standards, configuration management, and quality assurance. More specifically:
•The sections in the VSS on accessibility (Vol. I, section 2.2.7) and human engineering (Vol. I, section 3.4.9), and the guidelines on usability (Vol. I, Appendix C), need to be replaced in the VVSG. The text in the VSS is little more than a placeholder.
•Requirements dealing with voter verifiable paper audit trails, wireless communications, software distribution, and direct and indirect vote verification need to be added. The VSS sections dealing with protection against external threats need to be rewritten to better address electronic voting systems.
•The VSS sections on software design and coding standards (primarily Vol. I, section 4.2 and Vol. II, section 5.4) are obsolete and need to be replaced, per resolution 29-05.
•The VSS sections on configuration management and quality assurance (primarily Vol. I, sections 7 and 8, and Vol. II, section 7) need to be replaced, per resolution 30-05.
4The Analysis Tables
The Analysis Tables, namely Table 1 corresponding to VSS Volume I, and Table 2, corresponding to VSS Volume II, are presented in the Appendix.
4.1How to Read the Tables
Each table consists of 6 columns. In a given row,
•Column 1 is the number of that row.
•Column 2 ("G") is a letter representing the TGDC subgroup with primary responsibility for the requirement contained in that row. Some requirements are assigned to more than one subgroup. The letters are:
–C Core Requirements and Testing
–H Human Factors and Privacy
–S Security and Transparency
•Column 3 ("Section/Requirement") contains either a section heading in the VSS, or a requirement identified in and extracted from that section. The requirements are essentially those statements in the VSS specifying that some entity "shall" do something.
•Column 4 ("T") is a letter representing the type of disposition proposed for the given requirement. The letters are:
–E The requirement is satisfactory as currently written in the VSS, and can be extracted and retained as-is. There is no need to rewrite it.
–R The requirement is not precise, clear, or testable, or contains a typo, bad reference, or other mistake. It needs to be rewritten.
–M In Table 1: The requirement is not a performance requirement, but rather a testing requirement. It should be moved to Volume II of the VSS.
In Table 2: The requirement is not a testing requirement, but rather a performance requirement. It should be moved to Volume I of the VSS.
–D The requirement is obsolete, redundant, or otherwise unnecessary. It should be deleted.
Disposition is not proposed for requirements in those VSS sections that will be entirely rewritten. When this situation occurs, it is noted in column 6.
•Column 5 ("P") is the number of the organizing principle (reference) which the given requirement supports. Some requirements support more than one organizing principle.
•Column 6 ("Comments") contains questions, answers to previously submitted questions, and other observations about the given requirement. For many requirements, there is an indication as to whether or not the requirement is testable. Questions are highlighted in bold green.
4.2What We Found
Table 1
•There are 541 identified requirements for which disposition is proposed.
•The assignment of identified requirements to the TGDC subgroups is as follows:
–394 requirements assigned to the CRT subgroup
–72 requirements assigned to the HFP subgroup
–126 requirements assigned to the STR subgroup.
(Some requirements are assigned to more than one subgroup.)
•The disposition of requirements is as follows:
–310 requirements proposed to be extracted
–169 requirements proposed to be rewritten
–32 requirements proposed to be moved to Volume II
–30 requirements proposed to be deleted.
•The association of requirements to organizing principles is as follows:
–8 requirements associated with Principle 1
–32 requirements associated with Principle 2
–71 requirements associated with Principle 3
–57 requirements associated with Principle 4
–44 requirements associated with Principle 5
–378 requirements associated with Principle 6
–112 requirements associated with Principle 7
–41 requirements associated with Principle 8.
(Some requirements are associated with more than one organizing principle.)
Table 2
•There are 344 identified requirements for which disposition is proposed.
•The assignment of identified requirements to the TGDC subgroups is as follows:
–299 requirements assigned to the CRT subgroup
–31 requirements assigned to the HFP subgroup
–67 requirements assigned to the STR subgroup.
(Some requirements are assigned to more than one subgroup.)
•The disposition of requirements is as follows:
–282 requirements proposed to be extracted
–59 requirements proposed to be rewritten
–0 requirements proposed to be moved to Volume I
–3 requirements proposed to be deleted.
•The association of requirements to organizing principles is as follows:
–6 requirements associated with Principle 1
–3 requirements associated with Principle 2
–2 requirements associated with Principle 3
–10 requirements associated with Principle 4
–0 requirements associated with Principle 5
–323 requirements associated with Principle 6
–51 requirements associated with Principle 7
–1 requirement associated with Principle 8.
(Some requirements are associated with more than one organizing principle.)
-1-
Appendix
Table 1
G / Section/Requirement / T / P / Comments2. Functional Capabilities: Overall system capabilities
2.2.1 Security: all systems shall
S / a) provide security access controls that limit or detect access to critical system components to guard against loss of system integrity, availability, confidentiality, and accountability / R / 7 / High level requirement. Identify or describe what makes a "critical system component"
C / b) Provide system functions that are executable only in the intended manner and order, and only under the intended conditions / R / 6 / Needs precision: specify what is meant by "intended manner and order" and "intended conditions". These terms must be defined for each relevant system function for this requirement to be testable
C / c) use the system’s control logic to prevent a system function from executing if any preconditions to the functions have not been met / R / 6 / Need to specify preconditions for each relevant function for requirement to be testable
C / d) provide safeguards to protect against tampering during system repair, or interventions in system operations, in response to system failure / R / 6 / If this refers to equipment, then need further definition of "safeguards"
S / e) provide security provisions that are compatible with the procedures and administrative tasks involved in equipment preparation, testing, and operation / D / 6 / Not testable, too vague
S / f) if access to a system function is to be restricted or controlled, the system shall incorporate a means of implementing this capability / R / 6 / High level requirement. Vague
S / g) provide documentation of mandatory administrative procedures for effective system security / E / 6 / Documentation requirement.
Testable by inspection
2.2.2 Accuracy
2.2.2.1 Common standards, all systems shall
C / a) record the election contests, candidates, and issues exactly as defined by election officials / E / 6
C / b) record the appropriate options for casting and recording votes / E / 6
C / c) record each vote precisely as indicated by the voter and be able to produce an accurate report of all votes cast / R / 4 / Combination of several requirements – break it up.
C / d) include control logic and data processing methods incorporating parity and check sums (or equivalent error detection and correction methods) to demonstrate that the system has been designed for accuracy / R / 6 / Reword to update or eliminate reference to parity and check sums. What does it mean for a "system to be designed for accuracy".
C / e) provide software that monitors the overall quality of data read-write and transfer quality status, checking the number and types of errors that occur in any of the relevant operations on data and how they were corrected. / R / 6 / Need to specify a software interface as some level. Need to specify relevant operations. Testing section needs to further specify number and types of errors.
S / 2.2.2.2 DRE Systems Standards:
Voting devices shall record and retain redundant copies of the original ballot images / R / 68 / Will be rewritten by STS.
2.2.3 Error Recovery
To recover from non-catastrophic failure of a device or from any error or malfunction that is within the operator’s ability to correct, the system shall
C / a) restoration of the device to the operating condition existing immediately prior to the error or failure, without loss or corruption of voting data previously stored in the device / R / 7 / Reword "restoration of" as: "restore".
C / b) resumption of normal operation following the correction of a failure in a memory component, or in a data processing component, including the central processing unit / R / 7 / Reword "resumption of" as: "resume"
C / c) recovery from any other external condition that causes equipment to become inoperable, provided that catastrophic electrical or mechanical damage due to external phenomena has not occurred / R / 7 / Reword "recovery from" as: "recover".
2.2.4 Integrity
2.2.4.1 Common standards: all systems shall
CS / a) protect, by means compatible with these Standards, against a single point of failure that would prevent further voting at the polling place / E / 7
C / b) protect against the interruption of electronic power / E / 7
C / c) protect against generated or induced electromagnetic radiation / E / 7
C / d) protect against ambient temperature and humidity fluctuations / E / 7
C / e) Protect against the failure of any data input or storage device / E / 7
S / f) Protect against any attempt at improper data entry or retrieval / R / 7 / How does system know what is "improper"?
CS / g) Record and report the date and time of normal and abnormal events / R / 78 / Need list of types of normal and abnormal events – then would be testable.
C / h) Maintain a permanent record of all original audit data that cannot be modified or overridden but may be augmented by designated authorized officials in order to adjust for errors or omissions (e.g. during the canvassing process.) / E / 8
C / i) Detect and record every event, including the occurrence of an error condition that the system cannot overcome, and time-dependent or programmed events that occur without the intervention of the voter or a polling place operator; and / R / 8 / “Record” in what manner?
C / j) Include built-in measurement, self-test, and diagnostic software and hardware for detecting and reporting the system's status and degree of operability. / E / 78 / Implementation specific
2.2.4.2 DRE Systems Standards
C / a) Maintain a record of each ballot cast using a process and storage location that differs from the main vote detection, interpretation, processing, and reporting path; and / R / 23 / Reword as: "an accurate record".
How much of the system has to be different for the two processes and storage locations before this requirement is satisfied?
H / b) Provide a capability to retrieve ballot images in a form readable by humans. / R / 38 / Imprecise. Will be rewritten by HFP.
2.2.5 System Audit 2.2.5.1 System Audit Purpose and Context 2.2.5.2 Operational Requirements 2.2.5.2.1 Time, Sequence, and Preservation of Audit Records
C / a) Except where noted, systems shall provide the capability to create and maintain a real-time audit record. This capability records and provides the operator or precinct official with continuous updates on machine status. This information allows effective operator identification of an error condition requiring intervention, and contributes to the reconstruction of election-related events necessary for recounts or litigation. / R / 68 / Delete "provide the capability to". Based on 4.4. Implementation specific.
C / b) All systems shall include a real-time clock as part of the system’s hardware. The system shall maintain an absolute record of the time and date or a record relative to some event whose time and data are known and recorded. / E / 6 / Testable
C / c) All audit record entries shall include the time-and-date stamp. / E / 6 / Testable
C / d) The audit record shall be active whenever the system is in an operating mode. This record shall be available at all times, though it need not be continually visible. / E / 6 / Testable
C / e) The generation of audit record entries shall not be terminated or altered by program control, or by the intervention of any person. The physical security and integrity of the record shall be maintained at all times. / R / 6 / How can this be tested?
C / f) Once the system has been activated for any function, the system shall preserve the contents of the audit record during any interruption of power to the system until processing and data reporting have been completed. / R / 67
C / g) The system shall be capable of printing a copy of the audit record. A separate printer is not required for the audit record, and the record may be produced on the standard system printer if all the following conditions are met: / E / 68 / Printing capability is testable
- C
C / 2)The entries can be identified so as to facilitate their recognition, segregation, and retention / E / 6 / Testable by inspection
C / 3) The audit record entries are kept physically secure. / E / 6 / Requirement seems to be administration.
2.2.5.2.2 Error Messages
C / a) The system shall generate, store, and report to the user all error messages as they occur; / R / 6 / Who is the user?
What error messages are required in the first place?
H / b) All error messages requiring intervention by an operator or precinct official shall be displayed or printed unambiguously in easily understood language text, or by means of other suitable visual indicators / R / 6 / “Unambiguous” is ambiguous…
Testable with users to see if messages are understood.
C / When the system uses numerical error codes for trained technician maintenance or repair, the text corresponding to the code shall be self-contained, or affixed inside the unit device. This is intended to reduce inappropriate reactions to error conditions, and to allow for ready and effective problem correction; / E / 6 / Testable only by inspection.
H / d) All error messages for which correction impacts vote recording or vote processing shall be written in a manner that is understandable to an election official who possesses training on system use and operation, but does not possess technical training on system servicing and repair / E / 6 / Testable with users to see if messages are understood.
H / e) The message cue for all systems shall clearly state the action to be performed in the event that voter or operator response is required / E / 6 / Testable with users to see if messages are understood.
C / f) System design shall ensure that erroneous responses will not lead to irreversible error / D / 6 / Not testable.
C / g) Nested error conditions shall be corrected in a controlled sequence such that system status shall be restored to the initial state existing before the first error occurred. / E / 6
2.2.5.2.3 Status Messages
H / The system shall display and report critical status messages using unambiguous indicators or English language text. The system need not display non-critical status messages at the time of occurrence. Systems may display non-critical status messages (i.e., those that do not require operator intervention) by means of numerical codes for subsequent interpretation and reporting as unambiguous text. / E / 6 / Testable either by inspection or by users to see if messages are understood.
2.2.5.3 COTS General Purpose Computer System Requirements
S / To counter these vulnerabilities, three operating system protections are required on all such systems on which election software is hosted. First, authentication shall be configured on the local terminal (display screen and keyboard) and on all external connection devices (“network cards” and “ports”). This ensures that only authorized and identified users affect the system while election software is running. / R / 6 / To be testable, need more details on the nature of the authentication.
S / Second, operating system audit shall be enabled for all session openings and closings, for all connection openings and closings, for all process executions and terminations, and for the alteration or deletion of any memory or file object. This ensures the accuracy and completeness of election data stored on the system. It also ensures the existence of an audit record of any person or process altering or deleting system data or election data. / R / 6 / What does “operating system audit shall be enabled” mean? If defined, then testable by inspection.
S / Third, the system shall be configured to execute only intended and necessary processes during the execution of election software. The system shall also be configured to halt election software processes upon the termination of any critical system process (such as system audit) during the execution of election software. / R / 6 / Which are the “intended and necessary processes” as opposed to other process?
2.2.6 Election Management System
CS / The Election Management System (EMS) is used to prepare ballots and programs for use in casting and counting votes, and to consolidate, report, and display election results. An EMS shall generate and maintain a database, or one or more interactive databases, that enables election officials or their designees to perform the following functions: / E / 6