UPA 2001 Conference

Workshop 6: Exploring Measurement and Evaluation Methods for Accessibility

Jim Kindrick – principle member of technical staff, ERIM

Position Paper

Evaluation or design work including experience with any special user populations.

My background in evaluation and design has not been in the area of usability, but rather in the evaluation and testing of information models and protocols for interoperability between different IT systems that need to exchange and/or share information. This type of testing has been standards based – that is to say the evaluation of an implementation of a data model or exchange protocol is based on an external, standard definition of the model or protocol.

Current project work is focused on the Accessibility Forum, an ongoing collaboration among stakeholders affected by Section 508 including user, industry, government, and other communities in order to benefit employees and members of the public with disabilities. The Forum will identify, prioritize, and conduct projects that assist government in making informed decisions about Section 508 related to procurement, and allow government, industry, and users to communicate and highlight areas where further effort is needed

As a part of this work, we have been involved in a detailed technical review of the Access Board Standard for Section 508. One goal of this effort is to understand how standards-based evaluation and testing methodologies may be applied to evaluation of the Access Board Standard for Section 508. Early results from this review indicate a general partitioning of the technical provisions of the Standard into 3 broad classes of requirements:

-User Interface and HCI – generally focused on the GUI, issues relating to visual/auditory/tactile displays (output) and navigation/commands/forms (input).

-Information Content – primarily web-based content, focusing on syntax, structure, and some semantic content issues independent of the rendering of that content.

-Interoperability – issues related to the interface between E&IT products and the Assistive Technology (AT) products designed to present alternate user interface elements for special user populations, primarily user with visual, auditory, and/or mobility disabilities.

Interoperability related issues seem most likely to be addressable by more traditional evaluation and testing techniques drawn from standards based protocol testing.

Information content related issues are being addressed in other forums, most notably the W3C Web Accessibility Initiative (WAI). Testing and evaluation tools and techniques, largely based on inspection, are available from this and other web communities.

User Interface and HCI issues are most directly related to traditional methods of usability evaluation. These methods include development of standard tasks and standard performance measures tailored for the more specific requirements of accessibility, as a subset of overall usability.

Short description of at least 3 issues encountered (or may encounter) in doing evaluations.

  1. Difficulty in determining specific metrics that are not prescriptive with respect to product features but allow for innovation in product design to develop new features that meet the requirement for accessibility.
  2. Difficulty in specifying quantitative rather than qualitative metrics to enable repeatable testing for accessibility of electronic and information technology.
  3. Difficulty in providing assessment techniques and tools that evaluate cross-disability accessibility.