On Expert-Based Interface Evaluation of Web Resources Regarding Accessibility Issues

On Expert-Based Interface Evaluation of Web Resources Regarding Accessibility Issues: A Preliminary Investigation

Karoulis Athanasis
Dept. of Informatics – Aristotle Univ. of Thessaloniki – Greece
/

Polyxenidou Anastasia

Dept. of Informatics – Aristotle Univ. of Thessaloniki – Greece
/

Pombortsis Andreas

Dept. of Informatics – Aristotle Univ. of Thessaloniki – Greece

Abstract

In this study we investigate the expert-based interface evaluation methodology, in the context of web-based resources regarding senior users or users with disabilities. An expert in this context could be a domain expert, a system expert or an evaluation expert, yet a multidisciplinary team of experts is always of advantage. Although there is some research on assessing the accessibility of web sites with common evaluation methodologies, such as automatic evaluation tools and user-based (empirical) evaluations, there is almost no research on the evaluation of web resources utilizing expert-based approaches. We investigate the adaptation of the Cognitive Walkthrough and the Heuristic Evaluation approaches to evaluate web resources concerning their accessibility and propose some modifications in order to achieve this.

1 Introduction

According to Preece et al., (1994), evaluation is concerned with gathering data about the usability of a design or product by a specified group of users for a particular activity within a specified environment or work context. Regarding people with special needs, this definition implies a variety of aspects to be considered as well. The specified user group in our case is users with special needs (Zaphiris et al., 2001). Older users or users with disabilities can be considered as users with special needs. Some disabilities that may affect the use of web resources are visual or cognitive impairment or a functional limitation, such as blindness or low vision, hard of hearing or a mobility limitation (Michigan State University, 2003). The main issue to evaluate a web resource in this context is its usability and accessibility. There are a number of studies (eg. Zaphiris, 2000; 2001; Zaphiris et al., 2001) on the evaluation of web resources concerning their accessibility by utilising automatic evaluation tools, or common user-based (empirical) evaluations (Kurniawan & Zaphiris, 2001; Kurniawan et al., 2001). However, both approaches provide some limitations, discussed later, which limit their effectiveness. So the emerging question is whether expert-based methodologies could provide a valuable alternative in this direction. One must keep in mind that expert-based approaches are mainly designed as user interface evaluation methodologies aiding primarily to assess the usability of an interface. So, an adaptation to the particular context is necessary, in order for them to be effective. This is the main question of this investigation, namely the adaptation of the expert-based approaches to a domain, not yet studied.

2Background

According to the GVU 7th WWW User survey hosted by the GVU Center, Georgia Institute of Technology (GVU, 1997), speed is the number one complaint of Web users. To be specific, 76.55% of users that participated in the online survey stated that slow web sites were a major obstacle in their online experiences. This problem affects all web users, however senior users and users with disabilities experience additional problems. Zaphiris (2001) provides adequate research findings proving that as the depth increases, response time, error rate and perceived complexity increase as well, while performance and subjective satisfaction diminish. On the other hand, the navigation problem (i.e. getting lost or using an inefficient pathway to the goal) is getting more important for elderly people.

So, we can infer that the evaluation strategy to assess all these design approaches is important. A commonly used evaluation approach is to utilize automatic evaluation tools, such as Bobby ( or LIFT ( However, Zaphiris & Zacharia (2001) report some limitations when using automatic evaluation tools to evaluate web sites. There are important elements (such as the web navigation structure, the layout of information, the value of information, or various aesthetic aspects) which are not evaluated by the automatic tools. The authors argue that these limitations might imply that, although automatic evaluation tools provide a quick reference and a first step analysis of the web site’s accessibility and usability, formal usability and accessibility evaluation involving user testing combined with a series of other non-empirical methods (such as expert-based) still hold a major importance in the thoroughness of web site evaluation. Based on this claim we present two of the most commonly used expert-based approaches and we attempt to adapt them to the context of disabled users.

3Expert-based vs. User-based (Empirical) Evaluations

The most commonly applied evaluation methodologies are the expert-based and the empirical (user-based) evaluation ones, according to the taxonomy of most researchers on the field (eg. Preece et al., 1994; Lewis & Rieman, 1994). Expert-based evaluation is a relatively cheap and efficient formative evaluation method applied even to system prototypes or design specifications up to the almost ready-to-ship product. However, according to Lewis & Rieman (1994) «you can't really tell how good or bad your interface is going to be without getting people to use it». This phrase expresses the broad belief that user-testing is inevitable in order to assess an interface. It is even more important in the context of users with disabilities or elderly users. Why then, don’t we just use empirical evaluations and continue to research expert-based approaches as well? This is because the efficiency of these methods is strongly diminished by the required resources and by various psychological restrictions and biases of the participating subjects. On the other hand, expert-based approaches have meanwhile matured enough to provide a good alternative. Nowadays, expert-based evaluation approaches are well established in the domain of HCI. Cognitive Walkthrough (Lewis et al., 1990) and its variations, and Heuristic Evaluation and its variations (Nielsen, 1994) are the most encountered approaches in the field.

4Cognitive Walkthroughs and Heuristic Evaluation

4.1.The Cognitive Walkthroughs

The main idea in expert based evaluation is to present the interface supported tasks to a group of experts who will play the role of would be users andtry to identify possible deficiencies in the interface design. Designers guide expert evaluators to a walk-through of the tasks and offer them an appropriately structured questionnaire to record their ratings. The original Walkthrough method (Lewis et al., 1990) is a substantially slowerprocedure since the entire recording is done manually. A modification of the method, the Cognitive Graphical Walk- and Jogthrough (CGW-J) (Karoulis et al., 2001) utilizes a set of diagrams to record the evaluators’ opinions, speeding up significantly the procedure. The evaluators utilize a set of questions in order to assess the performance of the users, as follows:

1. How many users will think that this action is available, namely that the system can do what the user wants, and simultaneously affords the mode for it to be done.
2. How many users will consider this action, and not some other, to be appropriate for the intended goal?
3. How many users will know how to perform this action?
4. Is the system’s response obvious?
5. How many users will consider that the system’s response brings them closer to their goal?

4.2.The Heuristic Evaluation

A different approach is followed by the Heuristic Evaluation (HE), as described by Nielsen (1994). Heuristic Evaluation is not task based, it is criteria, or heuristic based. These heuristics derive from prior experience on the domain and are broadly accepted and accredited. HCI experts and experts on the domain are asked to judge the interface according to these heuristics. There are also many variations of heuristics, we present here the most recent web-adapted version, suggested by Karoulis & Pombortsis (2003):

1. Simple and natural dialog and aesthetic and minimalistic design.
2. Visibility of the system status – provide feedback
3. Speak the users' language: match between system and real world.
4. Minimize the users' cognitive load: recognition rather than recall.
5. Consistency and standards.
6. Flexibility and efficiency of use – provide shortcuts.
7. Support users' control and freedom.
8. Prevent errors.
9. Help users recognize, diagnose and recover from errors with constructive error messages.
10. Help and documentation.

5Adaptation

The main question of this work is to investigate the suitability of both lists (CGW-J, and HE) in the context of disabled or elderly people and to propose modifications to make the expert-based methodologies suitable in this context. Although there is yet no published research to our knowledge on the domain, our studies showed that it is possible to adapt them efficiently. More precisely, the Cognitive Walkthrough question list does not need any modifications. However, the method does. The cognitive parameter of the evaluation is of paramount importance in the context of disabled users, so at least half of the evaluators must belong to this user group and be simultaneously HCI or cognitive experts. It is a hard prerequisite to fulfill, however only these double-experts could perform satisfactorily in such an evaluation and simultaneously propose solutions, which is the second modification we propose. It is a known drawback of the method that it can not always propose solutions. We believe that in the particular context the double-expert evaluators could and should propose possible solutions, at least to major problems encountered in the interface. A Cognitive Walkthrough with double-expert evaluators, if combined with a participatory design approach, could provide a valuable tool to build usable and accessible interfaces for disabled and elderly people.

Concerning the Heuristic Evaluation, we need to modify the heuristics in a way they affect the special user groups considered in this work. More precisely, we propose the following heuristics:

1. Minimalistic design: follow the pyramid structure with simple and natural dialog
2. Visibility of system status in a variety of ways: provide multimodal feedback.
3. Speak the users' language: match between system and real world. Consider various representations of the same aspect.
4. Minimize the users' cognitive load: recognition rather than recall by utilizing multimodal approaches
5. Consistency and standards: adhere to guidelines on accessibility

6.Flexibility and efficiency of use – provide shortcuts and support customization

1. Support multimodal users' control and freedom.
2. Prevent errors. Build protection mechanisms that interact with the user in a multimodal way.
3. Help users recognize, diagnose and recover from errors with constructive error messages. Provide automatic recovery, under user control.
4. Help and documentation in a variety of ways.

As it is obvious, the notion of “multimodal” interfaces plays a vital role. According to Michigan State University (MSU, 2003): “just as multimedia content presents itself in several ways (text, audio, video) simultaneously, a multimodal interface allows interaction in several different ways (keyboard, voice, mouse)”. To follow guidelines on integrating multimodality in the design may be a good practice for accessibility. For interested readers, IEEE Multimedia journal has devoted an issue (Winter 1996, vol.3, No.4) in multimodal interaction.

However, expert-based evaluations are not limitation-free. Following table (Table 1) summarizes strengths and weaknesses of the expert-based and the usability testing approach:

Table 1: Comparison of expert-based and user testing attributes

Expert-based evaluation / Usability testing
Pro /

• Cheap methods
• Quick
• Can early be applied in the design cycle
• Easy to prepare and to conduct
• Applicable at all stages
• Can assess the severity of the problem
• Good effectiveness/cost = efficiency factor

/

• Unveils problems of real users
• Can pinpoint nearly all problems
• Efficient, even in complex interfaces
• Direct description of the problem
• Unavoidable, if one wants valid results

Contra /

• Doesn’t unveil all problems
• Needs experienced evaluators
• Difficulty in proposing solutions
• Often loses the picture of the «whole»
• Evaluators often forget or can’t play at all the role of the user
• HCI experts and domain experts are indispensable

/

• Expensive and difficult to materialize
• Needs representative users
• Difficulty to find the subjects
• Users tend to be in confusion about the severity of the problems
• Subjects can be biased
• Can be applied after a certain level of completion

6Conclusions

Accessibility, in the context of the World Wide Web, means maximizing the ability of all users to access information, services, and resources. However, things are far from being optimal. The accessibility problem of health information on the Internet is so severe that in a survey to the Northeast chapter of the American Medical Association, only 14% of the physicians said that they would recommend the Internet as a medical information resource for patients (Eder & Darter, 1998). An extensive study by Zaphiris et al., (2001) showed a high browser compatibility error rate in all groups (.edu, .gov, .com and .org) of sites. One possible reason is that web site designers tend to rely on web design tools that are compatible with only one particular type of browser.

Expert based evaluation can contribute in the direction of an easily applicable and less resource demanding approach to assess the accessibility of a web interface. By slightly modifying the methodologies, mainly in seeking the participation of double-experts and utilizing an enhanced set of heuristics, we can obtain important qualitative results. However, the on field validation of our claims remains open. We don’t yet propose any further research on the topic, since we believe that research has just begun, so we welcome any research attempt on the domain that will contribute to a better accessibility of web interfaces.

7References

Eder, L.B. and Darter, M.E. (1998): Physicians in Cyberspace. Communications of the ACM, 41 (3), 52-54.

GVU (1997). GVU's 7th WWW User Survey. Retrieved on January 31, 2001 from the Internet at:

Karoulis, A., Demetriades, S., Pombortsis, A., Karoulis, K., and Polyxenidou, A. (2001). The Application of the Expert-based Interface Evaluation Methods "Cognitive Graphical Walk- and Jogthrough". In Avouris, N., and Fakotakis, N. (edts) Advances in Human-Computer Interaction I, 121-126. Patras, Greece: Typorama.

Karoulis, A and Pombortsis, A. (2003). Heuristic Evaluation of Web-Based ODL Programs. In Claude Ghaoui (edt.) Usability Evaluation of On-Line Learning Programs, 86-107, New York: IDEA.

Kurniawan, S.H, & Zaphiris, P. (2001). Investigating the Age Effects on Subjective Assessments of Traffic Signs. In Proceedings of the Institute of Industrial Engineers (IIE) Annual Conference, May 20-23. Dallas, TX.

Kurniawan, S.H., Zaphiris, P., Ellis, R.D (2001). Involving Seniors in Designing Information Architecture for the Web. In C. Stephanidis (Ed.), Universal Access in HCI, 496-500. Mahwah, NJ: Lawrence Erlbaum.

Lewis, C. and Rieman, J. (1994). Task-centered User Interface Design - A practical introduction, Retrieved from ftp.cs.colorado.edu/pub/cs/distribs/HCI-Design-Book .

Lewis, C., Polson, P., Wharton, C. & Rieman, J. (1990). Testing a Walkthrough methodology for Theory-Based Design of Walk-Up-and-Use Interfaces. Proceedings of ACM CHI ‘90, Seattle, Washington. April 1-5, 235-242.

Michigan State University (2003). Web accessibility explained. Retrieved on 10 Dec. 2002 from the Internet at:

Nielsen, J. (1994). Heuristic evaluation. In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods, New York, NY: John Wiley & Sons.

Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., Carey, T., (1994). Human-Computer Interaction, Reading, Mass: Addison-Wesley.

Zaphiris, P. (2000). Depth vs Breadth in the Arrangement of Web Links. In Proceedings of the 44th Annual Meeting of the Human Factors and Ergonomics Society (HFES 2000), 139-144. July 30 – August 4, San Diego, CA.

Zaphiris, P. (2001). Age Differences and the Depth-Breadth Tradeoff in Hierarchical Online Information Systems. In C. Stephanidis (Ed.), Universal Access in HCI, 540-544. Mahwah, NJ: Lawrence Erlbaum.

Zaphiris, P., and Zacharia, G. (2001). Website Content Accessibility of 30,000 Cypriot Web Sites. In Manolopoulos, Y., & Evripidou, S. (edt.) Proceedings of 8th Panhellenic Conf. In Informatics, vol 2, 128-136. Athens, Greece: Livanis.

Zaphiris, P., Kurniawan, S.K., & Ellis, R.D. (2001). Usability and Accessibility Comparison of Governmental, Organizational, Educational and Commercial Aging/Health-Related Web Sites. WebNet Journal, 3 (3), 45-52.