Sarah Carrier
SILS/MRC
December 12, 2006
Application of Cognitive Walkthrough Methods for the Development of the NESCent Evolutionary DigiData Repository:Methodology: An Annotated Bibliography
Introduction:
The development of a leading digital scientific data and research repository for NESCent evolutionary biology collaboration requires an understanding of typical behaviors associated with data sharing and use in scientific communities. In order to develop a usability study for the Evolution DigiData repository currently in development, a collection of resources is useful to detail the steps for mini-scenario development through the usability inspection method of cognitive walkthough. The goal is of DRIADE is to create a system that will closely match the typical Evolutionary DigiDatasupport preservation, access, and sharing of published data and supporting data sets repository user’s work contexts. AnotherTo achieve this goal, we shouldis to involve evolutionary biologists in the planning and process of system development. A project on the scale of the NESCent endeavor requires a unique strategy for scenario development that will inform as to the creation, interpretation and application of scientific data. The cognitive walkthrough technique will identify any potential usability problems and inform the design process, therefore benefiting the users of the new system.
The bibliography is intended for the specific use of the NESCent repository development project, but could prove to be useful for others intending to develop a unique plan for scenario development in other repository projects.
The next section will give a brief overview of the cognitive walkthrough method, followed by the actual sources with a brief description of how each applies to the project. Sarah Carrier created the abstracts that accompany the citations. Few articles have been found that consider the specific application of cognitive walkthrough to repository development, therefore sources have been identified that could inform the process and be applied in a different scenario. The articles have been classified into three categories: Introduction and Overview, Assessments and Implementation of the Technique, and Modified Cognitive Walkthrough.
Summary of Cognitive Walkthrough:
Cognitive walkthrough is a technique used for evaluating a system design for ease of learning and identifies usability issues. It one of a range of u
Usability inspection methods. Usability inspection techniques such as cognitive walkthrough come into play after the design phase of a project has begun and focus on the assessment of existing interfaces. Cognitive walkthrough reveals whether a user can easily learn how to use a system by identifying certain correct actions in the accomplishment of a system task. Therefore, the interface design is a successful one if the user is able to step through the correct actions. In most cases, the designers and developers of the system are the ones that walkthrough the steps as a group.
The first phase of the technique involves the identification of the input conditions for the walkthrough, including the action sequences for a task and the description of the user population. The second phase involves the analysts themselves stepping through the action sequence. The results of the walkthrough are reflected in changes to the design of the interface.
References:
1. Introduction and Overview
Haynes, S., Purao, S. and Skattebo, A. (2004) Situating Evaluation in Scenarios of Use. CSW 2004, November 6-10, 2004, Chicago, IL, USA.
The authors describe the efficacy of scenario-based methods for system evaluation, specifically for the evaluation collaborative systems. They describe the methods and include a case study where it was successfully applied. The results suggest that scenario-based evaluation is quite effective in identifying the wide range of factors that ensure system success.
Wharton, C., Rieman, J., Lewis, C. and Polson, P. (1994) The Cognitive Walkthrough Method: A Practitioner’s Guide. In Nielsen, J. and Mack, R. (eds) Usability Inspection Methods. Chichester: John Wiley and Sons.
The authors provide a detailed description of how to perform a cognitive walkthrough, showing how the method fits into the process of system development. The authors then summarize experiences and evaluations of the method. The article is one of the fundamental references on the subject.
2. Assessments and Implementation of the Technique
Blandford, A., Kieth, S., Connell, I. and Edwards, H. (2004) Analytical Usability Evaluation for Digital Libraries: A Case Study. JCDL’04, June 7–11, 2004, Tucson, Arizona, USA.
The authors compare four different usability assessment techniques by applying them to digital libraries. The techniques compared were: heuristic evaluation, cognitive walkthrough, claims analysis and CASSM (Concept-Based Analysis of Surface and Structural Misfits). The authors found that heuristic evaluation and cognitive walkthrough address superficial aspects of interface design whereas claims analysis and CASSM can help identify more subtle complexities. However, none of the techniques fit seamlessly with existing digital library development practices, emphasizing an area for further work to support improved usability.
Gray, W. and Salzman, M. (1998). Damaged merchandise? A Review of Experiments that Compare Usability Evaluation Methods. Human-Computer Interaction. 13:3, pp.203-61.
In this review the authors examine the design of five experiments that compared usability evaluation methods (UEMs). The results show problems in the way the experiments were designed and conducted. As a result, the authors emphasize that the choice of evaluation method needs to be taken very seriously.
Jeffries, R., Miller, J. R., Wharton, C. and Uyeda, K. M. (1991) User Interface Evaluation in the Real World: A Comparison of Four Techniques. Proceedings of the SIGCHI conference on human factors in computing systems: Reaching through technology. pp.119–124.
In this study the authors have four groups evaluate a user interface for a software product, each applying a different assessment technique: heuristic evaluation, software guidelines, cognitive walkthroughs, and usability testing. The advantages of all the techniques are discussed, and suggestions for improvements in the techniques are offered.
Huart, J., Kolski, C. and Sagar, M. (2004) Evaluation of Multimedia Applications Using Inspection Methods: The Cognitive Walkthrough Case. Interacting with Computers. 16:2, pp. 183-215.
This article presents a study on the use of cognitive walkthrough for the assessment of multiple multimedia applications intended for the use of the general public. The study reveals the difficulties met by those attempting to employ the method. The authors therefore suggest the areas in which the method needs to be adapted.
Wharton, C. Bradford, J. Jeffries, J. Franzke, M. (1992) Applying Cognitive Walkthroughs to More Complex User Interfaces: Experiences, Issues and Recommendations. CHI 1992. pp.381-388.
In this study teams of professional developers employ the method of cognitive walkthrough to assess the usability of three software systems. The authors report evidence about how the methodology worked for these evaluations. The study focuses on five issues, including task selection, the cognitive walkthrough process and the interpretation of results. The findings show that the success of the technique is affected by many variables.
3. Modified Cognitive Walkthrough
Blackmon, M.H., Polson, P.G., Kitajima, M. & Lewis, C. (2002). Cognitive Walkthrough for the Web. 2002 ACM conference on human factors in computing systems (CHI 2002), pp.463-470.
The authors propose a new cognitive walkthrough method for the Web called Cognitive Walkthrough for the Web (CWW). The authors deem the adapted technique superior for evaluating how well websites support users' navigation and search tasks. The CWW method employs Latent Semantic Analysis to estimate the degree of semantic similarity between known-item searches and heading/link texts on each web page. For an example of utilization, CWW is applied to an existing website and identifies multiple problems in design.
Blackmon, M.H., Polson, P.G. and Kitajima, M. (2003) Repairing Usability Problems Identified by the Cognitive Walkthrough for the Web. CHI 2003. pp.497-504.
As a continuation of the previous study showing the efficacy of CWW, this paper describes two experiments that prove the effectiveness of the new method for both full-scale and quick-fix website repair needs. The authors also prove that if repairs identified by CWW are not implemented, then users will continue to face difficulties in navigation.
Lewis, L., Polson, P., Wharton, C. and Rieman, J. (1990) Testing a Walkthrough Methodology for Theory-Based Design of Walk-Up-and-Use Interfaces. CHI 1990 Proceedings. pp.235-242.
The authors derived a cognitive walkthrough method for evaluating the usability of an interface for walk-up-and-use systems. Testing of the walkthrough method itself reveals the consistency with which such a procedure can be applied as well as the accuracy of the results.
Rieman, J., Franzke, M. and Redmiles, D. (1995) Usability Evaluation with the Cognitive Walkthrough. Conference on Human Factors in Computing Systems, 1995.
This tutorial presents a simpler method of cognitive walkthrough, founded in an understanding of the cognitive theory that describes a user’s interactions with a system. The strengths and limitations of the walkthrough method are considered.
Rowley, David E. and Rhoades, David G. (1992) The Cognitive Jogthrough: A Fast-Paced User Interface Evaluation Procedure. Proceedings of CHI 1992, pp.389-395.
According to the authors, cognitive walkthrough procedures tend to be time-consuming and unpopular with evaluators when applied to large projects. A fast-paced methodology was developed to maximize the useful information obtained while minimizing the overhead of the procedure itself. The study is carried out within the constraints of a real-world product development environment, therefore rendering the results applicable to many environments.
Sears, A. (1998) The Effect of Task Description Detail on Evaluator Performance with Cognitive Walkthroughs. CHI 1998. pp.259-260.
In this study the cognitive walkthrough method was revised to speed the evaluation process. The results indicate that providing detailed step-by-step task descriptions significantly changes the types of problems found. The author provides suggestions for future research and how practitioners could apply this technique.
Spencer, R. (2000) The Streamlined Cognitive Walkthrough Method, Working Around Social Constraints Encountered in a Software Development Company. CHI 2000. pp.353-359.
The cognitive walkthrough method has been shown to be difficult to apply in a large software development companies due to social constraints therein. By enforcing changes such as streamlining the cognitive walkthrough method and data collection procedures, constraints in organizational structures can be overcome. The authors describe a modified cognitive walkthrough process that accomplishes these goals, showing how it has been applied in a real-world situation with successful results.