Collection Guides (“Finding Aids”) Usability Study-
Novice User Group
Report of the Findings
Susan K. Pauley
April 2010
Background
This study was designed to evaluate the usability of the Collection Guides Web site (otherwise known as the Finding Aids Web site), designed by the Special Collections Research Center, from the point of view of the novice user. As an example, a novice user could be a researcher, who, after submitting a query through Google, has been brought to the Collection Guides Web site. This study aimed to see how easily those who had never before or had seldom used finding aids for archival research would be able to navigate the Web site, search for desired information, and submit a request for materials from a collection.
Goals for this Study
The Collection Guides Web site had been designed to present guide information by way of two separate views: a tabbed view (see Appendix A screenshot) and a standard view(see Appendix B screenshot). The tabbed view separated categories of information about the guide into titled tabs; the user would click on the tab with a desired heading to find more specific information about this aspect of the guide. Tabs were not present in the standard view; instead, the user could navigate the guide by either clicking on the table of contents link to the desired section, or by scrolling the length of the guide. Both the tabbed view and standard view offered a search box as a means of searching for keywords throughout each guide.
The web designer had chosen the tabbed view as the default view. However, it was noted that positive feedback was given for the standard view by users familiar with archival research. The Special Collections Research Center (the stakeholders for this study) were interested in whether novice participants would prefer the tabbed view or the standard view of the collection guides. Features specific to each view were also tested and the results analyzed for their usability.
The stakeholders were also interested in seeing the participants’ approach to several select features in the search results screen, such as the “Sort by” drop-down menu, the Online Content checkbox, the “Add to Bookbag” method of saving collections, and the left hand menu facets under “Refine Your Search”.
Methodology:
The stakeholders in this study were interested in selecting both student and non-student participants, including faculty and/or staff members. In this way, a more diverse population of users would be represented.
Undergraduate and graduate participants were selected using two methods. Five participants were student workers selected at random by the Access & Delivery Services department within D.H. Hill Library, and asked to participate in the study. Two participants were student workers recruited from the Information Technology department, while another student worker participant was randomly chosen by the Interlibrary Loan department. All student participants were scheduled for participation during their normally scheduled work hours, and would therefore not receive additional compensation, but were thanked for their feedback and participation.
The faculty member and staff members recruited were acquaintances of a librarian in the Digital Library Initiatives department who requested their participation, and were not affiliated with the Libraries. The staff members and faculty member were also uncompensated for their participation, but were thanked for their participation and feedback.
Testing Environment
All participants were tested individually during a one hour-long session. Testing took place in the NCSU Libraries’ Usability Research Lab, which could be labeled as a single-room setup. The lab room’s dimensions are roughly 10 feet by 15 feet; the room was designed to accommodate one desk and chair for the participant, and one desk and chair for the facilitator on the participant’s right side. One additional chair and a low bookcase have also been arranged at the rear of the room but these were not used in this study’s sessions. There is one mirrored floor-to-ceiling window located next to the door into the room. This window is not facing the participant while he or she is seated at the computer. The room’s temperature is controlled remotely and is kept between 67 and 72 degrees.
The computer used for this study was a Dell Dimension computer running the Microsoft Windows XP operating system. The monitor, a mouse, a keyboard, and two speakers had been placed on the desk in front of the participant. The CPU tower had been placed on the floor underneath and to the left of the desk. A camera was mounted to the monitor and was positioned facing the participant to record his or her actions during testing. Software used in these sessions included Morae Recorder and the Firefox Web browser.
Testing Procedures
One facilitator, consistently the same person, facilitated all test sessions. For three of the test sessions an observer watched the test remotely from the facilitator’s office space using Morae Observer software on a PC and provided feedback to the facilitator about the observation post-session. Participants were all told their computer input, voices, and likenesses would be recorded, in addition to there being someone monitoring the session from a remote location. No observers were present in the room with the participant or interacted with the participant in any way. All participants were recorded using Morae Recorder software on the test machine.
Prior to each session, the facilitator entered the Usability Research Lab to arrange the physical setup of the room. She arranged a chair in front of the test computer so that the participant could comfortably sit in front of the computer screen and keyboard. She placed a second chair to the right of the participant, only close enough to view the screen and to access materials on the table. She powered on the test machine, logged in, then opened Morae Recorder and verified the settings were correct. Through Morae Recorder she also adjusted the screen-mounted video camera to point at the participant’s chair. She minimized the program to the task bar and cleared all other open icons running in the task bar. She cleared the cache in Mozilla Firefox, the browser to be used in the test, so that visited links from previous sessions would not be visible to new participants. She ensured that bookmarks for the two test sites (the main Collection Guides page and the example collection guide page) were viewable in the browser’s bookmarks bar. Finally, she returned in the browser to the library’s home page and left this window visible on the screen before exiting the room.
The facilitator printed a copy of the script (see Appendix C) for each participant. Prior to the test session she marked the date, time, and participant number on the script. She also used a hi-lighter to mark actions in the script that were crucial to the test’s success, such as “Start Recording”, and “Administer the Demographic Questionnaire”. If an observer was scheduled for a particular test, the observer’s name would also be added to the first page of the script. A copy of the script would then be left at the facilitator’s desk for the observer to reference and take notes upon while observing.
At the start of each test session the participant was brought into the Lab by the facilitator, who introduced herself. The facilitator then spoke directly from the script, explaining her reason for doing so: reading from the script for all participants would ensure consistency between all tests. The purpose behind testing was then explained and the paper Demographics Questionnaire (see Appendix D) distributed. Once the questionnaire had been completed, the facilitator described the nature of the test, assuring the participant that the site was being tested, not the participant, and mentioning that all feedback offered would be helpful and appreciated. The facilitator stressed the importance of the participant thinking aloud while working so that site designers could understand why users might make certain decisions on the site.
Since the sessions would be recorded and observed, participants were made aware that Morae software would be tracking their comments, facial expressions and computer activity while someone in another room would be observing the session. Participants were assured that the recorded videos would not be released publicly. Each was given a copy of the Video Release Form (see Appendix E) to read and sign, and told that he or she would receive a copy of this form at the session’s end.
Once any participant questions had been answered, the facilitator would begin the session. The facilitator would take notes on the script during each session and mark task success according to a rubric of task difficulty levels; these levels were given as Easy, Medium, Hard, and Fail. Neither the notes nor the rubric was shared with participants.
Privacy Assurance
All participants had been assigned a number based on the order in which they were scheduled. Participants were known only by this number for the duration of the study. No personally identifying information about the participant, including his or her name, was included in any written or verbal responses. Signed Video Release Forms were placed collectively in a manila folder and kept separate from facilitator script notes. Once the number of emotional expressions during testing had been counted, the facilitator used the Morae Manager software to remove the picture-in-picture video of the participant’s likeness from each participant test video.
Findings: Demographics Questionnaire
Once the purpose of the testing had been explained to the participant, a paper questionnaire concerning demographics was administered. Participants were asked seven questions, including their affiliation, year born, how many hours they typically spend on the Internet per week, their major or area of work, and their level of experience using archival materials for research.
Ultimately this study contained nine students and three non-students. Students presumably represent a greater segment of the site’s target audience. Twelve participants were selected to take part in this study; participants included eight undergraduate students, one graduate student, one faculty member, and two staff members affiliated with the University, but not with the Libraries. Undergraduates were between 20 and 23 years of age, while the graduate student was 28 years of age. The faculty member and staff members were between 47 and 50 years of age.
Roughly four fields of interest were represented in this study. Business-related fields represented included Supply Chain Management and Business Administration, while Science/Engineering fields included Mechanical Engineering, Electrical Engineering, and Textiles. International Studies, History, Sociology, and Psychology comprised the Humanities field. The final field of interest was made up of Law, Research Administration, and an undefined field.
The participants’ prior experience using finding aids and online collection guides was of considerable interest to Special Collections. Of the 12 participants, eight had never before used online collection guides or finding aids in general. The remaining three participants had used these tools anywhere from one to five times before this testing experience.
In terms of prior experience using archival material, five participants felt they had intermediate level skills. None considered themselves having expert skills, while four other participants considered themselves beginners with using archival materials.
As for computer and Internet usage, eight of the participants spend more than 10 hours per week on the Internet. Seven participants considered themselves having intermediate skills using computers and the Internet, while four participants considered themselves expert users of both computers and the Internet.
Findings: Impressions of Example Collection Guide
For the first portion of the test, participants were asked to respond verbally to a series of pre-test questions about an example collection guide (finding aid) to which the facilitator navigated for them: (see Appendix F). These questions were designed to gather feedback about their initial impressions of the Web site. This example was only shown to participants in the tabbed view; initial impressions were not gathered of the example using the standard view.
Prior to answering questions, participants were first asked to familiarize themselves with the page, scrolling up and down the page and clicking on the tabs. They were then asked to tell the facilitator their initial impressions of the page, and what struck them about it. Answers provided by participants were similar overall and were mostly favorable. Participants described the layout as being clean and well organized. Some described the large tabs as being helpful, while one participant stated it seemed easy to locate information because the tabs helped organize everything into categories. However, there were additional comments reflecting some dislike of the design. One person commented that the font was difficult to read. Two other participants spoke of the Collection Details tab being confusing. There seemed to be confusion regarding whether these materials were all online, such as wondering as to the meaning of “boxes and folders”.
Participants were also asked whether they found anything on the page confusing. Only one participant commented that it wasn’t clear how one would find a specific item, such as a photograph. All other participants remarked that at first glance, nothing seemed confusing.
Next, participants were asked if they liked or disliked anything about the design of the page. Participants mentioned favorably that it was not cluttered, one would not need to scroll to navigate the page, and the large tabs made reading and navigating a simple process. However, two participants said that the collection guides were too compressed on the page. Another participant stated that the gray background was undesirable.
Participants were asked what they felt the page was describing. All participants, to various degrees, described the elements of the William Ludwig Baumgarten Papers collection. Most cited specific elements of the collection, including papers, photographs, and student works. Others additionally described facts from Baumgarten’s history, such as his being a professor at one time with the University and an architect. All participants appeared to understand that they were looking at information about a person, however, it was unclear whether they associated the information with a collection of materials about this person.
Finally, participants were asked, from the example collection guide page, to whom this website belonged. Surprisingly, despite the large red Special Collections Research Center banner atop the page, only 1/3 of the participants knew the website belonged to Special Collections. Most of the participants felt it belonged to the Libraries, while 2 people felt it belonged to the University.
Findings: Tasks
Tasks for this study were created based on feedback provided by the Special Collections Research Center according to Web site features they requested be tested. Therefore, to ensure these specific features would be evaluated for their usability with novice users, it was not possible to allow participants to approach the site with their own specific information searches in mind.
During each test session the facilitator took notes on the script while rating the participant’s success with the task according to a defined rubric of task difficulty.
Difficulty Level / Numerical Code / DescriptionEasy / 4 / Participant completed the task on the first or second try with no difficulty
Medium / 3 / Participant completed the task on the first or second try but exhibited difficulty completing the task
Hard / 2 / Participant completed the task on the second or third try and expressed difficulty through verbal feedback
Fail / 1 / Participant stated he or she could not complete the task (essentially giving up), did not complete the task, or completed the task without reaching the desired result by any of several possible methods.
In the descriptions below, the phrase “average difficulty rate” refers to the average of the numerical codes across twelve participants. High values indicate easy difficulty levels, while low values indicate greater difficulty or failure with a task.
Participants were each provided a paper copy of the tasks to reference while working (see Appendix G).