Report for the Student Services Web Page Management Group

Report for the Student Services Web Page Management Group

FROM:Web Page Usability Testing Team

SUBJECT:Web Page Usability Test 1st December 2004

DATE:1st February 2005

BACKGROUND

The members of the Web Page Usability Testing Team had intended, individually, for some time, to use students to test web page usability before the opportunity to develop this exercise arose.The catalyst for action was the decision to change the Things Not Going Right / Signposts Questionnaire page.

Web Usability Testing Team Members:

• Gill Bartlett(Learner Support), who organisesfocus groups of new students to identify additional support requirementsand has had a long term interest in testing the web pages.
• Scott Castle(Team Leader of SSiD and Student Services Web Page Manager), who has had some previous experience in usability testing in the systems development environment.
• Gill Hughes(Information Assistant in SSiD), who gained recent exposure to many different web pages when helping her son decide which universities to apply to and initiated the development of the Useful Contacts for Current Students Page (

SSID Web Pages

There are some 800 pages in the SSiD suite of web pages. The ‘site’ receives over 200,000 hits (50,000 visits) from 25,000 unique siteseach month. Some of the more popular pages are Your Record (7,000), Course Information (10,000) and Examination Timetable (up to 20,000).

Most of the changes undertaken in the last year have been to improve accessibility to these pages, devolve responsibility for updating content and a limited amount of time has been available to improve individual page design, navigation and usability. This exercise was seen as a good opportunity to increase the priority of time given to usability, navigation and individual page design issues.

1. THE AIMS OF THE EXERCISE

A conscious effort was made to ensure that we were collecting data relating to the users actual behaviour rather than their just their opinions.

“To design an easy-to-use interface, pay attention to what users do, not what they say.” Nielsen (2001)

Furthermore we wanted the information collected to support changes rather than determine changes to the web pages.

“The purpose of testing is not to prove or disprove something. It’s to inform your judgement.” Krug (2000) p142

The students were not selected from specific groups but from those who had been used in a previous focus group and were known for their ability to express themselves.

Finally, the exercise was seen as a pilot test: an opportunity to test and refine the experimental procedure.

2. THE EXERCISE: PREPARATION, EXECUTION AND REVIEW
2.1 Preparation

2.1.1 Format

Following from our research into the subject and some discussion, it was agreed that:

a)the usability tests would be conducted over the course of one working day. There is published research to indicate that there is no need for big numbers of web testers. Even one will show you things that you can do to improve your site and 5 users will tend to uncover 85% of a site’s usability problems.

b)the tests would involve observing students navigating the current students (SSiD) web pagestrying to achieve objectives which would start with easy tasks but become increasingly challenging.

c)the tests would be conducted with one student at a time and involve a facilitator (to guide and prompt the student) and an observer (to take notes).

d)only first year students would be recruited for the tests as they were more likely to be relying purely on the usability of the site to find information. Second or third year students were more likely to have memorised where information could be found or be already familiar with our terminology through past experience of using our web pages.

e)The members of the testing team would take it in turns to act as facilitator and observer in order to build up expertise in both roles.

f)that the test for each student should last for approx 50 minutes.

2.2.2 Resources

Apart from our staff time, the resources required to complete the exercise were negligible.

As students were being tested individually, a large room was not required. The Student Services Meeting Room was used because it had adequate space, communications and was convenient. The room was set up with computer, audio and video equipment (more details about the hardware later in the report). All the necessary equipment was provided and set up by Audio Visual Services free of charge.

Five undergraduate students were recruited (of which four turned up) and provided with a £10 book voucher as a token of our appreciation for their help. Apart from the book tokens there were no other direct costs involved in the exercise.

Of the 4 students who were involved in the usability testing:

• 2 were male and 2 were female
• 1 was a very mature student
• 2 were local or ‘commuter’ students
• they were from varied departments/disciplines (Archaeology, Law, Mechanical Engineering and Psychology)

2.2.3 The Script

The team started with a script suggested by Krug (2000) (a version of which was downloaded from However significant changes were soon made to include the specific requirements for the test(See Appendix 3 for the script that was eventually used).

The intention was that the facilitators would stick to the script, although a certain amount of improvisation was allowed. This obviously, did not include leading the student or giving clues as to what they should do.

The script started by:

• informing student web testers why we were conducting the usability tests
• reassuring them that it was the web site being testing and not them – therefore they couldn’t do anything wrong.
• asking them to be as honest and as critical as they liked without worrying about hurting anyone’s feelings
• reminding them to do all their thinking out loud – i.e. to vocalise throughout the exercise what was going through their minds.

The students were asked for their permission to record both their dialogue and their navigation around the computer screen. (None objected).

2.2 The Test Itself

2.2.1 The Tasks:

a)Contextual Questions, to put the student at ease and gain an idea of their web usage experience and any bias.

b)Four ‘Scenarios’, which challenged students to find a particular piece of information. Each scenario consisted of two or more objectives: the initial task and a related progression e.g. find exam timetable (task), find exam clash form (progression). (See Appendix 1)

c)Home page evaluation, which consisted of asking the student their impression the Current Students ‘home’ page (

d)Evaluation of another university’s Current Students ‘home’ page.

2.3 Evaluation of the Test

a)Only minutes into the first test, it was obvious that some useful insights were being gained into how these web pages were used. Afterwards, once the recordings had studied it was realised that the exercise had indeed obtained a large wealth of information with minimal effort and cost.

b)The key task testing approach, using the four scenario exercises which involved the students finding information and performing key tasks,had proved to be particularly useful.

c)The least useful part of the test had been the evaluation and comparison of the ‘current students’ home pages for this and another institution. Although a few interesting comments had been made, students seemed to struggle to find things to say. This may have been due to several factors: high expectations by testers, reticence of the students, or the inappropriateness of the test for this task.

d)It was decided that the exercise would be repeated in February.

3. INFORMATION CAPTURE

3.1 Video and Audio recording

The equipment set up (scan converter, video recorder, microphone, monitor) allowed camera video taping and capture of the click streams (video) and student comments (audio).

3.2 Observers Note Taking

a)Contemporaneous notes. Notes were made by the observer during the test. These notes were generally reflections of the observer rather than the comments of the student which were colleted more comprehensively later from the audio recording.

b)Audio recording. A sound file was created (Wave format) from the Video, that could be listened to at a later time via a PC so that more detailed notes, from the students comments, could be made.

c) The notes collected in a) and b) above were recorded on a Word document. (See Appendix 2: Usability Test 011204 (Notes).

3.3 Tracking User Click Streams

Using the video, the user click streams were recorded onto Excel Spreadsheet (See Appendix 5: Usability Test 011204 (Clicks).

4. FINDINGS

4.1 Content Volume

The students commented on several occasions that there was a lot of information to read and that if it was not a test environment they would cease searching for the information.In Scenario 4 – Things Not Going Right, one student commented that, if you were distressed or anxious, you would not take the time to read four hundred words.

4.2 Accessing SSiD pages

There were a variety of methods that students used to access the SSiD web pages. These included clicking on current students on the table of contents, typing SSID in the left hand search box and going to the ‘departments’ page. [10] [13]

4.3 Students not knowing what they did not know.

When trying to find specific information the students experienced frustration by not knowingthe difference between information that may meet their requirements andinformation that was totally irrelevant.

a)There was confusion between what would be the most appropriate document showing examination results: Confirming letter / Certified Copies of Degree Certificate / Transcript etc. [4][14]

4.4 Finding Specific Information Sets

There were several issues generated when the student tried to find a Transcript:

a)Most did not know that a Transcript was an ‘official’ summary of examination results. [4]

b)All expected the link to the appropriate information to be located on ‘Your Record’ page (

4.5 Search Box Effectiveness.

The left hand search box did not seem to produce useful results and generated several negative comments from the students. [11] [15]

4.6 Choosing Appropriate Information Medium

In respect of scenario four, all the students said they would approach their personal tutor rather than trying to find information on the web. [7] [14]

4.7 Link Descriptions

Some link descriptions were identified as either inappropriate or leading to pages that provided content that was not expected or wanted.

a)A student followed a link on the Student Welfare page ( titled ‘Student Support & Guidance’ anticipating that it would be a link to useful information relevant to the issue she was trying to resolve. The link went to administrative information about the Student Support and Guidance section in the Student Services Department. As the student put it: “When I clicked on Student Support and Guidance I thought I would get something about Student Support and Guidance” [8]

b)More than one student noticed that when they clicked on a link title ‘Things not Going Right’ they were presented with a page that was titled ‘Signposts Questionnaire’. [9]

4.7 Links to Further information

One student came across a section on a page relating to Short Term Loans and assumed that this was the exhaustive content relating to the subject. No link existed to take the student to the definitive page. [16]

Notes referenced above can be found in Appendix 1:Web Page Usability Testing – Notes

5. CHANGES MADE TO WEB PAGES

5.1 Things Not Going Right

a)The introductory page was rewritten. The number of words on the page was substantially reduced.

b)Links were also added to the introductory page to categories that were separated into subject areas on a series of (five) new pages.

5.2 Examples Pages

Created a series of (ten) new pages which display a picture of particular documents that can be obtained from SSiD.

5.3 Current Students (SSiD) home page changes.

a)A series of images (8) which show two pictures: apicture of a student using the SSID facilities to find information for themselves and a picture of a student talking to Information Assistant at the SSiD front desk.

b)Revised the middle section of the page so that it looked more consistent with University home page style.

5.4 Summary of Results page

A new page was created to detail the purpose, components and appearance of Summary of Results report.

5.5 Your Record Links

Introduced links to the Transcript of Qualification and new Summary of Results pages. Also changed the order of the to make the ‘Changing your Address’ section visible when the page is first entered.

5.6 Certificate of Student Status page

Changes to this page to illustrate the variety of certifying letters there are available on request in SSiD. (taught, taught with fee, bank and term time or home address)

5.7 Expanded Frequently Asked Questions page

Arising out of the change to the ‘Things Not Going Right’ page, 84 new questions were added and the subject areas organised into nineteen categories.

5.8 Revised the A to Z index and Other Useful links pages

Separate links into those that are internal to the University and Union (A to Z index) and those that are external (Other Useful Links). Secondly made the list of links single column and added a navigation link to return the user to the top of the page.

5.9 Student Support and Guidance

Changes were made to the ‘home’ page of the Student Support and Guidance section. Thisinvolved redesigning the page to make it read more like an informative page to students rather than a list of administrative aims and objectives.

5.10New Frequently Asked Questions category: What is it?

A new page was created on the FAQ section. The questions try to explain some of the University terminology for the student and remove some confusion. Questions include: A Transcript of Qualifications – What is it? Access to Learning Fund – What is it?

5.11 Other Minor Changes

Minor changes weremade so that the page was more attractive and usable, a link to Summary of Results was introduced and an explanation provide, up front, about the likelihood of charges for the service.

Not all of the above changes were in direct response to student comments, suggestions or behaviour. Some changes were indirectly inspired by the exercise as a whole.

6. CHANGES MADE TO THE TEST MODEL

6.1 The Current Students (SSiD) home page evaluation will be deferred to a later date andnot included in the February test. Further investigation needs to be made intohow to make the test more effective / students more forth coming with views.

6.2 One or two additional scenarios can be added to February test as a result of a).

6.3 Some script changes will be made. In particular, more guidance will be given to the facilitator, to ensure that each student starts the particular task at the same point.

6.4 Specific scenarios to be repeated in the February test. This will be done principally to test changes we have made as a result of the December 2004 test.

6.5 The facilitator will demonstrate ‘thinking aloud’ to the student before the first scenario begins. Suggested by Nielsen (93) p 197

6.7 Scenario 4 will be developed so that the student will be encouraged to refer to the web pages before talking to a tutor. This could be achieved by adding the caveat that it is late at night or it is a subject you do not want to talk to your personal tutor about.

Bibliography

Advanced Common Sense Home page at
Krug, Steve (2000) “Don’t Make Me Think! A Common Sense Approach to Web Usability” New Riders Publishing
Neilsen, Jacob & Tahir, Marie (2002) “Homepage Usability: 50 Sites Deconstructed” New Riders Publishing
Neilsen, Jacob (1998) “Cost of User Testing a Website” May 3, 1998
Neilsen, Jacob (2000) “Designing Web Usability: The Practice of Simplicity” New Riders Publishing
Neilsen, Jacob (2000) “Why You Only Need to Test With 5 Users” March 19, 2000
Nielsen, Jacob (1993) “Usability Engineering” Morgan Kaufmann Publishers Inc.
Rosenfeld, Louis & Morville, Peter (2002) Information Architecture for the World Wide Web” ORielly & Associates
Rubin, Jefferey (1994) “Usability Testing: How to plan, design and conduct effective Tests” John Wiley & Sons
Nielsen, Jacob (2001) ‘First Rule of Usability? Don’t Listen to the Users’ August 5, 2001

Appendix 1: Scenarios for Web Page Usability Test 1st December 2004

SCENARIO 1: Examinations

Can you imagine that you have exams coming up that involve using a calculator. Find the dates of your exams and information about using calculators in exams.

Progression: You have checked the dates and times and realise that you are timetabled to take two exams at the same time. A friend has told you that you can report a clash of exams online. Find the Examination clash form.