Usability Test Plan for HolidayInn.com.cn

Holiday Inn Hotel Website for Chinese Speaking Users

Prepared for: Karen Bennett, Project Sponsor

User Experience Manager at IHG

Cc: Dr. Carol Barnum, Project Advisor

Completed by: Team CBR

Yufei Duan

Yina Li

Ying Li

Qianying Liu

Niven Sellars

Michael Somer

Date: March 02, 2008

Table of Contents

Purpose

Problem Statement and Test Objectives

User Profile for U.S.-based Participants

Methodology and Tasks/Scenarios

Number of Participants

Length of Sessions

Search Ground Rules

Test Procedures

Overview/Briefing (10 minutes)

Scenario 1 (5 min)

Scenario 2 (15 min)...... 8

Scenario 3 (5 min)

Scenario 4 (5 min)

Scenario 5 (10 min)

Scenario 6 (optional) (10 min)

Closing (5-10 minutes)

Testing facility...... 10

Evaluation Room...... 10

Control Room...... 10

Executive Viewing Room...... 10

Data Collection Methods...... 11

Quantitative Data

Qualitative Data

Project Deliverables

Appendix A: Screener Questionnaire

Appendix B: Pre-Test Questionnaire

Appendix C: Post-tAsk Questionnaires...... 15

APPENDIX D: POST-TEST QUESTIONNAIRE…………………………………………………………………..18

Purpose

The purpose of the Chinese HolidayInn.com.cn usability test is to collect feedback about how users use the Holiday Inn online hotel booking site in China, what problems they may encounter using the siteand what improvements they would like to see to make it easier to book a hotel room.

Ms. Karen Bennett, the User Experience Manager of InterContinental Hotels Group,is the project sponsor. She would like to learn about the user’s experience on the Chinese web site, as compared to their experience with a competitor web site: If time permits, she would also like to know how their experience compares with the U.S. booking site.

This test plan describes:

  • The problem statement and test objectives
  • The types of users to recruit for testing
  • The testing methodology and proposed tasks/scenarios
  • The test setup in the Usability Center
  • Plans for data collection and reporting
  • Project deliverables
  • The estimated project timeline
  • Questionnaires and other materials to be used in testing, in appendices:
  • Appendix A: Participant screening questionnaire
  • Appendix B: Pre-test questionnaire
  • Appendix C: Post-task questionnaire
  • Appendix D: Post-test questionnaire

Problem Statement and Test Objectives

This usability test of the Chinese Holiday Inn website will provide qualitative and quantitative data addressing IHG’s interests in understanding the Chinese user’s experience. We will assess the users’ experience with booking a room, as well as their overall satisfaction with the site. Tasks for testing include:

  • The general feeling/layout of the site: Does the layout suggest the route first-time users will take to book a hotel?
  • The procedure of booking a hotel online:

Basic search: Is it easy to use?

 Advanced search: Can users accomplish their goals on the advanced search screen?

 Entering personal information: Does the website require reasonable and suitable information for Chinese users? Do users understand all information requirements?

  • Language: Do users understand all the wording on the website? Are there any translation mistakes or misunderstandings?
  • Information in confirmation email: How quickly can users receive the confirmation email after booking a hotel online? Does the confirmation email contain enough information to suit users’ needs?
  • Perceived reliability of the site: Do users trust the website? Do they fill out the personal information readily?
  • Navigation: Can users find the most efficient navigation when they book a hotel or browse the website?
  • Satisfaction: Which aspects do users like and which aspects do they dislike?

These questions were devised using information supplied by our sponsor and our assessment of potential usability problems resulting from our heuristic evaluation of the site(submitted previously).

Thisusability study will be designed to allow us to extract mostly qualitative data.The focus of the study will be task-oriented and directed toward how the user subjectively responds to the issues listed above. Participants will be given a pre-test questionnaire to get information about their hotel booking experience and expectations, followed by scenarios that direct them to perform specific tasks. Post-task questionnaires after each scenario, and a post-test questionnaireare designed to obtain detailed and specific feedback about features.

Users will be scheduled in one-hour sessions to include pre-test and post-test questionnaires. Users will be given 40 minutes to complete five scenarios. If participants complete the scenarios in less than 40 minutes, we have designed an extra scenario to obtain additional feedback.

The test will be conducted in a full-scale usability lab at Southern Polytechnic < Logging and recording of sessions will be done using Morae and also with backup recordings on DVD. The participant computer will have
Microsoft’s Chinese character-based language installed.

User Profile for U.S.-based Participants

To recruit prospective test participants who represent the site’s actual users, the project team identified two primary users groups and created two descriptions of those users. Personas for these two user profiles were submitted previously.

Since the situation is that of testing the Chinese website within the United States, we will use only one of the user groups/personas, which will allow us to recruitthese users to participate in the study in the Usability Center.

The characteristics of the U.S.-based user group, along with other questions about prior experience, behaviors, and other criteria, will make up the screening questionnaire (Appendix A) for identifying suitable test participants.

A general description of the user characteristics is as follows:

  • Must be able to speak and read Chinese as well as English
  • First language must be Chinese
  • Travel purpose: either for business or for pleasure
  • Age: 24-55
  • Gender: female and male
  • Must have experience booking a hotel room online
  • Must use the Internet at least 5 hours per week
  • Must not have any prior experience with the Chinese Holiday Inn website

Methodology and Tasks/Scenarios

This section describes the testing methodology and scenarios we will use with six participants in one-hour sessions.

Number of Participants

We plan to test a total of six participants: three who are traveling for business and three for pleasure. To ensure an adequate number of participants, we plan to recruit eight users total based on our screener; two out of the eight are back-up participants.

Length of Sessions

The total length of each session will be an hour; including:

  • Welcome and pre-test questionnaire: 10 minutes
  • Task Scenarios: 40 minutes
  • Post-test questionnaire: 10 minutes

The estimated time for each scenario is noted below.

Test Procedures

Participants will not be allowed to use any resources that are not available on the Holiday Inn site. (IE: No use of Google or other outside sources.) This restriction will not be stated in the moderator’s introduction, so as not to encourage the participants to think about outside resources. However, if participants want to use these resources, the moderator will notify them that such action is not allowed for testing purposes.

Overview/Briefing (10 minutes)

The moderator will welcome the participant and have him/her sign the video consent form. Next, the moderator will explain the facilities and ask the participant to think out loud. Also, the moderator will provide the participant with the pre-test questionnaire (Appendix B).

Scenario 1 (5 min)

You and your spouse are thinking of going back to China for the Summer Olympics. You heard that theHoliday Inn has several locations in Beijing. You will visit the Chinese Holiday Inn website. (This will be minimized on the desktop.) Take a moment to look at the homepage without clicking on anything. After you have familiarized yourself with the homepage, tell us: What do you think you would do on the site to book a hotel room?

Scenario 2 (15 min)

You are going to book a hotel room for the trip back to China. You wantto reserve onehotel room with one bed for two people in Beijing. You will stay fromJuly 20th to July 25th. Here is the information you need to book the room:

  • First Name: Jing
  • Last Name: Li
  • Reserve for you and your spouse.
  • Location: Beijing
  • Check in date: July 20, 2008
  • Check out date: July 25, 2008
  • Credit Card Type: Visa
  • Credit Card Number: 1234 2345 3456 4567
  • Credit Card Safe code: 890
  • Credit Card Expiration Date: 10/2009
  • Email:

Scenario 3 (5 min)

You just answered a phone call from your relative who lives in Beijing. She wants you and your spouse to stay at her house during the visit. You accept the invitation, so you must now cancel your Holiday Inn reservation.

Scenario 4 (5 min)

You have some relatives inChangchun, Jilin province. You want to visit them during your trip to China. See if there is a Holiday Inn available in Changchun from July 26-30.

Scenario 5 (10 min)

Go to the competitor site See if a hotel room is available for you and your spouse between July 21 and 25 in Beijing. (Note: You are not going to actually book a hotel room on this website.) Look at the search results page and tell us:

  • What do you think of this site?
  • What do you like about it?
  • What do you dislike about it?

Scenario 6 (optional) (5 min)

Go to the Holiday Inn U.S. website (

Take a moment to look at the home page without clicking on anything.

Tell us what you think of this website, compared to the Holiday Inn China and eLong websites.

Closing (5-10 minutes)

Moderator will discuss the experience with the participant and ask the participant to complete the post-test questionnaire (Appendix E).

Testing Facility

The test will be conducted at the Southern Polytechnic Usability Center located in the Atrium Building on the campus, in rooms J-160-J163. The Usability Center includes three rooms:

Evaluation Room

The evaluation room is where participants will complete tasks. The room has the following equipment:

  • Lenovo PC (Windows XP, campus network connection)
  • Microsoft Chinese character-based language installed for this study
  • Keyboard and mouse
  • 16” Monitor
  • 3 wall-mounted cameras and a Logitec digital desktop camera
  • Microphone
  • Telephone/intercom to control room

Control Room

The control room is where the team will observe and record the test. There is a one way mirror between the evaluation room and the control room. The control room has the following equipment:

  • A logging computer with Morae logging software
  • Monitors which display what the cameras are capturing in the evaluation room
  • Headsets for communication among the team members
  • Video recording and editing equipment
  • A picture-in-picture digital mixing board
  • A phone/intercom for the participant to call the help desk
  • Lighting controls

Executive Viewing Room

The executive viewing room provides opportunities for additional observers from IHG to directly observe the test without interrupting the team. This room also serves as the place for team meetings, as well as the final presentation. The executive viewing room has thefollowing equipment:

  • Lenovo PC (Windows XP and Microsoft Office 2007)
  • 2 desktop monitors, which display picture-in-picture viewing of the session
  • A large screen with LCD projector to display picture-in-picture viewing of the session
  • Whiteboard for notes during sessions
  • Conference room table and chairs
  • Numerous data ports for laptop internet access
  • Wireless internet access

Data Collection Methods

Data collection will be a combination of quantitative measurements and subjective observations. The quantitative and qualitative data measures are listed below.

Quantitative Data

The following data will be logged during testing sessions:

  • Time to find an available hotel
  • Number of participants completing each task
  • Number of problems encountered during each task
  • Number of participants that fail to complete each task

From questionnaires, data will be collected regarding participants’ assessments of:

  • The level of difficulty in booking a hotel room on the site
  • The level of confidence using the site to book a room
  • The level of reaction to the appeal of the site

Qualitative Data

Non-verbal and verbal feedback and observations will be collected, including:

  • Facial expressions, such as:
  • Confusion
  • Uncertainty
  • Pleasure
  • Verbal comments when they think out loud, such as:
  • Complaining
  • Liking
  • Comments/questions, such as:
  • Asking questions
  • Calling for help

Project Deliverables

The findings will be presented to the sponsor and the course instructor in the following ways:

  • a formal report
  • a team presentation, including
  • video highlights of the top positive and negative findings
  • electronic files of the session logs
  • DVDs of the recorded sessions

These project deliverables will be presented and delivered on May 1.

Appendix A: Screener Questionnaire

Name: (please print): ______

Age: ______Gender: ___ Male

___ Female

Occupation: ______

  1. How often have you have booked a hotel online in the past year?

Never (disqualify)

1-3 times

More than 3 times

  1. Who makes most of your hotel reservations?

I book the hotel myself

Someone books it for me (disqualify)

  1. How often do you use the Internet?

Less than 2 hours per day

2-5 hours per day

More than 5 hours per day

  1. What do you mainly use the Internet for? Circle all that apply

Entertainment

Information / Research

E-Mail

Social Networking

Other______

  1. How many nights have you spent in a hotel in the last six months? ______
  1. How many of these nights were for

Business______

Pleasure______

  1. What websites have you used the most to book hotel reservations?
  1. Finish the following sentence: When it comes to providing my credit card number online:

I do not mind

I will do it if absolutely necessary

I don’t give out credit card information online

Appendix B: Pre-Test Questionnaire

Thank you for taking the time to participate in our usability test. Your participation will help to provide valuable feedback about your experience using a hotel’s website.

Before we begin, we would like you to fill out this brief questionnaire, so that we will have more information about your experience using the Internetand, more specifically, about your experience making travel arrangements.

Name: (please print): ______

  1. What features impress you the most when you use a hotel’s website?

a.

b.

c.

  1. What features frustrate you the most when you use a hotel’s website?

a.

b.

c.

  1. When booking a hotel room, what information is most valuable to you:

When booking a business stay:

When booking a vacation stay:

  1. Which of the following two typical booking activities do you prefer?

□ I use one or two preferred hotel websites that I’m familiar with

□ I browse different hotel websites to have as much information as possible

Appendix C: Post-task Questionnaires

Scenario 1: Look and feel/first impressions

Please rate the following:

Very Easy / Easy / Neither Easy nor Difficult / Somewhat Difficult / Very Difficult
Understanding the site navigation
Understanding the site terminology (words, links)
Understanding the first step you need to do to book a hotel room

Scenario 2: Book a room in Beijing for 2 people

Please rate the following:

Very Easy / Easy / Neither Easy nor Difficult / Somewhat Difficult / Very Difficult
Overall ease or difficulty of this task
Understanding the procedures of booking a hotel room
Understanding the search results page

What aspects of the site made this task easy for you?

What aspects of the site made this task difficult for you?

Scenario 3: Cancel Reservation

Please rate the following:

Very Easy / Easy / Neither
Easy nor Difficult / Somewhat Difficult / Very Difficult
This task was . . .

What aspects of the site made this task easy for you?

What aspects of the site made this task difficult for you?

Scenario 4: Look for hotel in Changchun

Please rate the following:

Very Easy / Easy / Neither Easy nor Difficult / Somewhat Difficult / Very Difficult
This task was . . .

What aspects of the site made this task easy for you?

What aspects of the site made this task difficult for you?

Scenario 5: Competitor site (elong.com)

Please rate the following:

Very Easy / Easy / Neither Easy nor Difficult / Somewhat Difficult / Very Difficult
This task was . . .

What aspects of the site made this task easy for you, particularly as compared to Holiday Inn China’s website?

What aspects of the site made this task difficult for you, particularly as compared to Holiday Inn China’s website?

Scenario 6 (optional): Holiday Inn U.S. site