Heuristic Evaluation on Proactive Website Page 1 of 37

Heuristic Evaluation on Proactive Website Page 1 of 37

Usability Aspect Report

Beran, Lynn; Ou, Jinghua;Xu, Yun; Zhuang, Zhong (names are in alphabetical order)

IS 2470 – Interactive System Design

March, 2010

URL:

Proactive is an adaptive job search system developed mainly to help students of University of Pittsburgh in their job search. We chose this website to evaluate because we have great interest in job searching.

According to the project documentation,Proactive tried to solve an observed dilemma in exiting job searching websites. The dilemma is on one hand, in order to cope with information overload problem, personalized information retrieval technology focusing on user’s exact needs is used; on the other hand, diversified user populations expect more ways to thoroughly explore various aspects of the information. Proactive is trying to explore a range of innovative personalized access approaches to meet these two contradicted requirements at the same time. Proactive attempts to bring together a range of technologies. The focus and the innovation of their work are twofold. First, they attempted to provide a true integration of these approaches where one approach can capitalize on the others and where weak sides of one approach are balanced by strong sides of others. Second, they attempted to integrate several personalization approaches in one systems using the ideas of recommender systems (Schafer, et al., 1999) and adaptive hypermedia (Brusilovsky, 2001).

However, when we started on this evaluation project four weeks ago, we realized the website does not have any job listings. As a result, we were not able to evaluate any functionality described in their project documentation. Nevertheless, we were able to findnearly 40 items from user interface aspects.

Project Completion Schedule

• Each member evaluated the website independently prior to the group meeting on 2/18.
• Lynn Beran wrote her observations, Yun Xu, and Zhong Zhuan added theirs. These constitute the 1st, 2nd, and 3rd versions of this evaluation paper on the group page on course web.
• Jinghua Ou added her observations; edited some items, added screenshots to, re-formatted, and re-ordered all items. Then she uploaded the 4thversion on 2/21.
• Zhong Zhuang added his commentary; Lynn Beran edited several items, renamed all items, and added background colors to all items’ titles to indicate good features by green and problems by red. These constitute the 5th and 6thversions.
• Lynn Beran made some minor editing,wrote the introduction and 2 summaries. Then she uploaded the 7thversion on 2/26.
• Jinghua Ou and Zhong Zhuang will present this evaluation project at class on 3/4.

Summary 1
Description / Heuristic / Good feature / Problem / Problem Severity
Register page (HE1 to HE4, 3 problems and 1 good feature)
HE1: The register page doesn’t provide a way back to the login page. / Provide clearly marked exits / / Severe
HE2: The error message is not user friendly and has a typo. / Good error messages / / Minor
HE3: Good error message in user registration / Good error messages /
HE4: After creating an account, the system doesn’t provide clear feedback that an account has been created / Provide feedback / / Medium
Loginpage (HE5 to HE6, 1 problem and 1 good feature)
HE5: The login page doesn’t provide assistance of recovering passwords. / Simple and Natural Dialog / / Severe
HE6: A “loading” message appears when an action takes a longer time. / Provide feedback /
Preferencepage (HE7 to HE16, 10 problems)
HE7: - The first instruction sentence on the top of preference set up page is misleading. / Simple and Natural Dialog / / Medium
HE8: No clear instructions on how to move on after adding preference. / Provide clearly marked exits
Simple and natural dialog / / Medium
HE9: Irrelevant instructions appear before the corresponding features appear. / Simple and natural dialog
Minimize user memory load / / Medium
HE10: Section title “Required information to activate Proactive” is confusing. / Speak the user’s language / / Minor
HE11: Misleading instruction in career category guiding information / Prevent errors / / Minor
HE12: Section title “Optional Information” is confusing and contradicting to the actual behaviors of the page. / Simple and Natural Dialog / / Severe
HE13: User can choose either PreferredState and City or PreferredMetroCity but the page does not indicate this. / Simple and Natural Dialog / / Minor
HE14: Adding required information works different than adding optional information. / Be consistent
Simple and natural dialog / / Medium
HE15: The flexibility options in preferred state and city selection is confusing. / Speak User’s Language / / Medium
HE16: No warning before session expires. / Simple and Natural Dialog
Provide Feedback / / Severe
Summary 1 (continue)
Description / Heuristic / Good feature / Problem / Problem Severity
Homepage (HE17 to HE23, 6 problems and 1 good feature)
HE17: The Home link on the menu does not show when user is on Home page. / Minimize user memory load
Be Consistent / / Medium
HE18: Use similar icons for different The description in home page is confusing. / Minimize user memory load / / Minor
HE19: The dropdown box before the search text field has no label. / Simple and natural dialog / / Medium
HE20: Category cannot be defined in job searching. / Speak the user's language / / Medium
HE21: Nothing happened after clicking on the Matched to preference tab on homepage / Prevent Errors / / Severe
HE22: The link “Visualize these list” doesn’t tell user what it will do. / Speak the user’s language / / Minor
HE23: Different tab colors on Home page match the background color when user clicks on the tab. / Minimize user memory load /
Advanced searchpage (HE24 to HE28, 4 problems and 1 good feature)
HE24: The longest option for Posting Date is two weeks and the default option is not clear. / Simple and Natural Dialog / / Severe
HE25: The options for the dropdown list of Experience is ordered alphabetically and includes shorten names. / Simple and Natural Dialog
Speak the user’s language / / Minor
HE26: The options for the dropdown list of Education Level is ordered alphabetically. / Simple and Natural Dialog / / Minor
HE27: Good error messages. / Good error messages /
HE28: The search criteria are not displayed with the search result. / Minimize user memory load
Simple and natural dialog / / Medium
Otherpages (HE29 to HE31, 3 problems)
HE29: The enter feedback link is difficult to find and its label doesn’t match the action exactly. / Speak the user’s language
Simple and natural dialog / / Minor
HE30: After clicking the “Proactive Home” link from Help page, user is logged out and asked to login again. / Be Consistent / / Minor
HE31: The first line in My saved jobs page is not consistent with other pages. / Be Consistent / / Minor
Summary 2
Heuristic / Number of problems / Sub total problems / Total good features
Severe / Medium / Minor
Be Consistent / 2 / 2 / 4
Good Error Messages / 1 / 1 / 2 / 1
Minimize user memory load / 3 / 2 / 5 / 1
Prevent Errors / 1 / 1 / 2
Provide clearly marked exits / 1 / 1 / 2
Provide Feedback / 1 / 1 / 2 / 1
Provide Shortcuts
Simple and natural dialogue / 4 / 5 / 4 / 13
Speak the user's language / 2 / 4 / 6
Total: / 7 / 15 / 14 / 36 / 3

//start of register page

HE_register_1 -- Problem
Description
Register page – The register page doesn’t provide a way back to the login page.
HeuristicsViolated
Provide clearly marked exits
Evidence

The “Cancel” button on the page only clears all fields; it doesn’t take user back to the login page. Under this circumstance, user has only one choice: submit the registration form.
Explanation
It is natural for user to return to the previous page if he/she does not want to register. Although most users may register as a member, there should be an exit for those users who might change their minds. User shouldn’t be “forced” to register.
Severity
Severe. There should be an obvious exit. Especially in interactive system design. This feature is as important as a key to a door.
Possible Solution
Use the “Cancel” button to send user back to the login page. It’s clearer to change the button to say “Cancel Registration.”
Observer
Jinghua Ou, Yun Xu
HE_register_2 – Problem
Description
Register page – The error message is not user friendly and has a typo.
HeuristicsViolated
Good error messages
Evidence

1. Spelling error for Please
2. Login Password is in upper case

Explanation
It is not good to have spelling error; user may take time to find out what it is. Upper case words take more time for user to understand and they seem like shouting at users.
Severity
Minor. User usually will click OK and continue
Observer
Yun Xu
HE_register_3 – Good Feature
Description
Register page - Good error message in user registration
HeuristicsSupported
Good error messages
Evidence

In user registration, system can detect same user name and prevent registration with a name already in use.
Explanation
Prevent the same user name registration is common in a web interface. In this error message, user is notified about the problem and a way to correct it.
Benefit
User will know immediately how to fix the problem.
Observer
Yun Xu
HE_register_4 -- Problem
Description
Register page and Preference page –After creating an account, the system doesn’t provide clear feedback that an account has been created.
HeuristicsViolated
Provide feedback
Evidence
After user clicks the “Create Account” button, the Preference page, a very complicated page, appears.
Explanation
Registering for an account is an important user activity so the system should provide clear feedback after the action. With proper feedbacks on previous actions and guidance on what to expect next would help user understand an unfamiliar process more easily.
Severity
Medium. Showing the Preference page can work as an implied feedback. However, it requires users to figure out on their own.
Possible Solution
Provide a message saying “An account has been created for you. Please set up your job searching preferences now” on the Preference page when user first registers.
Observer
Jinghua Ou

//end of register page

//start of login page

HE_login_5 -- Problem
Description
Login page - The login page doesn’t provide assistance of recovering passwords.
HeuristicsViolated
Simple and Natural Dialog
Evidence

There is no clear instruction on how to recover password on the login page. The instruction to How to Recover Password is listed in the help section, which is available only after login. User has to email the System Administrator to recover password.
Explanation
Although it has a “Contact System Administrator” link on the login page, it is not natural for user to know they should email the system administrator to retrieve password. It is very uncommon to list the instruction to How to Recover Password in a different page rather than the login page where the password is required.
After sending email, user has to wait until the administrator replies.
Severity
Severe. It is common to forget password. Without a way to retrieve password, user has to register to gain access again.
Possible Solution
A link “Retrieve Password” should be added to login page. There should be away for user to retrieve password without human intervention.
Observer
Lynn Beran
HE_login_6 – Good Feature
Description
Login page – A “loading” message appears when an action takes a longer time.
Heuristics Supported
Provide feedback
Evidence

After user clicks the “Log In” button, it takes a few seconds before getting to the home page. The “Loading…” message shows up at the top right corner indicating the system is processing the request.
Explanation
Without the “Loading…” message users might wonder why nothing happens after they click the “Log In” button. They might get confused and click the “Log In” button again and again. The “Loading…” message provides immediate feedback so users will know the system is working and they should wait.
Benefits
Users won’t get frustrated when the system takes longer than expected to perform a task.
Tradeoffs
The “Loading…” message might make the website look slow. It would be better to improve the performance of the action so users can see final result of an action quicker.
Observer
Jinghua Ou

//end of login page

//start of preference page

HE_preference_7 -- Problem
Description
Preference page - The first instruction sentence on the top of preference set up page is misleading.
HeuristicsViolated
Simple and Natural Dialog
Evidence
“Select your preference, then you are able to see jobs matchd to your preference. “
By reading this sentence, I expect to be redirected to the job list matched to my preference after I click the “Add Preference”button at the end of the page. However, after the button is clicked, the page stays there.
Explanation
User gets very confused about how to see the job list.
Severity
Medium. Although users may be able to figure out that they should click “Matched Jobs to My Preference” to see the jobs list, it is very annoying to get confused with the basic functionality of a website.
Possible Solution

1. The sentence could be changed to:

“Select your preference, save it, then click the link Matched Jobs to Your Preference to see the job list. “

1. The button at the bottom should be changed to “Save.”The page could be redirected to the result page after the button is clicked.

Relationships
HE5No clear instructions on how to move on after adding preference
Observer
Lynn Beran, Yun Xu
HE_preference_8 -- Problem
Description
Preference page - No clear instructions on how to move on after adding preference.
HeuristicsViolated
Provide clearly marked exits
Simple and natural dialog
Evidence


There is only one button at the end of the Preference page “Add Preference.” It only updates the preferences settings without sending user anywhere. The link to the “Matched Jobs” page resides at the top of the page together with other links.
Explanation
After adding goal and preference, user would want to see jobs matched to the preference right away. Also, user usually works through a page from top to bottom. After searching the entire page I found upper right a link marked: Match jobs to my preference. For an interactive system, putting exits near the last action position is very important. It matches the flow of user activities and saves user time so he/she does not have to scroll up and down to locate the exit.
Severity
Medium. User can find the exit: Match jobs to my preference. However, job search takes a lot of time and developer should make it efficient to commit every action.
Possible Solution
Set up a button Match jobs to my preference underneath the apply preference button and link it to the result page.
Relationships
HE4The first instruction sentence on the top of preference set up page is misleading
Observer
Yun Xu, Zhong Zhuang
HE_preference_9 -- Problem
Description
Preference page –Irrelevant instructions appear before the correspondingfeatures appear.
HeuristicsViolated
Simple and natural dialog
Minimize user memory load
Evidence

The second and the third instructions appear at the top of the page when user first comes to this page. At this time, no preferences have been selected there is no “x” icons nor flexibility options.
Explanation
This is the first page a new user sees after registering. A new user will try to read all instructions at the top of the page before performing any actions. However, without the corresponding features on the page, user would have a hard time understand the instructions. For example, user wouldn’t know where to expect an “x” icon. By the time a preference has been selected and an “x” icon appears, user’s eye focus has moved out of the instruction area and would have to recall what the instruction says or have to go back to read the instruction again.
Severity
Medium. Users can figure it out at the end. However, this increases user’s mental load and possibly overwhelms first time users.
Possible Solution
Show the instructions only when the related features appear and put them closer to the related features.
Observer
Jinghua Ou
HE_preference_10 -- Problem
Description
Preference page –Section title “Required information to activate Proactive” is confusing.
HeuristicsViolated
Speak the user’s language
Evidence

It’s not clear what “to activate Proactive” means.
Explanation
The message might be very clear to the developer of the website, but from a user’s point of view the word “activate” can mean many things: to activate the website, to activate the user account, or to active the job search functions.
Severity
Minor. Although users might be confused by the message, they probably will ignore it.
Possible Solution
A more user-friendly message can be used, such as “Please answer the following required information for Proactive to generate a list of jobs matched to your preferences.”
Observer
Jinghua Ou
HE_preference_11 -- Problem
Description
Preference page – Misleading instruction in career category guiding information
HeuristicsViolated
Prevent errors
Evidence

It says: double clicking a category, you can choose your specific job. Actually by clicking a category user can choose a specific job, no matter double-click or single-click.
Explanation
Instructions should match the behaviors of the corresponding features. Single-clicking is easier than double-clicking so the system shouldn’t make user double-click when single-clicking works. Although this is a simple syntax error, it might result in doubtsabout the website quality.
Severity
Minor. It is a small error and does not break the functionality. User will only consider this to be carelessness.
Possible Solution
Remove the word “double” from “double click”
Observer
Yun Xu
HE_preference_12 -- Problem
Description
Preference page – Section title “Optional Information” is confusing and contradicting to the actual behaviors of the page.
HeuristicsViolated
Simple and Natural Dialog
Evidence

A * appears next to the “Optional Information” title. Actually the information in the section is not completely optional. Users have to choose at least one preference, otherwise they will get an error message: Please choose your preference.
Explanation
There are some norms that have been accepted by normal computer users, and putting a * next to required fields is one of them. It is easy to confuse user with both * icon and text optional. Also, the word “optional” means none of the fields are required, and this contradicts to the actual behavior – requiring at least one field answered.
Severity
Severe. User cannot add preference if they leave all optional information blank. Besides, they may make mistake to both required information and optional information.
Possible Solution
Delete the title * from this title and change the title to say “Please select at least one of the following preferences.”
Observer
Yun Xu, Zhong Zhuang
HE_preference_13– Problem
Description
Preference page - User can choose either PreferredState and City or PreferredMetroCity but the page does not indicate this.
However, this is an intermittent issue. Sometimes it cannot be re-produced.
HeuristicsViolated
Simple and Natural Dialog
Evidence

After user has chosen State and City, if user chooses PreferredMetroCity, the selected State and City become un-selected.
Explanation
Only after user has selected both, user realizes he/she could choose either instead of both. This is a waste of user’s time.
Severity
Minor. Although this is a nuisance, it is not a big deal. Since there are much more severe issues on many today’s online applications, users have become very tolerant.
Possible Solution
Developer could simply add the word “or” between those two selections.
Observer
Lynn Beran
HE_preference_14 – Problem
Description
Preference page –Adding required information works different than adding optional information.
HeuristicsViolated
Be consistent
Simple and natural dialog
Evidence

After user clicks a career category, the “Add My Goal” button appears for user to add career goals; however, for optional information, no button appears after an option is selected, rather user has to scroll to the bottom of the form and click the “Add Preference” button, which is visible all the time.
Explanation
The process of adding goals works pretty natural, the necessary action appears right next to the selection. After that user would expect the rest of the form works the same way. However, after selecting an optional preference, no button appears next to the field. User would have to search the page to find the correct button at the bottom.
Severity
Medium. The inconsistence of behaviors increases the learning time for users, but they can adapt to them eventually.
Possible Solution
Move the button “Add Preference” next to each optional preference field, and make it appear only when a new preference is selected.
Observer
Jinghua Ou
HE_preference_15 -- Problem
Description
Preference page - The flexibility options in preferred state and city selection is confusing.
HeuristicsViolated
Speak User’s Language
Evidence

By selecting In the neighbor city or state option, then system fails to specify which cities or states.
Explanation
Users may get confused by selecting this, because the scope of neighbor city or state can be very big.
Severity
Medium. User may lose this flexibility of choosing neighbor city or state.
Possible Solution
Specify how many miles from the selected city or state.
Observer
Zhong Zhuang
HE_preference_16 -- Problem
Description
Preference page - No warning before session expires.
HeuristicsViolated
Simple and Natural Dialog
Provide Feedback
Evidence
After I chose all my preference and clicked the Add Preference button, I was logged out and redirected to login page without warning.
Explanation
User could waste time on selecting all the preference without being able to save it.
Severity
Severe. To get matched jobs to user’s preference is the number one purpose of using this application.
Possible Solution
Developers could issue a popup window prior to session expiration and ask users if they need more time on this page. If user chooses yes, session should be automatically reset.
Observer
Lynn Beran

//end of preference page