ISO Intranet Usability Test Results
ISO Intranet Usability Test Results
Monday, May 7, 2007
Prepared by Amy L. Marr
Introduction
The following report presents the results of a usability test conducted over a period of five days during April and May 2007 among thirty (30) employees of the ISO at MassMutual. Results are analyzed and next steps recommended, including changes to the draft information architecture (IA) and a follow-up test.
The Test
The Enterprise Usability team assisted the information architect in developing and performing this usability test. The test itself consisted of an expectancy test and employed a reverse card sorting technique to determine whether or not placement of 25 content items within the proposed portlet structure groups the information in the way users expect and whether or not the organization maximizes the probability of users being able to find information.
Reverse card sorting
§ enables us to validate understand how 'real people' are likely to group items,
§ identifies items that are likely to be difficult to categorize and find, and
§ identifies terminology that is likely to be misunderstood.
Employees of ISO were selected to represent each department or area and were tested one-on-one. Comments were manually recorded and specific attention was paid to users’ comments regarding category (portlet) names they were unsure of or found confusing and why they felt certain content items fit within a particular heading.
Section One: Demographics
The first four questions presented during the test pertain to the person being tested and are designed to both put the tester at ease and to provides some idea of the level of understanding the test pool has of MassMutual, the ISO, the internet in general, and the current ISO sites in particular.
1. How long have you worked at MassMutual?
Years / Responses / %0-4 / 5 / 17%
5-9 / 7 / 23%
10-14 / 11 / 37%
15-19 / 2 / 7%
20-24 / 4 / 13%
25+ / 1 / 3%
2. What is your current position?
Job Groups / Responses / %Administrative Assistant / 2 / 7%
Assistant Vice-President / 1 / 3%
Business Consultant / 1 / 3%
Director / 1 / 3%
Financial Analyst/Salary Administrator / 2 / 7%
Information Security Specialist / 2 / 7%
Project Manager / 4 / 13%
QA Consultant / 1 / 3%
Service Owner / 2 / 7%
Systems Analyst / 4 / 13%
Systems Consultant / 1 / 3%
LAN Administrator / 1 / 3%
Systems Specialist / 6 / 20%
Technical Consultant / 1 / 3%
Telecommunications Specialist / 1 / 3%
3. On a scale of 1 – 5, 1 being a novice and 5 being very knowledgeable, how would you rate your technical knowledge or comfort level with the internet?
Level / Responses / %1 (novice) / 0 / 0%
2 / 0 / 0%
3 / 3 / 10%
4 / 15 / 50%
5 (very knowledgeable) / 13 / 40%
4. How often do you use the ISO site or ISO department sites?
Frequency / Responses / %Daily / 8 / 27%
2-3 times/week / 4 / 13%
Weekly / 6 / 20%
2-3 times/month / 1 / 3%
Monthly or Infrequently / 6 / 20%
Never / 5 / 17%
Section Two: Heading Names
In this portion of the test, participants were asked what they would expect to find under the headings given, within the context of an intranet site for ISO employees. As this question is largely qualitative, the comments listed are representative of the testing group – answers only given by one or two participants are not included.
Overview
· General introduction to what ISO is, how it supports the business
· Mission, goals, values
· What each group in ISO is responsible for; who to contact for services
· Org charts
· Where to find things in the site (site map
· Overview of information in the site
· Too generic of a heading; don’t know
General Observations: Many participants struggled to come up with answers for this, primarily due to the lack of site context (which comes up frequently throughout this report).
In the draft IA, “Overview” is the name of the first portlet, which is standard for sites within MMInfo. However, the testers did not see the draft, and therefore have no context in which to place the portlet name, or even the fact that it is a portlet. Many testers assumed that these headings are “menu items” that would need to be clicked in order to view content, despite the fact that all were told about the new ISO site being part of MMInfo. Either the members of the testing group don’t use MMInfo frequently enough to know the general look and feel of it, or they simply as a whole did not consider it during the test.
News
· What’s happening in ISO
· Promotions, moves, new hires
· Technical news
· Enterprise news impacting ISO
· Updates from ISO management (Mike Foley specifically mentioned twice)
· ISO events
· Project news
· ISO Newsletter
General Observations: A few people mentioned “enterprise news and events”
Policies & Procedures
· NOT HR-related
· ISO policies broken down by subject (development, project management, security)
· Best Practices (for developers, project managers)
· IT QMG, Compliance, SOX
· ISO Methodologies, standards
Systems & Tools
· Applications and what they do
· Applications supported and/or built by ISO
· Specifically SAP, Maestro, Aventail, Symantec, Polaris
· Resource planning tools
· Time entry
Development Best Practices
· Not sure what would go here
· “How do I” content for developers
· J2EE and like standards
· Development lifecycle information
· Best practices broken down by language/system
· Methodologies, requirements for developers
· IT QMG
Enterprise Governance
· Don’t know, not sure what that means
· IT QMG, Quality Control, project management, SOX
· Compliance
· Standards
· Who’s who in QMG, who to contact
· Same as “best practices”
· Forms
General Observations: Many testers paused before answering this question, and several asked for a definition of the label “Enterprise Governance,” but none could be given during the test. This particular label was tested at the request of Cristine Arnold and Laura Meacham, who run the current IT QMG site and have requested the name be changed during the site migration.
Learning Resources
· University @M
· Books 24x7
· Skillsoft
· Online classes, educational opportunities outside MM
· Technical Web sites, knowledge bases
Resources
· Not sure/No idea/Too vague
· Everything that didn’t fit somewhere else
· Same as Learning Resources - confusing
· Org charts, contact information
General Observations: This is the first of the blue portlet names and we see again the problem with a lack of context. Within MMInfo, this box generally houses links to key information or documents pertaining to the subject matter of the Web site. The general user is not aware of this definition, but understands it within the context of the site itself. One question that came up frequently had to do with the definition – does “resources” refer to people or ‘stuff’?
Related Information
· Not sure/No idea/Too vague
· Everything that didn’t fit somewhere else
· Same as Resources – confusing
· “Related to what?”
General Observations: Within MMInfo, this box generally houses links to key sites or documents external to the subject matter of the Web site. The general user is not aware of this definition, but understands it within the context of the site itself.
Section Three: Reverse Card Sort
In this portion of the test, participants were given a stack of 25 content items (listed below) and were asked to place each item under the heading they felt was the most appropriate. The test required that a) all items be placed, b) each item had to be under a single heading, and c) that participants talk aloud so that their thoughts (for example, on whether an item could go under two headings) could be recorded.
The content items were:
Project/WBS Elements List / A&D ToolkitUniversity @M / J2EE Technologies
Books 24x7 / Application Migration
Telecommunications / IT Quality Checkpoints
Information Security / IT Standards
IT Service Center / Component Database
ISO Values & Expectations / Building Codes
Org Charts / Computer Password Security Standards
Project Updates / Removing Equipment from the Company
Change Management System / Acceptable Use
NetTracker / Maestro/Tivoli
The quantitative results of the reverse card sort, as well as a summary of key qualitative results (comments) can be found on the next page.
Section Four: Conclusions & Next Steps
One of the major drawbacks of this particular test was the lack of context. Because testers couldn’t visualize the portlet structure of MMInfo, or in fact did not understand that the headings would be portlet titles, the Resources and Related Information portlets simply did perform well in the test. Within the context of the site, however, these headings are meaningless to the end user – they’re simply containers for helpful and related links.
Based on the results, no major changes to the information architecture that was tested are recommended. Some of the content items that scored high in areas other than the ones recommended may also be appropriate to included in other areas of the site. However, the basic categories should remain the same. The sole exception is Enterprise Governance, a label which is simply unfamiliar to the majority of the testers, and caused confusion. Laura Mecham’s group will take steps to communicate the new name prior to site launch, and the wording formerly IT QMG will also be used to assist users in finding the content in the short term.
A second round of usability testing is recommended, and should include:
· A task-based test, which asks users to tell the tester how they would perform a particular task (where they would click, etc.), and
· A categorization and naming exercise, where users would be asked to categorize the various content items under Systems and Tools and Policies and Procedures, to see what sort of organizational hierarchy and naming convention would be most useful to them.
Page 1 of 7
ISO Intranet Usability Test Results
Card Title / Overview / News / Policies & Procedures / Systems & Tools / Development Best Practices / Enterprise Governance / Learning Resources / Resources / Related Information / CommentsProject/WBS Elements List / 7% / 4% / 0% / 14% / 0% / 14% / 0% / 46% / 14%
University @M / 0% / 0% / 0% / 0% / 0% / 0% / 100% / 0% / 0%
Books 24x7 / 0% / 0% / 0% / 0% / 0% / 0% / 100% / 0% / 0%
Telecommunications / 5% / 5% / 9% / 36% / 0% / 0% / 0% / 32% / 14% / Many testers didn't realize this is its own site.
Information Security / 10% / 0% / 31% / 7% / 0% / 10% / 0% / 34% / 7% / Many testers didn't realize this is its own site.
IT Service Center / 10% / 0% / 0% / 17% / 0% / 0% / 0% / 66% / 7% / Many testers didn't realize this is its own site.
ISO Values & Expectations / 52% / 0% / 31% / 0% / 10% / 7% / 0% / 0% / 0%
Org Charts / 55% / 3% / 0% / 3% / 0% / 7% / 0% / 31% / 0%
Project Updates / 61% / 0% / 6% / 6% / 8% / 0% / 14% / 3%
Change Management System / 0% / 14% / 55% / 0% / 14% / 0% / 17% / 0%
NetTracker / 0% / 0% / 90% / 0% / 0% / 0% / 7% / 3% / Many testers didn't know what this is; guessed correctly
Maestro/Tivoli / 0% / 0% / 93% / 3% / 0% / 0% / 3% / 0% / Many testers didn't know what this is; guessed correctly
A&D Toolkit / 0% / 0% / 67% / 13% / 7% / 13% / 0% / 0% / Majority didn't know what this is; guessed incorrectly
J2EE Technologies / 0% / 4% / 61% / 25% / 0% / 7% / 4% / 0% / Majority didn't know what this is; guessed incorrectly
Application Migration / 0% / 12% / 15% / 15% / 45% / 6% / 0% / 3% / 3% / Many testers didn't know what this is; guessed correctly
IT Quality Checkpoints / 0% / 3% / 0% / 24% / 62% / 0% / 0% / 0% / Most weren't sure what E.G. meant; guessed
IT Standards / 4% / 0% / 39% / 0% / 32% / 21% / 0% / 4% / 0% / Most weren't sure what E.G. or IT Standards meant; guessed
Component Database / 3% / 0% / 0% / 69% / 3% / 0% / 0% / 24% / 0%
Building Codes / 0% / 0% / 23% / 12% / 31% / 27% / 0% / 8% / 0% / Majority didn't know what this is; guessed incorrectly
Computer Password Security Standards / 0% / 0% / 83% / 0% / 10% / 7% / 0% / 0% / 0%
Removing Equipment from the Company / 0% / 0% / 100% / 0% / 0% / 0% / 0% / 0% / 0%
Acceptable Use / 0% / 0% / 76% / 0% / 14% / 7% / 0% / 0% / 3%
Page 1 of 7
ISO Intranet Usability Test Results
Section Five: Unrelated Comments & Suggestions
Several testers provided unsolicited comments and suggestions, the most popular of which are listed below:
· The site needs to have a better search tool than the one MMInfo currently has.
· The site should have a site map.
· The site should include a list of the departments, what they do, and who to contact.
· Org charts are in SAP – why are they listed separately?
· I use the My Links feature of MMInfo frequently
· Would like to see some “industry news” in the News section
· Used to looking for things by department; this is very different
· “Enterprise Governance” seems like it doesn’t belong in ISO (enterprise-wide?)
· “Systems & Tools” – would be nice to have a list of each one and what it is/does
Page 1 of 7