Art Museum Browser
CarnegieMellonUniversity / July 21st, 2004
Introduction
In this second round of usability testing, our Horizontal Cityscape version was compared to Berkeley’s Flamenco Browser. Overall, the results were encouraging; significant progress was achieved since the last round and the gap between the established Flamenco browser and our own nascent version was perceived to be slight, both qualitatively and quantitatively. However, there are still some critical issues, both at a conceptual and an implementation level, that seem to be thwarting our users’ abilities to intuit and therefore use certain key features of our browser.
Sample Group
This round consisted of four users, each of whom are CMU students in computer science related fields, and therefore could be considered to be extremely technically savvy. As a result of this bias, it may be wise to assume that the “typical” user would have more difficulty, on average, than the members of this sample.
Quantitative Analysis
The quantitative scores were very positive in this round, and were a considerable improvement from the previous round. In each of the four categories, the scores were better by at least 2 points. Further, the Horizontal layout was perceived as only slightly worse than Flamenco in three of the four categories, and slightly better in “Nice to Look At”.
Easy to Understand / Easy to Use / Nice to Look At / Would Want to UseHorizontal / 7 / 7 / 8 / 6
Flamenco / 8 / 8 / 7 / 8
Average Subjective Rating, 1-10 scale
The distribution for the Horizontal Layout was negatively skewed, such that 3 of the users scored the browser very highly, but one scored it very low.
Task Analysis
Task 1: Find 4 artifacts that depict battle scenes in Europe in the 17th Century
The first task, a directed browsing activity, was completed successfully by all users, but in varying amounts of time. In cases where the user understood the relationship between the different galleries (i.e. that clicking a category in one gallery would refilter all other galleries as well), the user quickly and easily narrowed the data set to find the appropriate images. Unfortunately, there was only one such case. In the other three cases, the users were simply unable to combine multiple filters to carry out the task until they were explained this conceptual relationship. As will be explained later, this dissillusion regarding inter-gallery filtering probably represents the largest usability hurdle for the Cityscape Browser.
Task 2: Find Picasso’s “Little Dove” lithograph.
The second task was a direct search given 3 clues: “Picasso”, “Little Dove”, and “lithograph”. Users all successfully completed this task in relatively similar amounts of time, however each did so using a slightly different approach. Two users relied heavily on the text search feature, querying on both “dove” and “Picasso”. Another combined multiple category filters (“Picasso”->”Birds”->”Dove”) to narrow the data set to a manageable size, and then searched through the remaining items. The final user used only the Artist gallery, and manually searched through 782 works.
Task 3: Find 3 interesting artifacts that you would want to see if you visited the SF Fine Arts Museum.
The third task was an open-ended browse activity. When this task was given in the first round of usability tests, users were so frustrated with the interface that few earnestly looked for interesting works – the others didn’t even try. This time, however, each user used the interface and found at least 3 works that they found appealing (interestingly, most used their home country or region as a starting point).
Qualitative Analysis
Major Issues
1) Users don’t immediately grasp theconcept of filtering between galleries.
This is without a doubt the biggest point of confusion for users when they first interact with the Cityscape Browser. When they click on a category link, they unanimously think that they are entering that category in that gallery, and will be shown the images that reside within it. If asked, “what do you think will happen to the other galleries if you click that category”, most say “nothing”. This is a critical problem, because without this understanding users are not able to combine category filters together for complex searches, and are often even tentative to click on any category filter alone (they are apprehensive about just “diving in”).
The reason users do not intuit this relationship, I believe, is due to three things: (1) this concept is different from all other searching conventions users have confronted on the web, (2) the current layout defies a basic assumption in information architecture, and (3) the terminology we use contradicts the inter-relatedness of categories.
Regarding the first point, all major directory sites on the web that present categories in a similar fashion use a drill-down approach to search. By clicking on one category, you are essentially drilling down into it (i.e. taking a step into a world of information defined by that category). It initially seems anathema to a user that clicking on one category would have anything to do with another unrelated category in a different gallery, since directory sites never combine category filters.
With respect to the second point, I believe that the layout was structured in a way to imply the concept of containment, which may be inappropriate for this browser. Take for example the following basic layout below:
Although there is no semantic content in the picture above, there are certain relationships we can expect to derive between the blocks A, B, C, and D. The most relationship is that of independence, and we derive it based on containment. Both C and B are contained within A and have no points of intersection. Therefore, it would be perceived that these two blocks are independent of one another – what happens in one should not affect anything in the other. However, in Cityscape, despite using this layout paradigm, the concept of independence does not hold.
Lastly, the terminology “Gallery” seems to reinforce the independence between different types of categories. In the real world, you would not expect one physical gallery to be affected when you step into another. Using similar logic, users derive expectations of what will happen to one virtual gallery when you click on a link in another.
2) Interface is initially intimidating.
Three of four users voiced concerns, unprompted, that the interface was “confusing” or that they didn’t have a grasp of “what was going on” within the first 2 minutes of the study. In one case, the user complained that the images were distracting, and suggested that they shouldn’t be shown in the initial screens. In other cases, users scrolled up and down through the interface trying to apprehend some order for the tool. This initial intimidation effect of the browser, I believe, will be a critical hurdle for any type of adoption. Once the users become accustomed to the interface, however, they are more comfortable with the “chaos”, and sometimes even prefer it.
The confusion, I believe, is a result of a feeling of “information overload” or “over-stimulation”. When users first enter the browser, they are hit with a packed screen full of different images and text. Initially, they seem to scan the page for some type of anchor point, a reasonable place to start. If they find one, it is typically the Text Search box, and they will try to use this exclusively. Most times, however, they are consumed by all the text and all the images, and have a difficult time creating a conceptual model for the tool.
Flamenco, on the other hand, offers users a very simple first view of the interface, so that users are at least not initially turned off (however, as soon as they hit the second or third screen, they are!).
It may be useful to create an “opening game” for the Cityscape Browser, or alternatively have an option for showing preview images and default it to not show them for the initial screen.
3) Showing abundant information.
A better paging system needs to be designed for navigating through large amounts of categories or images within a gallery. Users needed to heavily rely on manual scanning, which is time consuming, error-prone, and frustrating.
4) Using the browser’s Back button to remove filters.
Unfortunately, filters are stored in the session, so are preserved even when the user uses this Back button.
Minor Issues
1) Text Search should be tokenized.
The text search feature should allow for multiple words, and then split them appropriately. Further, it should search not only item names, but facet names as well.
2) Gallery controls are confusing, and seldom used.
The drop-down controls for the galleries cause confusion, and are rarely used. Users have never used the minimize, maximize, or move features (in either round), and rarely used the sort or view features (and if they did, they only tried it once). We might want to apply the usability principle “make the easy things easy, and the hard things possible”, and either hide these controls or remove them completely.
3) Images are too small..
Both in the detail view and the browse view, the images are too small for proper inspection.