VIRTUALLY THERE1
Virtually There:
An Evaluation of the Usabilityof American Art Museums’ Digital Collections
Carolyn English
17:610:553:90
School of Communication and Information
Rutgers University
April 27, 2015
Abstract
American art museums display some of the most beautiful works of art on their own walls, but how do their virtual collections look? In this paper, the usability of four museums’ digital collections—The Art Institute of Chicago, Museum of Fine Arts, Boston, National Gallery of Art, and Philadelphia Museum of Art—were reviewed, rated and compared to reveal some of the most user-friendly features of current digital collections. Specific elements of the user experience in this case relate to searching and browsing capabilities, metadata, images, social media, and help resources. Both qualitative and quantitative methods were employed to paint the most detailed picture of each collection. Efficient faceted searching and detailed images were some of the author’s favorite features, while poor navigation through search resultsand a lack of help resources were some of the most troublesome areas.
Virtually There:
An Evaluation of the Usability of American Art Museum Digital Collections
American art museums displaysome of the most beautiful works of art on their own walls, but how do their virtual collections look? Museums of the future will still be revered and visited for the masterpieces on display, but they will also be accessed virtually for detailed information and images of their entire collections, both on and off display. These virtual museums will need to satisfy a range of users—from travelers planning visits to those who wish to reflect on their visits to artists to students and educators. It is critical that they provide a user experience that satisfies people with a variety of abilities and intentions while also promoting the museum’s reputation through good design and an efficient interface.
The author was inspired by the Smithsonian Institute’s impressive 3D scanning project ( which will provide virtual visitors with as close to an actual physical experience as possible by intricately capturing its artifacts using 3D technology. Art museums likely have a long way to go before taking on a project like this, but they face the same future with the need to develop and promote a strong online presence. It is this prospect that calls for a review of the current usability of art museums’ digital collections.
In this study, four traditional American art museums’ digital collections were explored—The Art Institute of Chicago, Museum of Fine Arts, Boston, National Gallery of Art, and Philadelphia Museum of Art. Several specific factors relating to the user experience were described, quantified, analyzed and compared to determine strong and weak user experience features. The author walked through each website examining the overall design, like color scheme and interface, searching and browsing navigation, metadata, images, social media presence, and help resources. Certain elements, like number of advanced search fields, were measured, while other features, like quality of images, were subjectively quantified on a 5-point scale to provide a clear comparison of the databases. Commentary is also included to paint a picture of the usability of these four digital collections.
Literature Review
There has not been extensive research completed on this exact topic, but there are many discussions surrounding the concepts of usability, user experience, digital libraries and cultural heritage collections, social media,and visual resource management.
Before discussing the specifics of the user experience, we should fully understand the need for it in the art museums of today and especially the future. In a study of use policies on art museum libraries, Esther Roth-Katz (2012) stated that the benefits are two-fold, “Not only is there a belief that the public is entitled to access to resources, but that increasing awareness of these resources can be mobilized as a marketing tool for the museum itself.” But while such access is expected, “Art museums must consider the implications of allowing open access to valuable and fragile resources” (Roth-Katz, 2012). Such precious information has never been so freely available, and museums must adjust to the new infrastructure. Unfortunately, there has not been much formal research completed on the matter. Roth-Katz calls for more, which should ultimately benefit both the institutions and the users of their websites, so the author would like to contribute to the remote access conversation.
It is also important to understand who the users are. Pallas and Economides (2008) described the diversity of virtual museum visitors in their study of the evaluation of art museum websites in which they developed a framework to do so, “A wide number of users, from pre-schoolers to art historians and researchers, may visit digital museums.” Kris Wetterlund’s research from 2008 focused on bringing the art museum to the classroom. The development of their digital collections has been a welcome resource to educators. “Before the existence of digital images of works of art from art museums, teachers who wanted to use works of art in the classroom arranged field trips to the museum or were restricted to slides, overhead transparencies, or printed reproductions” (Wetterlund, 2008). So if educators are utilizing their collections, so might the students for assignments or for independent use should they be inspired by their education. Other users may include people planning a trip or revisiting a work of art, time period, or style of art from a previous visit.
Now users have been established, what they wish to do online also determines the requirements of the digital collection. A recent study exploring web searching behavior for cultural heritage institution visitors (Skov and Ingwersen, 2014) described four motivational categories of visitors to remote access visitors to the Science Museum in London in 1998:
- Gathering information for planning an upcoming visit to the physical museum;
- Self-motivated research for specific content information;
- Assigned research for specific content information; and
- Engage in casual browsing.
Again, the finding is that a digital collection must be comfortable for a novice user to browse, but rich enough to benefit a serious student or professor in the field.
This growth of virtual visitors might make one question the consequences to physical visitation. Wetterlund (2008) calms museum professionals’ worries by citing a study performed by the Minneapolis Institute of Arts that was funded by the Institute of Museum and Library Services, which “found that 78% of online visitors to their Web site had been to the real museum as well.” So if anything, a strong virtual presence only encourages visitation or enhances the overall experience by extending the access beyond the buildings’ walls. “Another study found that most visitors to the museum Web sites are scholars and teachers” (Wetterlund, 2008). Since many of the visits seem to have an educational focus, the need for a digital collection similar to a library is likely the experience these users are used to and would expect from a similar institution, because a museum is quite comparable to a library of art.
Interestingly, just like information literacy for librarians, museum literacy remains an issue when utilizing their digital collections, especially when it comes to K-12 instruction.This tells us that any supplemental information, particularly in a narrative form, would greatly enhance the experience for those who are not well-versed in technical art terms or have a pre-existing knowledge of the works. Wetterlund(2008) explained that museums have posted more educational resources on their websites.Seamlessly including supplementary information within the search database would enhance the user experience for both novices and experts. The need for museum or information literacy doesn’t just apply to users but museum professionals too. Paul Marty (2006) conducted interviews with 21 museum professionals to determine what skills were necessary to enhance their work. He discovered that “there is a growing sense among museum professionals that the skills taught in Library and Information Science (LIS) programs are increasingly relevant to their everyday needs” (Marty, 2006). This type of expertise would directly contribute to the type of information available in digital collections and their design, which is critical, since “the emergence of this new role has coincided with the growing belief that museum information resources should be as readily available as information resources are in most libraries” (Marty, 2006). This is why a review of art museums’ digital collections from the perspective of a budding information professional is relevant.But while this is simply a review of the current state of affairs, future direction should also be considered. Collections will only continue to improve.
Eventually, as technology is developed and more readily accessible by a range of budgets, museums will likely follow the Smithsonian in providing even more interactive and realistic virtual visit. Patel et al. (2005) described a new project called ARCO, or Augmented Representation of Cultural Objects:
It is notable that although many museums have now established an online presence on the Internet, currently this presence is almost invariably a 2D one; that is associated Web sites comprise 2D images and textual descriptions. ARCO [Augmented Representation of Cultural Objects] on the other hand recognizes that objects are 3D in nature, that they have a front and back, top and bottom, mass and volume. ARCO seeks to enhance the awareness and experience of cultural objects by providing technologies for creating 3D digital surrogates of artefacts and allowing user to interact with them.
The trend is catching on, so museums must be prepared to update their digital collections to become even more dynamic and user-friendly.
So, when user experience is mentioned, what exactly does it mean? Hariri and Norouzi(2011) performed an extensive literature review to determine specific criteria for evaluating digital libraries, which helped direct the author’s checkpoints. What is interesting is that they admit that specific identification of such information is lacking. At the same time, “the quality of user interface design has improved dramatically” (Hariri Norouzi, 2011). They listed 22 specific factors in order of usage in evaluations, of which 11 matched the purposes of this study (highlighted in bold font): navigation, searching, design, guidance, error management, presentation, learnability, user control, consistency, language, feedback, ease of use, match between system and the real world, customization, user support, user workload, interaction, compatibility, visibility of system status, user experience, flexibility, and accessibility.Xie (2006) also provided a detailed list of usability. Not only is there the idea of usability in general, but interface usability pertains specifically to search and browse, navigation, help features, view and output, and accessibility (Xie, 2006). Clearly, there are many factors that overlap and contribute to the overall experience.
Such factors also require some context within the concept of usability. Even evaluation itself requires context, as it can pertain to technical or conceptual ideas. The question could be posed as whether or not a system performed a certain function, how it performed it, or how its performance affects its user. This is an idea of a user-centered evaluation versus a system-centered evaluation. Saracevic (2000) provided two questions to demonstrate the difference. A user-centered evaluation might ask, “How well does a given interface provide and support access, searching, navigation, browsing, and interaction with a digital library?”, whereas a system-centered evaluation might ask, “How well is the collection or information resources selected, represented, organized, structured, and managed?” He encourages the use of both, and for this, both quantitative and qualitative functionswill be examined so both areas are incorporated into the overall comparison.
Another layer in the user experience is the user’s point-of-view. Xie(2008)reported on the user perspective for evaluating user experience and stated that, “Digital libraries are designed for users to use. However, most of the research on evaluation of digital libraries has applied criteria from researchers themselves.” While this study is performed by a student of library and information science, it could also be considered a user’s perspective as someone who is a museum visitor (albeit, infrequent) and not highly practiced in the area of website evaluation, which matchesa novice user profile. While the evaluation criteria is quite specific, the author will also note general sentiments towards each collection. For the purposes of formal research, each carefully selected factor will be explored, but it does not require a professional to understand them. “The best way to evaluate digital libraries is to actually use them” (Xie, 2008).This study could be considered an inspiration for others to simply use websites and take notes. If specific criteria are selected, it becomes easier to compare strengths and weaknesses. Ultimately, each element builds up to an overall opinion, but many times usability is subconsciously decided when users either like or do not like their experience on a certain website.
Usability specific to visual resources differs from a traditional library because of the nature of the content. Zhang, Liu, Li, and Zhang (2013) referred to a study by Yee et al. in 2003 about faceted metadata in image searching, “Overall the participants preferred more faceted metadata display, and evaluated it as more informative, flexible, and easier to use.” Faceted searching would give specific control to the users to pick and choose select metadata from which to expand or eliminate search results. Another area of control for users applies to the look of the interface itself. In another study mentioned by Zhang et al. (2013), in 2005, Chen, Magoulas and Dimakopoulos“suggested a flexible interface (e.g. switching visual cues, offering successive options) to accommodate the preferences of users with different cognitive styles.” This would take a lot of planning and sophistication to create such a dynamic website, but it would ultimately provide a user experience that’s practically guaranteed to satisfy all users.
Menard and Smithglass(2014) also addressed issues in regards to access to digital images with the intention of creating bilingual interfaces. Unfortunately, there are many studies about general behavior, but no specific studies on current systems, so this evaluation should begin to fill in the gaps. They did note that “Museums offering images online structure the information almost exclusively according to collections and using traditional descriptive methods. We suggest this is one reason for a high level of consistency between museum websites in terms of search functionalities” (Menard Smithglass, 2014). This study will reveal that while there are basic structures in common with the following four museums, there are many other subtleties that do affect the ultimate user experience of each collection.
Metadata is also facing changes with multiple sources of information in the advent of social tagging. This allows users to provide their own metadata in addition to the descriptive metadata provided by museum professionals. In a study focused on folksonomies in art museums, the authors concluded that a blend of professionally provided metadata with user-supplied tags created the best atmosphere for navigating through images (Boston-Clay, Mahoui,Jaebker, 2013). To understand what the user-supplied information is, Boston-Clay,Mahoui, and Jaebker (2013) explained that “tagging is commonly referred to as ‘social tagging’ whereby typical users describe and classify resources in a shared environment.” While metadata is critical in terms of providing accurate information about works of art, when it is complemented with keywords and terminology more broadly understood by the average user, the search process is enhanced. What will be interesting to note is whether or not these highly prestigious institutions allow users to interact with their works.
Methodology
Four museums were selected for this study—The Art Institute of Chicago (AIC), Museum of Fine Arts, Boston (MFA), National Gallery of Art (NGA), and Philadelphia Museum of Art (PMA). After gathering a list of major art museums’ websites from across the country, candidates were narrowed down through quick browsing and searching sessions. Those with the most sophisticated websites and unique features were included. Two other museums were almost considered in the study—Los Angeles County Museum of Art and the Museum of Modern Art—but due to the extensiveness of the qualitative selections and the realization that the last two featured more contemporary and modern approaches (to both their collections and their websites), the four finalists seemed like a suitable group to evaluate together.
While these collections are not being evaluated by their ability to replicate a literal virtual tour through the museums’ halls, it was important to examine how well they could provide a digital tour of their collection, relating to information access and easy navigation through the virtual halls. Online visitors are likely researching information about artists, specific works, styles, origin, media, or time periods. How well each museum could aid them in these quests is critical, as they house so much of this information in both their staff members found at the museums and in their extensive databases.