ARtSENSE


ARtSENSE (Augmented RealiTy Supported adaptive and personalized Experience in a museum based oN processing real-time Sensor Events) was a three-year European co-funded project implicating seven technological and three museum and cultural heritage partners. The goal of ARtSENSE was to examine the potential of Augmented Reality for the museum and gallery experience. The project introduced new wearable technologies for sensing continuously and non-intrusively the user’s context (visual context, eye-tracking, audio tracking, 3D spatialisation, physiological sensing) in order to determine the user’s current interests.ARtSENSEtackled a very important problem in the modern usage of ICT in cultural heritage domain: bridging the gap between the digital world with the physical in a highly flexible way in order to enable a novel, adaptive cultural experience. The ARtSENSE projectaimed to develop an active assistant which looks over the user’s shoulder (physical world) and react on any change in a visitor’s state of interests (user’s world) by adapting the guide (digital world) accordingly.

Besides introducing novel wearable technologies for sensing continuously and non-intrusively the user’s context (visual eye-tracking, audio, physiological biosensing) in order to determine the user’s current interest (mental engagement),ARtSENSEexamined the way on which adaptive assistance can be realized: using cutting-edge technology (low weight bidirectional see-through displays) that enables overlaying reality with digital information transparently, including gaze-and gesture-controlled interaction, so that visitors have the feeling that physical objects are directly responding to them. In that way artworks become active artefacts that react on users’ attention and emotions and provide more information about them! This leads to the new generation of mobile museum guides based on the novel concept we call Adaptive Augmented Reality (A2R).

The visitor can use natural gestures to interact with the multimedia content delivered to the view through the glasses in the form of virtual overlays (Figure 2). Audio augmentations are also provided, as well as 3D sound effects, while the acoustic and affective attention of the visitor is continuously monitored through the use of audio and acoustic sensors. Thus the visitors have the feeling that physical objects are directly responding to them; the artworks become active artefacts that react on users’ attention and engagement levels.

FACT’s participation in theARtSENSE project provided an extra opportunity for the investigation of core questions related to public engagement, gamification and shared experience using Augmented Reality not only within but also beyond the museum and gallery walls. The project features a collaboration among a large team of museum professionals from three totally different yet complementary Cultural Heritage institutions from Spain (MuseoNacional de ArtesDecorativas – MNAD), Paris (Musée des arts et metiers – MAM) and the UK (FACT).

During the first life-circle of the project, the main focus was on encouraging synergy among the cultural heritage professionals and technological partners so as to identify requirements and needs that would be then transformed to meaningful augmented visiting experiences. FACT, however, presented an even more challenging case study since no permanent collection exists, as is the case with the permanent, well-established and internationally significant collections of MNAD and MAM. The solution to this challenge was two-fold:

  1. Think of the building itself as an artifact – with its own history, context and relationship to its public;
  2. Create a new work of art that interrogates both the problem and the technology.

Both proposals were put forward to the consortium. FACT suggested a number of different artworks that had been part of the building’s construction as artifacts to “augment.” The starting point should be an artifact with a deep and rich set of additional materials that would be extensive enough to give multiple variables for different levels of “personalised” engagement, and be granular enough to satisfy the expert and the casual visitor alike in an adaptive way.

The VIP signature pillar

The VIP signature Pillar reflects both the context and the history and the relational meaning of the building to its public (Figure 3 and 4). The Pillar is anad hocdocument of “Very Important Person” (VIP) visitors to the FACT building, with signatures from film directors, artists, musicians and actors that have visited and worked with FACT. The pillar provides an indication of the breadth and scope of practice that goes on in the building. It is a way to document the passage of these guests, and a fun and “punk” approach for theseauteursto leave their mark on the building. Moreover, it is generally the case that each of the signatures has been “captured” in some way on camera and in interview. This enables FACT to underpin each signature with rich media content and make the user engagement with the proposed ARtSENSE technology a deep and valuable one. The visitor equipped with the system would be able to get to this rich media content in a novel way and there was enough content to be able to adapt to the users’ level of engagement and supply them with appropriate material.

Audio accompaniments to almost all of the signatures on the pillar are available to listen to here. You can also watch footage of Ken Loach, Apichatpong Weerasethakul and Krzysztof Wodiczko at FACT.

Commissioning an Artwork

Though FACT is very experienced in commissioning artworks that incorporate technologically challenging elements and presenting them to the public, ARtSENSEwas a little different: here was a research project where the technology would be constantly under evolution. FACT had to find artists who were not only proficient with emerging technology in line with the core aims of the project – developing a biologically integrated adaptive AR system – but who would be willing to be part of the experiment and the project itself. The vision was that the successful artists would bring a new approach and by default open up new and exciting avenues that had previously not been considered.

It was important that the artists understood the concepts behind the work before we undertook to commission them. To this end, PhD researcherAnetaKrzemien working on her doctoral thesis “The Role of New Media in Art, Culture and Society at the Turn of the 21st Century”, helped us to identify a number of artists whose work was a good fit with the curatorial and technical framework of the research. The consortium decided that the work of artists collective Manifest.AR was the most appropriate to take forward.


Manifest.AR

Manifest.AR are an artists’ collective based across the world who are working with the medium of Augmented Reality. Augmented Reality, or AR for short, is a new technology that allows you to use a smart phone or tablet to alter or enhance your view of the world. When users examine their environment through their mobile devices, they see information (text, images, 3D objects) superimposed on the space around them, integrated into physical locations as though they existed in the real world. Manifest.AR use this technology to create interventions, applications and games to encourage debate on how we interact with the world and new technology allowing us to become ‘critical citizens’. The group are relatively young as an organization, having endorsed their own AR Art Manifesto as recently as the 25January 2011. The group currentlycomprises eight core founding members and 20 affiliated members.

In their ‘AR Art Manifesto’ they state that the future ‘is without boundaries’ and that they wish to ‘transform Data’ into something physical and real. Their work inTurning FACT Inside Outcould be used to deliberately “delete” parts of the city, use the sky as a blank canvas for creativity or populating the entire city with wild growth vegetation. Manifest.AR’sInvisible ARtaffectsfeatures six new games and applications which could be played at FACT or using your own smart phone/tablet. Watch the short film about Invisible ARtaffects here.

They have shown their work in major international galleries - often uninvited! Manifest.AR have invaded galleries including MoMA and Tate Modern where they ‘installed’ their AR games and visualisations completely without the agreement of curators and gallery staff. Their works are usually created and framed by social contexts, and are always interactive and participatory, often in a “live” dynamic. In working with FACT they also took on board the FACT-specific interest and practices of working in and with the community of Liverpool.

“Manifest.AR’s art practice is always relative to site; siteas social space; site asarchitectural sign; site as political event; site ascommunity memory; site aspicturesque landscape; site as global issue. Augmented reality, a medium which itself suggests inversionand knows no physicalboundaries, renders the institution permeable, spatiallyand symbolically. Thecategories of body-mind, the institution-city, and thevirtual-physical are crisscrossedand reconfigured. FACT is revised as alaboratory medium for experiments in mind,matter and community.” – Press Release for Turning FACT Inside Out, Will Pappenheimer, 2012

The first iteration of the collaboration between FACT, Manifest.AR and the ARtSENSE consortium was presented in the “Turning FACT Inside Out” exhibition in June 2013. FACT’s existing and historic public realm work has demonstrated value in long form projects embedded in a variety of communities and interest groups and “alternative” spaces, but how can we create a more adaptable and open public realm, capable of responding to the evolving needs of our public, the building users and communities? In essence Manifest.AR was asked: “How do we really use the digital public sphere?”

Click here for a full overview of the work and projects conceived and developed by Manifest.AR as part of the “Turning Fact Inside Out Exhibition”.