Manchester Quality MetricsPilot 2013

Evaluation Session Report

Facilitators Inés Soria-Donlan & Sarah Gbeleyi

The Project

Developed by Australian company Pracsys and John Knell of Intelligence Agency, Culture Counts is a prototype measurement tool designed to capture feedback on the quality of arts and cultural events.

Using a tablet computer to record responses, it aims to assess the quality of arts and cultural events across the UK in a more measurable, accessible and engaging approach that allows for instant access to data and a shared database of statistics for artists, funders and members of the public.

In December 2013 the Manchester Quality Metrics Pilot was launched to pilot the use of the app at 8 different cultural events across the Greater Manchester region, spanning a range of art forms.The pilot was funded by Arts Council England.

As part of this Pilot period, Contact’s Creative Experts were commissioned to hold an evaluation session with some of the interviewers taking part in the pilot to explore their responses to the app. The session was designed to evaluate:

  • The User Experience journey for both interviewers and interviewees
  • Advantages and/or disadvantages of the digital app over traditional paper surveys
  • Clarity and relevance of survey questions posed to interviewees

The session took place after a matinee performance of Advent Avenue, where a Manchester Metrics pilot session had taken place.Two facilitators conducted this evaluation session, and also partook in the pilot as interviewers.

The Session

Date: 14th December 2013, 4.00 – 4.45pm

Location: Space 2, Contact

Lead Facilitator/Report Author: Inés Soria-Donlan

Assistant Facilitator/Researcher: Sarah Gbeleyi

Number of participants: 3

The Report

This report summarises the qualitative feedback that was collected during this evaluation session. Due to the low number of participants, quantitative feedback activities were unsuitable and a focus was placed on discussion and reflection. The session was divided into 3 key activities: Key Words, Exploring Opinions, and Unpacking the Process. This report details the raw data collected during these exercises and considers what this data tells us about responses to this prototype, and what recommendations can be made.

Key Words

Participants were asked to write down 5 words that summarised their experience and opinion of the app as a tool for capturing audience feedback:

Although intended as an exercise to capture both positive and negative adjectives describing the app (whereby participants distinguished between positive and negative words using colour coding), all words offered by the participants were positive. One participant did however note that ‘Questions were not always relevant’, a factor that is expanded on further below. Overall, however, a strong positive trend for the app is visible here, with a particular focus on positive user experience (‘simple’ ‘fast’ ‘fun’) demonstrated.

Exploring Opinions: Where Do I Stand

  1. The app is preferable to pen and paper.

Participants demonstrated a strong preference towards the app than pen and paper surveys, commenting that it was a more ‘engaging’ format for audiences, although some audience members remained a little daunted by the technology. According to participants, this high level of engagement was in part because, as a relatively new device, it draws curiosity from audiences, but also because it required (at least as this pilot stage) a level of interaction between interviewer and audience member that kept audiences engaged. They also felt that the app gave them, as interviewers, a higher sense of ‘legitimacy’ than pen and paper surveys.

Some participants voiced concerns regarding the extent to which such human interaction might influence the answers of audience members, whereas a pen and paper survey can be taken away and completed in private. Audience members also appeared less keen to ‘reflect’ on their answers before moving the slider, perhaps because of the presence of a waiting interviewer, which could impact on the accuracy of their answers. Similarly, some participants felt that a pen and paper survey that required definitive answers (eg. tick boxes) may gain more reliable results.

  1. I had to keep explaining what the questions meant when I was interviewing.

Participants’ positions on the axis demonstrated that the statements were clearly structured and easy to understand. They acknowledged that a few questions needed to be elaborated on, including the ‘Local Impact’ question (see Unpacking the Process below), but that on the whole questions were easily comprehensible.

As a result, the need to read the statements aloud to audience members was questioned, both because the statements did not require it and because they felt this was a point at which they were at risk of influencing the audience member’s decision. They felt, however, that they needed to retain this ‘role’ whilst holding the app to avoid an ‘awkward atmosphere’ whilst they waited for the audience member to complete the survey.

  1. I felt confident about the technical stability of the app.

Whilst participants’ overall response to the app was positive, some concerns were expressed regarding its technical stability. The majority of comments related to the impact of slow internet connections on the app. Participants expressed that this was inconvenient as survey takers but also had the potential of impacting on audience member behaviour, such as tapping the screen repeatedly while it loads, which sometimes resulting in questions being missed in error.

Participants also raised issues regarding the end process of each survey, voicing that the process to finish and save the survey was confusing. This meant that they were reluctant to let audience members complete the survey independently to ensure it was saved correctly, which therefore impacted on the audience members’ user experience, both in terms of feeling confident with the software and offering more ‘honest’ answers (which participants felt might be more forthcoming if audience members were able to complete the survey on their own).

All participants acknowledged, however, that as a prototype the app was considerably stable and that they expected these issues could be rectified.

  1. I can think of a better way than this to collect quality data on audience experience.

Whilst participants positioned themselves mid-way on the scale, they did not offer any alternative methods for collecting quality data on audience experience, expressing overall satisfaction with the app as a measure of data collection.

Despite their earlier comments regarding their potential influence on audience member responses, they felt that the human interaction was an important positive element of the process, as it helped to engage audiences, but that a pen and paper addition could be a useful accompaniment to the app should audience members want to leave any additional comments.


Unpacking the process: Interface and Questions

The Activity

Participants were asked to annotate printed screenshots of each screen of the app, focusing in particular on user experience regarding the interface and questions posed. They were asked to record their own views in one colour, and comments from audience members in another. Only those screenshots that received comments have been included in this report.

KEY: Participant ViewsAudience Responses

User Experience and the App Interface


Generally, as was discovered with the Where Do I Stand exercise, participants felt that the overall design and concept of the app was strong and engaging for audience members, while some issues of technical instability remained (i.e. Speed of app’s response to touch)impacting on user experience. Some participants experienced apprehensive responses from older audience members when asking for ‘optional info about you’, although it could be argued that this is relevant to any form of survey taking where basic demographic information is requested.

With regards to the visual design of the question pages, a leading priority for participants was to ensure interviewees gave accurate and reflected answers. A design feature that allowed audience members to have a limited choice of answers (rather than a gradable scale) was suggested, as some participants felt that audience members automatically swiped to the end of the scale with each question. Space for audience members to write any additional comments was also put forward. This begs the question of how this additional qualitative data would be ‘measured’ or compared across organisations.

Question Structure and Content

“Might not be a fair question in some occasions (e.g. parents struggled to answer this at a children’s show).”

“Felt this was harder to understand for young audiences.”

“Can responses that link to personal relationships to the piece (e.g. seeing your son act) really be used to measure the quality of the work?”

“Audience will generally check ‘strongly agree’ for this; unsure whether this this is very useful.”

“One man said he felt this was more of a yes/no question, described the bar as ‘inappropriate’.”

“Some people [who] weren’t local said this wasn’t relevant to them, they didn’t have an opinion.”

While questions concerning the production, content and structure of the piece received no comment, suggesting that these were well formed and easier to understand, those questions that drew on audience members personal and/or emotive responses to the event received more response from participants, suggesting that these were more difficult to define.

As is demonstrated above, drawing on their experience of using the app at a range of different events and with a range of audiences, participants felt that some questions, particularly 4: RELEVANCE and 9: LOCAL IMPACT were not always relevant.They suggested a possible system of having a bank of optional questions within the survey that accompanied a set of ‘core statements’ that organisations opt in or out of according to their relevance to the event.

Whilst the Results page,where audience members are able to compare their results with those already collected, was of interest to the participants in their capacity as interviewers, it was regarded as something of little interest to audience members.

The concern by some participants that the results page was perceived by some audience members as a ‘comparison’ between right and wrong answers should be noted, and perhaps reinforces the potential pressures for audience members that accompany the ‘public’ element of the app interview, as opposed to a pen and paper survey that can be completed and handed in anonymously.

Recommendations

  • Develop a series of ‘core’ and ‘optional’ statements, allowing organisations to tailor surveys to their events whilst ensuring a minimum level of comparable data across sites.
  • Create a short comments box with restricted wordcount at the end of the survey, allowing a structured space for qualitative data collection. A drop down menu above the comments box could allow these to be more easily catalogued by organisations (e.g.Is your comment about ‘Production’, ‘Topic Area’, ‘Staff’’, ‘The Building’).
  • Remove the results page from the end of individual surveys and instead have it as a link from the event home page, allowing interviewers to see a developing picture of responses.
  • Replace the slider with a drop-down menu or ‘tick-box’ format to encourage more reflective answers from audience members.
  • Simplify the end process of the survey, so the ‘Finish’ button automatically saves the survey and returns to a new page.
  • Making customers aware of the surveys at point of sale will raise awareness of the app and survey.
  • Conduct a pilot to measure interviewer influence on audience answers, by comparing answers from audience members asked to complete the survey with an interviewer, and others who complete it independently (e.g. at a fixed ‘survey station’) could help to determine the level of influence that the presence of the interviewer might have.
  • Conduct a full evaluation processwith all participants after the completion of the pilot, encouraging participants to compare their experiences across venues.

Summary

Whilst it is recognised that this small sample of participant data cannot offer a comprehensive evaluation of a new product, overall participants had a strong positive response to the app as a digital alternative to traditional surveys, reflecting a high level of positive user experience and a largely appropriately structured survey.

Despite being a technology product, it was clear that the role of the interviewer in the audience user experience remains central, at least at this pilot stage. This creates a slight dichotomy between participants’ view of the interviewer as an important part of the survey process (acting as a promoter of the survey, a guide to the app’s function, and a form of security for the tablets) and as an influence on audience behaviour due to their presence and interaction with the audience member during the survey. On the whole, there was generally positive feedback from audiences when they were told that thetechnology intends to be accessible by smartphone, which may alleviate such issues of interviewer influence, although the role of host staff as promoters of the app will be central to this success. This future developmentof the app is, therefore, highly recommended.