User Testing Highlights: Round 1 (Notes)
User Testing Highlights: Round 1 (Notes)
During a span of 3 weeks, 11 usability tests were completed with a total of 13 people. A driving simulator was used in order to immerse participants in the task of driving. Participants were first asked to become familiar with the driving simulator, then with each radio (both the Roadcasting prototype and XM Satellite radios were used). Then, participants used each radio while driving to find a station they liked. The order of the radios was counterbalanced to reduce confounding variables. The notes taken during each session were summarized and the most pertinent incidences and quotes are included below.
Figure 1. Picture of the GM-Network’s driving simulator.
ROADCASTING Results
Text is hard to read (font small)
“Fonts seem funny, a little squished. Hard to read” (U3, U7)
Album names would be nice to know
“Would be nice if display had what album the song was on” (U7, U10)
Important to have peripheral info
“It’s important to be able to pick everything out in your peripheral.” (U10)
Want to see time info for songs
“There’s no display for time of song. Would be good to have a timer. A countdown timer, not a count-up timer.” (U3)
Customizing would be nice
“It would be nice if there were a couple different choices of beeps so able to pick which one want to hear - would like to customize or turn it off.” (U7)
Participant wants to be able to add/drop categories. (U3)
Counter is helpful
“You vote near the end of the song.” Participant was able to figure this out by looking at the time remaining indicator and from past experience. (U12a)
Station names are evident (only one test to support)
“This is ‘Rick's music,’ this is his little broadcast station.” (U12b)
Physical buttons are desired
Participant wants to be able to feel the buttons. "Don't want a touch screen." (U4)
Categories/Stations:Stations/categories have unpredictable order, wrapping around
Stations/categories navigation good
“Ok, got it now. Category is up down; next back is different artist in category” (U3, U8, U9, U10, U11)
Categories are broad
Participant thought that categories were very broad - would like break down for more fine-grained surfing. (U6)
“Wrong” categories
“I wouldn't have expected this song on this station (hip hop instead of electronic).” (U6, U7)
Stations disappearing are confusing
“It's not going back to that song. I don't know where it is” Participant attempted to go back to a station that she flipped past too fast, using the Next button. (U12a)
Large category text is good
“How many categories are there? 6 or 7.” Participant then named off virtually all of the categories from memory. (U12a)
Voting:Most didn’t remember what they voted for
Participant couldn’t remember if the song that came up was in fact the song he voted for. (U4, U7, U10)
Voting not noticed right away
Took a long time for the participant to realize they could vote (beep came first). (U7, U9, U12 started with voice though)
Voting too often is annoying
“Every 5 minutes if you’re asked to vote again, it’s going to be miserable.” (U9, U5, U10)
Ability to turn voting off (1/3 of people)
“I would definitely like to be able to turn off voting, especially if doesn't want any of the choices.” (U3, U7)
Mixed feelings about when vote will play
“I don’t really get how the voting works. I saw that I would vote, but is the song played immediately or would I have to wait?” (U11, U9, U3, U4)
“I’d expect one of the three songs to come up next.” (U5, U10, U11)
When participant held the voting button down: “I’m moving forward in the same song?” (U11)
“I was expecting voted songs to come up in the queue after existing songs; I did not expect instantaneous selection of winning song.” (U3)
“We voted for Janis Joplin but then it didn't come up.” This was caused by the participant either changing the station or the prototype freezing. (12b)
“Maybe if we don't vote, it would go to another artist.” (U12b)
“If I pressed 2 here, what difference does it make?” (U8)
Voting seen as rating
“1 is that I want to hear it, 2 is that I’m indifferent, 3 I don’t want to hear it” (U11)
Participant thinks that rating things well means they'll be more likely to be played. (U4)
What prompts voting?
“I’m still not sure what criteria leads to voting prompt being given.” (U3, U11, U12a)
Song name not clear when voting
Hard to differentiate songs when voting because only shows artists. (U2, U4, U6)
How others voted could be interesting
“It might help to know if other people voted also, knowing the totals; but it's a novelty, it's interesting.” (U7)
“What are these bars going up? Is it like a histogram?” (U12a)
Participant thinks that voting involves other people, and he doesn’t always win. (U3, U5)
Voting time not adequate when listening to clips
Participant didn’t have enough remaining time to actually vote after shed listened to all three audio clips. (U2)
Mixed feelings about the idea of voting
“I sincerely doubt people would vote, people have lots of other media like cds and mp3s. Why would people depend on radio to play what they want to hear?” (U9)
“The voting feature is pretty interesting. It’s sort of fun. Usually when you listen to the radio you just take what's coming up next (…) It gives people a feeling of control. (U10)
“If I were interacting with this while I was driving, I would kill about 20 people.” (U2, U12b)
I don’t think I’d vote. When I’m driving, I don’t want to think about this. (U2)
“I don't know why we voted if it didn't come up.” (U12b)
“They should have a station that explains how to use your radio - like digital cable, they have a station that explains how to use it.” (U12b)
“It's a radio, you shouldn't feel like an idiot.” (U12b)
Voting only on highway/long trip/at stop light
“Wouldn’t use it while driving. Would use at a stop light, but not on highway or have the passenger do it.” (U2, U7, U8)
“In real life, I would only fiddle with the radio on a straight way.” (U6)
“I was looking at this the whole time - I don't know how you would have done it if you were alone in the car.” (U12b)
“I probably wouldn't vote on the way to the grocery store, but might do it on long trips.” U3)
Beep Vs. Voice:Mixed feelings about beep/voice
Participant initially thought that the beep was for an emergency broadcast or to indicate that they’d gone over the speed limit. (U7, U8)
“If I heard the voice all the time, would get really annoying…. Like at an amusement park where they tell you to stand to the left over and over again.” (U7, U3)
Beep is too loud. "Distracts from the actual song." (U2, U3)
“(Beep) is useful. it definitely gets my attention, but it isn't annoying.” (U10, U9)
Voice is "more noticeable" than the beep. (U5)
“I definitely like the voice better” he tunes out beeps, assumes it is a cell phone message ring. (U2, U6)
Audio Clips:Very few people discovered holding down for clips
Participant guessed that audio clips could be heard by pressing down the 1,2,3 buttons. (U7)
Confusion about listening and placing a vote
If you hold it down to listen to the clip, does it also place it as a vote? (U2)
Liked clips if not familiar with song or if lacked other info (ie song title)
He likes hearing clips, in particular if isn't song he's familiar with. (U3, U5)
He would use audio clips, only because not enough info to vote otherwise. (U4)
Participant doesn't feel like holding their finger down. Wants an auditory cue. (U6)
Song clip too short
It’s hard to tell sometimes in 5 seconds what a song is going to be like, especially if you get a weird snippet of it. (U2)
Miscellaneous:Song/Artist/Category control and feedback
“Is there a way to block an artist?” (U6)
Participant would want rating buttons, to rate "whether or not i would want to ever hear a song again, or a similar song, if I could categorize." (U4)
Robust buttons are needed
“c'mon go fast, do something” Participant repeatedly hit the Next and Category Up button. (U12b)
Out music selection is not sufficient
“We haven't heard one song we wanted to hear.” (U12b)
XM Satellite Radio results
Buttons are unlabeled, too small and similar in size
“It’s hard to see the interface, unlabeled buttons” (U7)
“Would be terrible in the night time -- can't see buttons. Not enough light. (U4, U9)
“I have no idea what any of these buttons do; I can't even read them.” (U10)
“Buttons should be bigger, they all feel the same, so it’s hard to know where you are… there’s no dot on the 5. (U7)
Button labels are ambiguous
“Category ‘a’ and ‘b,’ don’t know what is meant by this.” (U8, U9)
Song title and artist name are necessary
Participant thinks that seeing the song title is best... “There'd be absolutely no conclusion if I saw (artist info)” (U10)
“See, that's why I need artist name, because I thought that was daft punk.” (Misidentified by title) (U10)
Station numbers are helpful
Participant was able to recall a station due to its number. “It's nice to have a number. I sort of have a number-based mind.” (U10)
Station names are obscure
Participant has little trouble finding genres of stations, but doesn't know particular stations well enough to choose them. (U3, U6)
Scrolling display distracts from driving
“I hate scrolling displays; they distract your eyes from the road.” (U10)
Volume dial is awkward
“The volume sucks. I don't like the rotary knob deal. It's too small (U12a)
Navigation:Categories strangely ordered
“Categories aren’t in any order. I would have expected them to be in alphabetical order.” (U6, U7)
Participant might want to blend categories better, so you went from, like, urban to soul to pop. (U10)
Unintentional category switching
“I think I switched categories somehow, but I didn’t mean to” (U7)
“Only after you go to a specific channel do you know what genre it is in (U6)
“I can change the channels, but I don’t know how to go back to something I liked” (U8)
“I don’t know if there’s only one station of 80’s, I’d think there would be a couple more. Or a sub-category within a category?” (U6, U8)
Mixed reaction to dial
“Scanning within the category… I’m struggling; I don’t know how to do it.” (U8)
“I would like to browse all these different stations, but I don’t know how to move up and down.” Participant thought that the knob was for volume. (U9)
“Knob is stations within categories.” (U12a)
Confusion about dial button
“It goes there automatically, I don’t have to press anything. It’s harder b/c I want to leave the list up and not select something while I have to drive.” (U11)
“Why is it that sometimes I have to press the button and sometimes I don’t?” (U11)
“If I use the category button, I have to press the middle button. But if I’m scrolling within in a category, I don’t need to press the button” (U11)
Miscellaneous:Customization desired
Participant wants to set up own category, with all the stations he likes. He wants to eliminate what he'll never listen to. (U4)
Attention Required:Mixed results on required visual attention
Participant likes not having to look at interface in order to use it. (U4)
“So, when there’s a curve, I’m not going to change songs.” (U11)
Participant changed to a different song in the same category and then crashed into the side. (U11)
16 June 2004 GM-Network 7