Rick Chen
Blake Thomson
Pallavi Damera
Justin Pai
Kevin Merritt
GreenBean: Aftermath of Pilot Study
I Introduction
What is GreenBean? GreenBean is an Android application that acts as a single point of resource for shoppers to find information on products that they purchase. It guides shoppers into making smarter and greener purchases. GreenBean has a few key features that we tried to focus our high-fidelity prototype on. We wanted to show GreenBeans capabilities to display information about a products “greenBean rating”, to search and compare any given product and to track the history or a shopper’s purchases.
In our user testing we wanted to find major flaws in our UI that we could fix in our final prototype, we didn’t want to focus on the nitty gritty making sure every feature worked because it is out of the scope of our timeframe and it will not need user tests to implement a database or ensuring scanning accuracy. Our main purpose for running these tests was to hone our UI and make it as intuitive as possible, while ensuring information architecture is sustained and that our 3 main goals are still the focus of our interface.
II Methods
We decided to choose a demographic of young male students because of three main reasons. One, we wanted to target people who shopped independently and who have relatively recently just started shopping for themselves. We also wanted users with smart phone experience and were environmentally friendly. Finally, students were the most accessible demographic for the study. We ended each test with a compensation of candy bars!
III Apparatus
We ran our tests in a control condition in a office. We were originally thinking about running test in a grocery store, however due to time constraints and availability of testees we were not able to do so.
We had these following items:
Dummy Barcode
Video Camera
Digital Camera
Timer
Notes
Table + Chair
Office
IV Tasks
Scenario1:
You’re at the grocery store and would like to compare different types of granola. Search for “granola” and find the cheapest granola bar per unit.
Scenario2:
You're at the grocery store shopping and you need to get laundry detergent. You found Kirkland detergent but want to evaluate how environmentally friendly it is. Scan the product bar code to look up Kirkland detergent details and to help you decide if it is the type of detergent worth buying. (Pretend that the barcode for this notebook is the barcode for laundry detergent.)
Scenario3:
You want to find out your personal cumulative impact of your shopping choices to see how “green” you have been. greenBean knows your shopping purchases in different stores. Check your greenBean score and identify your shopping trip with most "green" items. On a specific date what is the least green item you purchased.
V Procedure
After introductions we had each participant sit down in front of a G1 so they immediately knew what type of study they were about to embark on. We decided that it would be best to give a general introduction of the purpose and features of GreenBean which would come in marketing material of the application, which the user would be introduced to before using it. We then followed by asking them to complete our three scenarios beginning one at a time. We asked them to think aloud and talk us through each step and what they were thinking. Many users would report on confusions and errors in this manner. Lastly, we ended with some debrief questions where we asked directed questions about the observations we made and asked them to explain. For example, we asked many users a question similar to “Is there a reason why you used the phone in landscape mode the entire time?” These types of questions helped us understand how people use this phone and how to change our UI to better fit users mental models.
VI Measurements
We measured the level to which they understood the tasks and why they were important to our study. We decided to measure this because it is imperative that the user understands the point of our application and why it would be helpful to them. Most users thought that our tasks were reasonable and helpful to them.
We also measured the time in which it took users to complete a certain task. This is important to measure because if a task takes too long to complete frustration and the overall usage with the app will decline. We discovered that the time range was similar in many users except for the scan feature which took a bit of toying around and tweaking to make it work correctly.
VII Improvements
On reflection we could have done many things better in the study alone. Our study was very invasive. The user had all of us looking at him/her and recording and taking pictures the entire time. I believe that this could cause a change in behavior of the user. In the future perhaps we could all leave the user to solve problems on their own with minimal face time and group pressure. We also were setup in an unnatural environment. We could have setup a grocery store type of environment in the office and we had the participant sit down which is a rare occurrence in a grocery store.
VIII Results
After performing the study we gained valuable feedback, primarily in form of both positive and negative comments given by the users during their progression through the tasks. These were gathered from five users who were environmentally friendly, did their own shopping, and most were familiar with smart phones.
Task #1- Search:
The first task went fairly smoothly for the participants, with all them finishing the task between a minute and a half to two minutes. Users had trouble with the application when interface problems appeared, such as the item list turning black when scrolling and the sorting tabs not actually sorting the items. Also, the interface in landscape mode left little space to display list items, which made it harder to browsefor some users who stayed in landscape mode after entering their search query. However, most of the users agreed that the end result of a list of comparable items was quite useful, although some users thought our search term ‘granola’ was more of a category to be browsed than an item to be searched.
Task #2- Scan:
The second task went less smoothly, and varied more in the time it took to complete it, from a minute and a half to four and a half minutes. The scanning feature took a while for the first user due to insufficient lighting, but after increase the amount of light in the room, the rest of the users had no problems with it, and some were impressed by how fast it was. Some users found the ‘search for similar’ button to be too small and hard to press and one user was used to stylus interaction and small button afforded interaction with his nail, which doesn’t work with the G1. Some users also thought the product details should have more information on what each ingredient/symbol means. Most users found the scan function to be quite useful in quickly scanning and comparing products, but one of the users thought that the scanner would be clumsy to pull out and that if the user were actively being green, they could recognize a product’s greenness without needing to scan it.
Task #3- Score:
The final task went quickly for all the users, with the time ranging between half a minute to a minute and a half. Finding the history screen and specific shopping trips and items wasn’t problematic, but most users didn’t care too much about their greenBean score. Although they did appreciate being able to see past shopping trips. Some users wanted to see more information about their trips, such as how many of each item were purchased and have partial green bean scores, and wanted to be able to categorize the history by stores instead of trips. There was also some confusion as to what ‘average recent score’ meant in terms of time frame as well as how the score was actually calculate.
IX Discussion
The participants were able to provide good feedback regarding the interface of our application, which is our primary focus right now and how to make it as intuitive as possible. Since our application UI is pretty cleanly divided by the tasks used in the study, we can approach the changes that need to be made in the same sections.
Search
Some changes were made to the search box prior to the study to place it more front and center on the screen. During the study, most of the users had little trouble with the task overall, although their performance was slowed by bugs in the prototype. This included the list turning black when scrolling and the sorting tabs not actually sorting the list. Also, some users left the phone in landscape mode after entering their search query. Unfortunately, the UI was designed in portrait mode, so when the UI was stretched for landscape mode, the tall sort tabs left little room to view the list in.
Based on the feedback we received, the primary changes we will be making to the UI will be to complete implementation of features and fix the interface bugs users ran into. The item list will also be redesigned to work well in both portrait and landscape mode, possibly by moving the tabs to the side instead of the top in landscape mode.
Scan
Some changes were made to the scanning screen prior to the study were that the orientation was changed to be horizontal and most of the instructional text was removed. During the study, most of the users were able to perform the task fairly quickly, although some problems arose with the scanner not scanning when there was insufficient light, but since this isn’t a problem with our implementation, there isn’t much we can do to fix it, although we can add a note about it. The product details page and comparison of products was well received, but users wanted the details to be more descriptive, such as having the ingredients ordered by greenness or linking to descriptions of what they are and having a key for the recyclable symbols. Some users also found the compare button to be a bit small and hard to press. The primary changes we will be making to the UI will be to label how the ingredients are sorted and possibly adding a key in the help screen.
Score
Quite a few changes were made to the history screen, it now lists the previous trips and the items bought on each trip and has different sorting options, the graph and categories were removed. The history section and the recent score part of the home page probably require the most changes based on the feedback we received. While the task went smoothly for all the users, most of them didn’t really care about what their green score was. Just having a greenBean count didn’t provide any motivation to care about their score, especially when the user wasn’t actively trying to be green and they didn’t know how the score was being calculated. Users did suggest that having groups or people to compare their score to would make their score more important to them. The primary change will probably be to change the score visualization on the main page from an average count to something more interactive. This might involve having a plant that grows as you make green shopping purchases and withers if you weren’t, similar to the UbiGreen interface. Other alternatives discussed were having a sort of virtual pet you fed beans to and it perform different actions based on beans fed, etc. However, the goal would still be to find an interesting visualization that could make a user care about their score even if they weren’t particularly green conscious.
Further Study
Overall we got a lot of useful feedback from the user study, albeit only from a small number of participants. If we were to expand the study to a much larger one, then we would make some changes to the study to get more thorough results including those listed in the Improvements section above. In our study, we tried to get users who tried to be green, were smart phone users, and shopped for themselves. Unfortunately we were only able to get users who fit into two of the three categories. Because our application is targeted towards users who fit all three criteria, the feedback we received was slightly skewed. The actively green users weren’t smart phone users and weren’t used to the interface and would be able to see the greenness of a product off hand without the application, whereas the smart phone users weren’t actively trying to be green and didn’t care so much about their green score and as a result treated the application as more of a shopping assistant with the aim of saving money. Hopefully in a larger study we will be able to find users who fall directly into our target demographic and will be able to provide more accurate feedback regarding functionality.
X Appendix
User study script
Observer brief:
explain roles
note taking
equipment handling
questions at the end
no hinting during the study
Read out to participant purpose and method of study:
Hi ___, I am _____.I will be mainly facilitating the study while X and Y will be observers. Our team is working on the design of proposed product: greenBean that helps users to make smarter and greener choices in their shopping.
This study is not to evaluate you, but to evaluate our prototype to find what works and what does not for users like you. So, while you perform the tasks please explore and use it the way you would feel comfortable to help us figure out problems with this design. It will have 3 tasks that you will have to perform on the mobile phone. While you perform the task we would encourage you to think aloud what your thinking and looking for in the product's interface. We will demonstrate to you how to think aloud before we start the tasks.
Feel free to stop or ask questions at any point of the study. As you're working through the exercises, I won't be able to provide help or answer questions. This is because we want to create the most realistic situation possible. Even though I won't be able to answer your questions, please ask them anyway. It's very important that I capture all your questions and comments to understand the issues with the design. The study is going to take 30 minutes approximately. With your permission we would like to take a video as your interact the product to help us note time and details of your interactions. But we assure no personally identifying information will be used for the study. As a token of our appreciation at the end of the study we would like to treat/reward you with a ______. Thank you very much for your time and interest to help us with this project. If you are ready we can begin the study now."
Pre Study Questionnaire
Could you fill in the Demographics questionnaire please?
Age?
Gender?
Education level?
Major?
Question related to experience with similar tasks and gadgets:
What kind of phone do you own?
Are you familiar with Android?
On a scale of 1 to 5 how environmentally friendly are you? 5(most friendly)
Demo android and think-aloud-protocol
Why and How to think aloud:
- We have found that we get a great deal of information from these informal tests if we ask people to think aloud as they work through the exercises.
- It may be a bit awkward at first, but it's really very easy once you get used to it.
- All you do is speak your thoughts as you work.
- If you forget to think aloud, I'll remind you to keep talking.
- Would you like me to demonstrate?
Use Android to check Weather information:
Sequence:
<include sequence of the above tasks>
Demo Android:
Now I will demo few features of Android, that you may need for the tasks.
Use of context menu
Sequence:
<include sequence of the above tasks>
Use of scan feature
Sequence:
<include sequence of the above tasks>
<Demonstrate both outside the context of greenBean>
NOTE:
All of these steps are to be done by the researcher.
Tell the participant that they can ask questions any time
And to Think aloud while doing all tasks.
Intro to tasks:
Do you have any questions before we start the study?
greenBean is a proposed mobile application that helps shoppers make smarter and greener choices while shopping.
It helps people track their shopping behavior and make them aware how different purchases contribute to their overall green bean score. Key functions of greenBean are: