Header Space reserved for Publication
A PROPOSAL OF AN ANALYSIS METHOD FOR INTERNET DEVICES SETUP TASKS
DaisukeIto1 Masahiko Munechika2Masaaki Kaneko3
1Waseda University, Japan, JSQC, >
2Waseda University, Japan, JSQC, >
3Waseda University, Japan, JSQC, >
ABSTRACT
The use of internet devices in the home has increased dramatically in recent years. However, the number of novices who have no experience of ITC devices has also increased. Owing to their inexperience, these users typically make mistakes when undertaking setup tasks. As a consequence, companies strive to improve their products so that novices can complete setup tasks without making mistakes.
Adopting the theme “the novice can setup without mistakes,” we investigated whether it is possible to observe a user setting up devices in a usability testing room.However, there are two associated problems that need to be addressed. Firstly, there are no forms to record user behaviour and, secondly,the method of observationmayaffect the behaviour of the testee.
In an attempt to solve these problems, we firstly devised arecording form to grasp place of failure without omissions. Secondly, we specified the point of failure on the recording form by observing the testee setting up internet devices. Finally, we used videos of eye movements and behaviour corresponding to a 3-stage information processing model in order to identify the reasons for failure without having an effect on the subjects.
At last, we improved certain products, which enabled 6 of 7 novices to succeed in setting up from 12 in 24 novices. Therefore, we propose an analysis method for internet device setup tasks that utilizes “reason for failure patterns,” which we identified during our investigation.
Keywords: Cognitive Model, Internet, Usability
BACKGROUNDAND PURPOSE
Internet devices for homeusage have begun to be widely usedin recenttimes. This has been instrumental in developing information technology and the governmental FTTH (Fiber-To-The-Home) policy. The number of novices without prior experience with ITC devices who now have begun to use them is on the increase. Such users generally make mistakeswhile setting up systems because of their inexperience. Therefore, companies provide customer support through call centres or engineerswho are sent to users’ homes to setup systems.
To break the status quo, companies have been trying to provide clearer instructions for product usage so that novices can correctly perform setup tasks. However,such efforts have not borne fruit because engineersdeveloping products do not know exactly where the problem lies;this lack of knowledge can be ascribed to the large differencesin the ways individualsset up systems.
This paper conducts an experiment and proposes a method of analysis in order to enable even novicesto correctly setup Internet devices.
APPROACH
We set the goal as “Novice set up without mistakes”. To achieve this goal, it is effective to observe how a user performs setup tasks in the usability testing room. Then, we identify the stage where the user fails (hereafter failure place) in performing the setup tasks, and the reason for failure (hereafter failure reason). All these findings are then looked at in conjunction and an improvement plan is proposed to help users successfully set up Internet devices.
In order to effect improvements, failure places need to be precisely extracted. To grasp the present state of how user operations are observed, we looked at the investigations of corporation A that sells Internet communication equipment. The engineers took the observation memo that is based on the experience and intuition of the engineer, and there were omissions to grasp failure places. For this reason, we realized that a record form with which we can correctly identify the failure place is needed.
When we understand the reason for failure, the protocol analysis,or themethod of hearing when testee made failure, can be put to good use. During interviews, the engineers remarked, “When we use the protocol analysis, the testee can’t perform the setup tasks as usual because this method givesa lot of mental load to the testee. When we use the hearing method, the result of conforming to actuality is not obtained because the testee is blocked the conduct of the subject.”
In order to determine the reason without seeing the effect on the subject, first, we create a record form to identify all the failure places. Next, we specify the failure places on the record form by observing the testee set up Internet devices. Lastly, we specify the failure reason by watching videos of the user’s eye movements and behaviour, and frame solutions for better instructions for product usage.
ANALYSIS OF SETUP
Extracting Element Tasks
In order to improve the instructions for product usage, it is necessary to observe the actual failure place and specify it clearly. To this end, we conducted observations as a preliminary examination. The users performed the tasks listed in the manual (hereafter unit tasks) in a random order, as part of the test. The unit tasks included several operations, and the users carried out the unit tasks by changing the order of the operations. Consequently, by merely observing and recording the size of a unit task, the change in order of the performed operations cannot be captured. To add to this problem is the fact that users who successfully perform the setup tasks do so in widely different ways. We divided each unit task into several necessary operations (hereafter element tasks). The extracted element tasks gathered from the observations are given in Table 1.
Table1 -Unit tasks and element tasks
Unit task / Explanation of unit tasks / Element task / Explanation of element taskStep 1 / Connect ONU and Internet router with the blue LAN cable / Step 1-A / Connect the blue LAN cable with ONU
Step 1-B / Connect the blue LAN cable with router
Step 2 / Connect PC and Internet router with the green LAN cable / Step 2-A / Connect the green LAN cable with PC
Step 2-B / Connect the green LAN cable with router
Step 3 / Connect phone and Internet router with the modular cable / Step 3-A / Pull off modular cable from wall
Step 3-B / Connect the modular cable with phone
Step 4 / Connect outlet and Internet router with the power cable / Step 4-A / Connect the power cable with outlet
Step 4-B / Connect the power cable with router
Step 5 / Check connection / Step 5 / Check connection
*ONU: Optical Network Unit
If the element tasks in Table 1 have not been performed, the user is said to have failed in the setup task. Such a breakdown of tasks makes it easy to identify the failure places. In addition, it also helps recognize the change in the order of execution of the element tasks, though not that of the unit tasks.
Making a Record Form
In order to successfully perform the setup task, the user should perform all the element tasks obeying the order. Given this, in order to identify the failure place, we should record the success or failure of each element task, listed in table 1; it is also necessary to record the order of execution of the element tasks.
In addition, according to the thought of KLM (Keystroke Level Model), we regard “When time is spent on a certain element task, there is every possibility where there is a problem in the place.” The record form can be compiled in the manner elucidated above; with this form, we can record the execution order and the execution time by setting the column name as testee number and the row name as element task.
The record form enables us to easily identify where the failure place is. In addition, it helps keep track of task achievement, average time taken for each task, and the standard deviation ratio of every element task. When most problems occur during the execution of a certain element task, we can grasp that element task.
Observing Testees’ Behavior
In order to specify the failure place, we observed the setup tasks of the users. We thought that presuming the failure reason was useful for recording the testee’s eye movements, because a user carries out tasks by reading product setup instructions. Then, we took a video of the user’s behaviour and eye movements when the testee was performing the set up. The summary of the investigation is given below.
We recorded testees’ behaviour in this investigation using the record style that was elucidated in section Making a Record Form. The result is shown in Table 2.
Table2 - Testees’ behaviour
The failure of an element task is shaded in the box.In addition, the restoration task simultaneously helps record the cases where the testees made mistakes.
From Table 2, the number of testees who succeeded in performing the setup task was found to be 9 from a total of 24, and the number of testees who carried out all the element tasks was 12. Further, considering the cases in which some testees failed to perform multiple element tasks, the number of failure places stood at 29. In the performance of every element task, there were differences in the task achievement ratio, mean time, and variation.
Analyzing Failure Places
In order to tie the extracted failure place to improved instructions for product usage, it is necessary to understand the failure reason. First, we observed what kind of actions the testees performed at the 29 failure places. We classified these actions into 4 patterns: “testee did not try to execute the element task,” “testee erroneously performed the element task,” “testee stopped before the completion of the element task,” and “testee made a mistake in the task order.”
Table 3 - Failure behaviour pattern
Failure behaviour pattern / Amount / Explanation / Amounttestee did not try to execute the element task / 17 / Did not perform step 1-A / 1
Did not perform step 2-A / 1
Did not perform step 5 / 12
Did not perform step 3 / 3
testee stopped before the completion of the element task / 5 / Could not perform step 1-A / 3
Could not perform step 3 / 1
Could not look for the LAN port in PC / 1
testee erroneously performed the element task / 4 / Connected the blue LAN cable with PC / 1
Pulled off the modular cable from the phone / 1
Connected PC and ONU / 2
testee made a mistake in the task order / 3 / Performed step 4 first / 3
amount / 29 / - / 29
In order to grasp the failure reason, we further analyzed testees’ behaviour. Novices set up Internet devices by reading the instructions and deciding how to perform the setup.
In order to guess the failure reasons, we have to identify in which stage mistakes occur by using the three-stage information processing model (Research Institute of Human Engineering for Quality of Lifeet al. 2005) that breaks information processing down into three stages: perception, judgment, and action.
Figure 1- Three-stage information processing model
In order to find a pattern in the failure reasons, at the perception stage, we analyzedeye movements; further, at the judgment and action stage, we analyzed the failure reason pattern in Table3. The result is shown in Table 4.
Table 4- Failure reason pattern
Failure / Cause of decision / Stage of mistake / Failure reason pattern / AmountEye movements / Behaviour
order / did not look / testee made a mistake in the task order / perception / did not perceive the task order / 2
Looked / testee made a mistake in the task order / judgment / misjudged the task order / 1
element task / did not look / testee did not try to perform the element task / perception / did not perceive the existence of the task / 7
Looked / testee erroneously performed the task / judgment / misunderstood task content / 4
Looked / testee stopped / did not comprehend task content / 5
Looked / testee did not try to perform the element task / judged the task as unnecessary
forgot about the task / 10
Looked / - / action / could not perform the task / 0
By Table 3, it is possible to grasp the failure reason. Then, it is possible to analyze the primary factor, in order to plan countermeasures, and study the improvement policy for better instructions for product usage.
Improvement and Verification
Using the result obtained in table 4, we examined the improvement object and improvement policy by tying them with the pattern of failure reasons. The examination result is shown in Table 5.
Table 5 - Improvement object and policy
Failure / Stage of mistake / Improvement object / Failure reason pattern / Improvement policyorder / perception / presentation of instructions / did not perceive task order / choose position and color to make it conspicuous
judgment / contents of instructions / misjudged task order / make it easier to understand
state the necessity clearly
element task / perception / presentation of instructions / did not perceive the existence of the task / choose position and color to make it conspicuous
judgment / contents of instructions / misunderstood task content / scrutinize in order to gain an understanding
did not comprehend task content / scrutinize in order to make it easy to understand
judged the task as unnecessary
forgot about the task / state the necessity clearly
action / operation instrument
contents of instructions / could not perform the task / make the instrument easy to use
add notes to the sequence of operations
From the improvement policy obtained in Table 5, we examined countermeasures like “In order to be conspicuous, change the color of the background of the product illustration in the manual.”
We then offered testees to set up internet devices using the improved product; six in seven (85.7%) testees successfully performed all the element tasks under the conditions stated in section ObservingTestees’ Behavior.
We think that the upgraded product was better than the product initially used, wherein 12 in 24 (50.0%) were able to successfully execute all the element tasks. However, it is necessary to conduct further verifications because the number of testees is few.
PROPOSAL FOR A METHODOF ANALYSIS
On the basis of the analysis mentioned earlier, we propose the following analytical method.
【step 1】Extraction of element taskGrasp the element tasks forming the setup task.
【step 2】Compilation of a record form
Create a record form using element tasks
【step 3】Observation of testees’ conduct
Observe setup task. In this step, videotapeeye movements and behaviour.
【step 4】Identification of failure place
Identify the failure place using the record form.
【step 5】Attaching failure place to failure reason pattern
Match with the failure reason pattern in Table 4 by analyzing video of failure places.
【step 6】Framing an improvement plan
Frame an improvement plan by using the improvement policy given in Table 5.
This procedure rearrange the analysis and improvement plan that we had mentioned in Chapter Analysis of Setup. This proposal can have wide-ranging applications because the record form, video observation, and failure reason pattern used in this research, can be applied to other similar products.
DISCUSSION
Implications of This Paper
The method used to set up Internet devices varies across users. Because of this, better instructions for product usage cannot be provided if there exists difficulty in identifying failure places.
In this paper, we found it easy to identify the failure place clearly by observing testees’ behaviour with the record style that uses element tasks in advance. In addition, we made it possible to presume the failure reason and frame an improvement plan by analyzing the failure place. This approach was found to be useful.
In this paper, we also assumed that it was possible to lessen the influence of the examination method on the testees by using the video showing testees’ eye movements and behaviour. Future research is necessary to compare the degree of influence in our method with that in the methods explained in chapter 2, such as protocol analysis.
Possible Uses of Failure Reason Pattern
In this paper, we create a failure reason pattern by understanding novices’ actions in the three-stage information processing model. By observing setup tasks, it is possible to frame an improvement policy. Furthermore, in the process of developing a product, we expect to preempt certain problems that users might face, by creating a checklist.
We believe that the failure reason pattern can be used not only for Internet devices but also for every product that users set up by following instructions and understanding their content.
CONCLUSION AND FUTURE TASKS
In this paper, in order to analyze user behavior, we created a record form by observing the performance of element tasks. We also framed an improvement policy by analyzing the failure places and creating a failure reason pattern. On the basis of these steps, we proposed a method of analyzing setup tasks of Internet devices.
As future tasks, we list the further verification of the improvement policy, comparison with other method such as protocol analysis, and improvement of the failure reason pattern.
ACKNOWLEGEMENTS
We are grateful to Mr. Miyamoto from Company A for his help. We also thank all the members of the Munechika Laboratory.
REFERENCES
Research Institute of Human Engineering for Quality of Life (2005), Work shop human life engineering part 3- usability of interactive system, maruzen corp.
ISO 9241-11:1998 (BS EN ISO 9241-11:1998) Ergonomic requirements for office work with visual display terminals (VDT)s - Part 11: Guidance on usability
1