Usability Testing Plan
Usability testing on the Underground Coal Mine Supervisor Training will be conducted during the week if May 1st. The purpose of the usability testing is to gather the users’ reaction on the overall visual features and feels of this training program; to gather data of the observations on the functionality of this program; and to have the experts validate the content of the training.
Methods
The focus of the usability testing is on the three aspects of this training program:
- Functionality
- Content accuracy
- The reactions on overall appeal
The methods to accomplish the purpose of the usability testing include one-to-one evaluation, expert review and focus group, and the instruments. Specifically:
Evaluation on the functionality:
- Evaluation focus: the focus of the evaluating on the functionality of the web-based training program is to see if the system do what it is supposed to do, specifically,
It serve well the purpose of the web site
Interactions between the site and the users: Is it easy for users to navigate through the prototype?
Do the media elements such as videos serve well the purpose of the content presentation
Is the structure of the web site easy to follow?
- Methods: The method for evaluating the functionality is one-to-one observation. The evaluators are from TTAC team members. MSHA team will observe how each evaluator react and interact with this training program. A behavioral observation check list is used to record the information, behaviors and reactions from the evaluators.
- Instrument: The instruments include an observation check list and a Think-aloud protocol for recording the behaviors of the evaluators.
Evaluation on the training content:
- Evaluation focus: The focus of the evaluating on the content of the training program is to get feed back from experts on the content accuracy and validity. The content for this evaluation is the training sections: Check Roof and Draw Rock in pre-shift examination (Duty 9 in JTA), and the training content for the problem solving skills under emergency situation in Duty 11.
- Methods: The methods for evaluating the content include expert review and expert focus group. By expert review, the experts will review the training content of the prototype, and fill in their comments in the evaluation questionnaire focusing the specific content. After the expert review, a question list with the disagreements a focus group of the experts will discuss the disagreement on the training content.
- Instruments: A questionnaire is developed for the expert review. A question list will be developed based on the disagreements on the content from the expert review.
User Reactions on Overall Appeal:
- Evaluation focus:The focus is to gather the user reactions on the overall appeal
of the training system. The target users are coal mine supervisors.
- Method: The method of gathering the users’ reactions is to allow users to go through the training system and fill in a web-based evaluation form. An instruction will be developed to direct the users to complete the evaluation form.
- Instrument: A web-based evaluation form with quantitative measurements is to gather the data from users to quantify the reactions on the overall appeal of the training system.
The three evaluations will be conducted to evaluate the three different aspects of the system: functionality, training content and overall appeal. The data and feedback from the evaluations will be analyzed and categorized to help the team to improve the training system.