An Evaluation 1
Running Head: AN EVALUATION
An Evaluation of Computer-based Training Options for Annual Ethics Training at Raytheon NetOps and Information Solutions
Jessica Foley
Virginia Polytechnic Institute and StateUniversity
Table of Contents
Abstract
Introduction
Research on Use of Evaluation for Computer-based Training
Overall Objectives of the Ethics Training Requirement
Raytheon NIS Employees
Raytheon NIS Technical Environment
Basic Evaluation of Products Within the Environments
Product Evaluation – Business Ethics: An Introduction
General Product Description
Product Title
Product Developer(s)/Producer(s)/Publisher
Intended Learning Environment
Media Characteristics
Material Organization
Content
Basic Content Description
Stated Objectives
Content Evaluation
Instructional Context Description and Evaluation
Type(s) of Learning Context(s)
Functions of the Learning Context(s)
Effective Context(s)
Motivational Strategies
Context Evaluation
Instructional Components and Conditions Descriptions and Evaluation
Instructional Design Components
Components Not Addressed
Conditions Addressed
Conditions Not Addressed
Type of Feedback
Level of Feedback
Instructional Message Display Description and Evaluation
General Message Display Description
General Message Display Evaluation
Text use and Display Description
Text Use and Display Evaluation
Pictures and Illustrations Use and Display Description
Pictures and Illustrations Use and Display Evaluation
Diagrams, Charts and Graphs Use and Display Description
Diagrams, Charts and Graphs Use and Display Evaluation
Audio Use Description
Audio Use Evaluation
Navigation Description
Navigation Evaluation
Final Thoughts on Product
Product Evaluation – Ethics reConnection 2008
General Product Description
Product Title
Product Developer(s)/Producer(s)/Publisher
Intended Learning Environment
Media Characteristics
Material Organization
Content
Basic Content Description
Stated Objectives
Content Evaluation
Instructional Context Description and Evaluation
Type(s) of Learning Context(s)
Functions of the Learning Context(s)
Effective Context(s)
Motivational Strategies
Context Evaluation
Instructional Components and Conditions Descriptions and Evaluation
Instructional Design Components
Components Not Addressed
Conditions Addressed
Conditions Not Addressed
Type of Feedback
Level of Feedback
Instructional Message Display Description and Evaluation
General Message Display Description
General Message Display Evaluation
Text use and Display Description
Text Use and Display Evaluation
Pictures and Illustrations Use and Display Description
Pictures and Illustrations Use and Display Evaluation
Diagrams, Charts and Graphs Use and Display Description
Diagrams, Charts and Graphs Use and Display Evaluation
Audio Use Description
Audio Use Evaluation
Navigation Description
Navigation Evaluation
Final Thoughts on Product
Summary Table with Scores
Conclusion
References
Abstract
This paper will discuss historical research in the area of computer-based training evaluation and detail an in-depth formative evaluation of the two computer-based training products:
- Business Ethics: An Introduction
- Ethics reConnecion 2008
The paper conclude with a recommendation as to which training module, or a combination of both, would be best suited for meeting the learning objectives in the Raytheon NetOps and Information Systems employees in their work environments.
Introduction
Raytheon Corporation purchased Houston Associates, Inc, (HAI), approximately 3 years ago and recently renamed it to Raytheon NetOps and Information Solutions (NIS). Since NIS remains a wholly owned subsidiary, they have the option of picking and choosing which benefits and resources are adopted from the parent company. However, all benefits come with a monetary cost and must be evaluated to determine whether or not the employees will benefit from the offering. In January of 2009, Raytheon will require all employees, in corporate, subsidiary, and customer locations, to complete ethics training by utilizing one or more of a variety of Raytheon-provided training products. Currently, the NIS Human Resources Department has narrowed the choices to two training products and is willing to purchase both modules if the combination proves more effective. Their choices were simply based on basic subject matter and price. The titles of the two modules are:
- Business Ethics: An Introduction
- Ethics reConnecion 2008
This paper will discuss historical research in the area of computer-based training evaluation and detail an in-depth formative evaluation of the two training products, concluding with a recommendation as to which training module, or a combination of both, would be best suited for meeting the learning objectives in the NIS environment by NIS employees.
Research on Use of Evaluation for Computer-based Training
More than twenty years ago, Hubbard (1987: 227–228) published a list of why computer-based training is often more difficult to fully evaluate than conventional teaching materials, such as text books: it is a comparatively new medium; the ‘hierarchical structure and branching’ found in many programs is harder to efficiently evaluate than the traditional ‘linear’ materials, such as books; computer-based training often has an interactive element; etc. Since the point in time when Hubbard published these concepts in 1987, the explosion of digital media, such as CD-ROMs and the Internet, has exacerbated the issues he noted and presented even more factors to be considered.
As educators and interested parties began to tackle the problem of evaluating these new technology-based training programs, various criteria from scholars and professionals emerged to assist in the process. The most basic was proposed by Garrett (1991: 86), who felt that only two ‘absolute’ criteria existed for evaluating ‘technology-based language materials’: the first is that “the language context be correct, authentic, and appropriate”; the second is that “the program run as it should, without bugs, without crashing.” Garrett believed that other than these two ‘absolute’ criteria, no other criteria should be used because “a package judged unacceptable by one teacher for any number of reasons may adequately fill a specific need for another.”
Garrett was not alone in his quest to create a checklist for evaluating the new technology, however each group or individual came up with a new set of criteria. Taylor (1987) believed there should be three ‘fundamental instructional factors’ in evaluating software, the York Educational Software Evaluation Scales utilize four (Dudley-Marling & Owston, 1987: 26) and Barker and King (1993: 309) proposed 12 unique criteria. The Educational Software Evaluation Consortium published 22 important evaluation factors (Borton & Rossett, 1989: 437). In the end, studies (Borton & Rossett, 1989; Murphy, 1995) show that there is little to no agreement on the minimum or mandatory criteria for evaluating this new breed of courseware.
In 1998, the editors of the Guidelines for the Evaluation of Instructional Technology Resources (Vaille, 1998) introduced a change in focus that has persisted through the years, as they noted that evaluators are now emphasizing “curriculum content, instructional design, and the needs of the learners...” (p. 2) when evaluating computer-based training. These basic categories are the foundation for the evaluation that follows.
The emphasis of this paper is on “the assessment of the quality and potential of a software application before it is used with students” (Squires & McDougall, 1996: 147). This evaluation is not intended to take the place of the highly valuable information obtained through summative evaluations (McDougall and Squires (1995a: 271), but rather to draw on the usefulness of an evaluation checklist, containing written assessments of curriculum content, instructional design, and the needs of the learners, to assist in narrowing the field of products on which a summative evaluation could be performed prior to full fielding of the training.
Overall Objectives of the Ethics Training Requirement
According to the Raytheon Business Ethics and Compliance Home Page (2008), “…Ethics education helps employees build an ethics check into their decision-making process, and thereby make decisions consistent with our values….this enables employees to spot issues and seek guidance proactively, before a problem arises.” The site states that “all employees will complete yearly training through the Raytheon Ethics Program” to meet these objectives. Based on this statement, the selected training should meet the following objectives:
1)Given education through the Raytheon Ethics Program, employees will build an ethics check into their decision-making process
2)Given education through the Raytheon Ethics Program, employees will make ethical decisions consistent with the Raytheon values
3)Given education through the Raytheon Ethics Program, employees will spot ethical issues
4)Given education through the Raytheon Ethics Program, when faced with an ethical decision employees will seek guidance proactively
The selected training, be it only one or a combination of both products, should support all of these objectives. If it does not, additional or alternative training should be selected.
Raytheon NIS Employees
Raytheon NIS consists of 237 employees located at 5 sites within the continental United States and 3 undisclosed overseas sites. NIS specializes in the engineering, operation, management and maintenance of mission critical, high-performance networks that support the communication of integrated voice, data, video, and imagery in a wide-area environment. The networks primarily exist in Department of Defense and state-level health care environments. The employees range from basic administrative assistants with a high school diploma and less than a year of experience to doctorate level engineers with over 20 years of experience. Two of the employees have significant hearing deficits and one employee has a significant vision deficit. All employees are fluent in English, but 36 list English as their secondary language. Based on this profile, training must be useable without sound and understood by a text-reader, comprehended at the high school reading level, and in English.
Raytheon NIS Technical Environment
The following chart depicts the minimum technical environment at each of the 5 continental and 3 overseas sites:
Site / Network / Model / RAM / Personal Storage / OperatingSystem / CD Drive / Speakers
1 / NIS Headquarters (Raytheon) / Gateway FX400 XL / 512MB / 80GB / Windows 2000 / Yes / Yes
2 / Defense Information Systems Agency / Dell Latitude D610 Laptop / 512MB / 20GB / Windows 2000 / Yes / No (disabled)
3 / Department of the Interior / Dell Dimension 4700 / 512MB / 160GB / Windows 2000 / Yes / No
4 / Midwest Operations Office (Raytheon) / Gateway FX400 XL / 512MB / 80GB / Windows 2000 / Yes / Yes
5 / District of Columbia Health Care / Dell Dimension 2250 / 1GB / 20GB / Windows 2000 / Yes / No
6 / Undisclosed DoD / Dell Latitude D830
Laptop / 4GB / 160GB / Windows XP / Yes / Yes
7 / Undisclosed DoD / Dell Latitude D830
Laptop / 4GB / 160GB / Windows XP / Yes / Yes
8 / Undisclosed DoD / Dell Latitude D830
Laptop / 4GB / 160GB / Windows XP / Yes / Yes
Basic Evaluation of Products Within the Environments
In order to verify that the training modules would basically function in all environments, visits were made to the four sites within driving distance of Washington, DC, and CDs were mailed to voluntary representatives within the Information Technology Department at the other four sites. The following questions were answered for each CD at all sites utilizing the above stated minimum technology environments and the baseline software installation for that location:
Question / Site 1 / Site 2 / Site 3 / Site 4 / Site 5 / Site 6 / Site 7 / Site 8Does the CD drive read the CD? / Yes / Yes / Yes / Yes / Yes / Yes / Yes / Yes
Does the training product automatically begin playing? / Yes / No / No / Yes / No / No / No / No
Does the training product load? / Yes / No / No / Yes / Yes / Yes / Yes / Yes
Once the product has loaded, can you utilize the navigation? / Yes / N/A / N/A / Yes / Yes / Yes / Yes / Yes
Can you hear the audio in the training segments? / Yes / N/A / N/A / Yes / No / Yes / Yes / Yes
Can you interact with the assessments? / Yes / N/A / N/A / Yes / Yes / Yes / Yes / Yes
Do the assessments keep score for you? / Yes / N/A / N/A / Yes / No / Yes / Yes / Yes
Based on these results, the six of the sites will have to manually start the training after the CDs are inserted, and one site is unable to hear the audio. More concerning are the two work sites that are not able to utilize the products on their work machines and one work site where the assessment feature does not keep score. In order to overcome these issues, allowances will have to be made for employees at those sites to partake in the training at home or another facility.
Product Evaluation – Business Ethics: An Introduction
General Product Description
Product Title
Business Ethics: An Introduction
Product Developer(s)/Producer(s)/Publisher
LRN Corporation
Intended Learning Environment
This software is custom-made and is presumably intended to be used at various Raytheon work sites world-wide. Individuals are assumed to have access to a personal computer or laptop, with optional speakers, and the ability to set aside 30-60 minutes to complete the training. Learners are assumed to have access to a facsimile machine in order to report completion of the training.
Media Characteristics
This product is presumably not intended for use in conjunction with live instructors. It contains text, audio, simple animation, and interactive questions and responses.
Material Organization
The material is organized in a linear fashion, with and introduction, two main lessons that include sub-sections and “self-check” questions, and a capstone quiz at the end.
Content
Basic Content Description
This product contains two main lessons, consisting primarily of text and audio information accompanied by illustrations, that cover the theory behind business ethics, why business ethics is practical, the definition of compliance, and how business ethics and compliance function together. Throughout the lessons the learner’s knowledge is tested with questions. At the end of the main body of training is a quiz that the learner is required to pass in order to receive credit for the product.
Stated Objectives
Within the introduction, the following objective is stated:
- Teaches the basics about business ethics.
Content Evaluation
This content is geared for individuals who are not familiar with business ethics. It does cover the basics of the subject, but does not discuss the specific regulations and values of Raytheon Corporation.
Instructional Context Description and Evaluation
Type(s) of Learning Context(s)
This training product utilizes the tutorial and drill & practice learning contexts. It presents new information in a linear format and requires a minimal amount of practice using the information. This use of the tutorial context does not require the learner to become heavily involved in detailed tasks, but rather allows them to sit back and take in the presented information before confirming what they have learned. Following the main body of the training is a drill & practice section that provides practice and feedback concerning the new knowledge, defined concepts, and rules.
Functions of the Learning Context(s)
In the case of this training, the tutorial learning context is both orienting and instructional. The product initially presents the learner with the big picture concerning the topic of ethics by describing both ethical and unethical situations. From that point, learners enter the first lesson, where they are presented with text and audio information and then quizzed on what they have learned. The second lesson follows the same format, with information preceding a knowledge check. Minimal instructor intervention is required, as the product provides immediate feedback. At the end of both lessons, learners enter into the drill & practice section in order to be assessed.
Effective Context(s)
The tutorial learning context is effective in the instructional function within the main body of the training, but the orienting function at the beginning of the training would be better enhanced by use of a more engaging context, such as simulation or situation exploration, so that students would feel more involved and eager to continue. If a live instructor were present, this section could be enhanced by involving the learner in brainstorming situations that may fall into either the ethical or unethical category. Unfortunately, the intended environment for this training does not support that technique.
Motivational Strategies
The product uses attention, relevance, and satisfaction to motivate learners throughout the training process. Incongruity is used to gain and retain the attention of the learner. Both ethical and unethical examples exist at various points within the training, which introduces equally plausible facts or principles, only some of which can be true. Learners are motivated to continue through the lesson to learn which situation is indeed ethical. Statements of future usefulness are employed to showcase the subjects’ relevance by asking learners to relate the instruction to their own future goals within NIS. Satisfaction is achieved by the immediate use of both positive and negative outcomes in the knowledge checks within the two lessons. Learners are congratulated for a correct answer and prompted to try again for an incorrect answer.
Context Evaluation
In summary, this product is primarily effective in its context; however there is a change that could be made to improve the effectiveness of the orientation section. A more engaging context could be used within the orientation section to initially capture the learner’s interest. This change would enhance the product’s context and effectiveness. Additionally, the product does not address the confidence category within the ARCS strategies. It would be helpful to include a statement that mentions that the pursuit of excellence within the knowledge check and quiz does not mean that anything short of perfection is failure. Without such a statement, the knowledge checks and capstone quiz are rather daunting.
Instructional Components and Conditions Descriptions and Evaluation
Instructional Design Components
This product introduces the instruction with an initial page that describes the tutorial that will follow. It is not very successful in gaining the learners’ attention, but does describe the big picture with this component. It utilizes the information/activity component of instructional design through the two main lessons, where concepts are discussed, examples and non-examples are given, and knowledge checks are performed without instructor guidance and intervention. Feedback is immediately provided as the learner answers the knowledge check questions. Assessmentis performed at the end of the training through a 10 question quiz.