Evaluation of the effectiveness of resources and expert feedback in a New Hire Technical Orientation
Submitted by Jee Park
ED 229D, Prof. D. Schwartz
12 June 2001
Abstract
In recent years e-learning has become a popular, cost-effective choice for corporate training, and large companies like Cisco Systems are developing in-house training offerings that are delivered via the internal network. Although some content, such as procedural knowledge and technical skill building, may be appropriate for technology enabled learning, other cognitive knowledge proves to be a more challenging fit. Cisco Systems is developing a New Employee Technical Orientation, a web-based course that will help new employees learn about the tools that they will use in their work at Cisco. This study evaluates the effectiveness of the resources included in the tool on learning strategic knowledge. Two groups of three subjects each were tested individually. Although both groups took Pre- and Post-tests and the Meeting Maker module, only Group A had full access to the resources in the Meeting Maker module. The results suggest that the resources are helpful in learning more efficient ways to use the application, but these results are not conclusive. Due to limitations on the number of subjects tested, the variability of the prior knowledge of the subjects, and the nature of the questions, the results are only subtly in favor of the benefits of using such resources. Furthermore, an e-learning offering intended to teach strategic knowledge about group practices may be augmented by the use of collaborative, offline instruction.
Introduction
E-learning has become the corporate training panacea, and companies like Cisco Systems are devoting resources to build in-house training offerings that are delivered via the web. Although some content, such as procedural knowledge and technical skill building, may be appropriate for technology enabled learning, other cognitive knowledge proves to be a more challenging fit. Cisco Systems is developing a New Employee Technical Orientation, a web-based course that will help new employees learn about the tools that they will use in their work at Cisco. The existing multimedia tutorials, online user guides, and video on demands teach employees the procedural, step by step processes in order to use the tools. However, knowing how to use the tool does not necessarily mean that the user knows how to use the tools strategically. The NETO tries to address this disparity by teaching employees how to use the tools in a way that takes into account the work of their colleagues. The aim of the NETO is to build awareness of an individual employee's place in a larger team of employees and to suggest ways in which to use the tools and problem solve situations so that the collective is taken into account. Problem solving processes and strategic knowledge are ambiguous skills that have multiple, equally effective alternatives. This type of content is difficult to put into an e-learning framework. However, since Cisco is determined to offer the NETO via its intranet, the challenge is to teach problem solving and strategic knowledge about working on a team, through a medium that is often solitary and concerned with absolute truths.
With consideration of the limitations of the delivery mechanism, the NETO is structured around scenarios employs cognitive apprenticeship as its main pedagogical method. The NETO asks learners to situate themselves in the scenario, which poses a prototypical problem that they are likely to face while on the job. After receiving the benefits and limitations of the usage of various tools that may help them solve the problem which are a synthesis of interviews conducted by on-site experts, the learner decides how to solve the problem. After the learner submits his or her own problem solution, he or she receives feedback, which consists of "expert" problem solutions and the rationale behind them.
This study will determine if the resources provided to the learner in the NETO, including the benefits and limitations and the expert feedback, improve the strategic problem solving abilities of learners. More broadly, the study will also help in determining if e-learning is a plausible way to teach individuals cognitive skills such as their impact on the work practices of others and strategic problem solving, two inherently ambiguous domains.
Two groups of three subjects were tested. In the Pre-Test both groups read the NT scenario from the NT server module of the NETO. Then, they answered two questions from the module and provide reasons for their solutions. The purpose of the pre-test was to assess current strategic knowledge. Group A took the Meeting Maker module in entirety. Group B took the Meeting Maker module, but was not able to view the chart of benefits and limitations of the tools used to solve the problem nor the feedback provided when checking their answers. In the Post-Test both groups read the scenario from the NT server module of the NETO and answer the same two questions as in the Pre-Test. The Pre-Test and Post-Test answers were compared for both groups and also compared between both groups.
My hypothesis is that the subjects who have access to the resources, particularly the benefits and limitations comparison chart and the expert feedback, will provide better answers in the post-test than the subjects who did not have the same resources. Explicitly, I expect that the answers to the post-test from the first group will be accurate, detailed, and include rationale that is suggested in the learning tool which is focused on networking, collaboration, and accountability to the team. In comparing the pre- and post-tests of both groups, the following characteristics will determine the effects of the resources on learning:
- the accuracy of their answers as compared to the expert problem solving methods provided in the NETO
- the level of detail in the problem solution
- the vocabulary used in the problem solution
General Description of the Tool
The NETO is a scenario-based, interactive web-delivered learning offering. Learners decide how to solve prototypical work problems that are posed in the scenarios. Currently there are two modules of the NETO that are completed. Eventually, there will be five stand-alone modules. Each of the modules incorporates one of the major tools that employees use in their work. The two completed modules focus on Meeting Maker, an individual and group scheduling application, and NT servers, the servers that house employees' public and private folders, facilitate the use of networked applications such as Meeting Maker, and provide collaborative space for teams and work groups.
The first part of the NETO is the Overview. The Overview is independent of the modules. The purpose of this section is to outline the purpose of the NETO and to situate the learner in the right frame of mind. Learners usually expect training to involve content that is procedural. They expect to acquire new, tangible technical skills after completing the training. This is not NETO's primary goal. NETO teaches strategic knowledge and beyond how to use to tools, it teaches how to use them effectively and in the context of team work. In an attempt to adjust the expectations of the learner, the Overview provides an analogy that demonstrates the difference between knowing how to solve a problem using a tool and solving the problem efficiently and while keeping the impact of the solution on others in mind.
Once the learner's frame of mind is shifted, he or she can begin with either of the two modules. Although both incorporate the same "virtual team", each of them is independent of the other. The scenarios in one module are not contingent upon the scenarios of the other module.
Both modules are structured in the same way. First, a brief introduction about the larger problem associated with the tool in the module is provided.
In order to understand the structure and nature of the content in the NETO, I will describe one module in detail. The Meeting Maker module begins with four introductory background sections. The first of these three sections includes an overview of the module, which outlines its purpose and goals. Following this brief section is a self-assessment that consists of three questions. The self-assessment provides a framework and context for the training. It is intended to shift the mindset of the subject in order to prepare him or her for the type of training he or she is about to begin in the module. It introduces the training as based in strategic knowledge instead of procedural knowledge, which may be the content expected by the subject. After the self-assessment, the subject is introduced to the larger problem surrounding the particular tool, Meeting Maker. That is, the importance of keeping a schedule at Cisco and the different ways in which to do this. There are three common ways that employees at Cisco keep a schedule and Meeting Maker is one of them. Meeting Maker is an individual and group scheduling application, and part of the goal of the module is to persuade the subject to use the tool. However, the larger goal is to teach the subject ways to use the tool strategically, while keeping in mind the implications of the subject's personal use of the tool on the workflow and practices of his or her entire team and, more broadly, the entire company. Once the subject is made aware of the different ways of keeping a schedule, he or she is provided with a chart comparing the benefits and limitations of each method. This is the last of the four introductory sections.
The next four sections are the bulk of the module. The first of these contains information about the framework of the following sections. In this section, the subject is introduced to the context of the problems that he or she will be solving in the scenarios. Both scenarios in the module pertain to the activities of a virtual team, of which the subject is the project manager. This first section introduces the members of the team and provides pertinent background information about each person. In the second section, the subject is then posed with a scenario that contains a problem. In order to solve the problem, three ways to solve the problem are suggested by three different members of the subject's virtual team. The subject can use these suggestions in order to formulate his or her own solution to the problem posed in the scenario. The third section, a second scenario is posed that is related to the first. The second scenario contains a problem that is more difficult than the first and requires him or her to rethink the original problem. Again, the subject is provided suggestions by another member of the virtual team. He or she may use these suggestions in order to formulate his or her own solution. The last of these four sections includes a chart of "What if" situations, which are a list of prototypical other problems and solutions of the same that the subject may face on the job. This section is much like Frequently Asked Questions sections that are common to many applications and web pages.
At each decision point in the aforementioned four modules, the subject can "check" his or her answer. If the subject chooses to check the answer, he or she receives feedback, which is paraphrased "expert" knowledge that was collected from observations and interviews during the development of the New Employee Technical Orientation.
The final two sections of the module are "Key Ideas", which is a summary of salient points from the module in entirety and "Resources", which is a list of URL's that are contained in the module. Each of the modules has one or more scenarios. All of the scenarios involve a "virtual team" of which the learner is a project manager. The learner is supposed to solve the problem in the scenario as if it was his or her own.
General Description of Experiment
This study was conducted by me and Heidi Chang, the other designer and developer of NETO.
For this study, only parts of the NETO were used. The Overview section was omitted. Subjects in both Groups A and B started with a Pre-Test which required them to answer the two scenarios from the NT server module of the NETO. The NT server module was chosen for two reasons. First, it was the only other module that existed in the NETO, and it was already in a format consistent to that of the Meeting Maker module. Second, the content of NT server module was unrelated to that of the Meeting Maker module in that knowing the NT server information did not explicitly affect the subjects' performance on the Meeting Maker module. The NT server module, therefore, served as a transfer problem. The NT server module was implicitly related to the Meeting Maker module in that both modules involved the broad ideas of networking, group processes, and implications of individual work on the group's work practices. However, the content of one did not explicitly relate to that of the other.
In the Pre-test, subjects read the first NT server scenario and answered aloud questions pertaining to it. We did this in order to have a point of reference with which to compare their Post-test answers. The first scenario had two questions. The first asked the subject to solve the problem posed in the scenario based on his or her own prior knowledge. The subject did not have any resources or other reference material to help him or her develop the solution. We asked the subject to explain the rationale for the solution he or she provided. We recorded the responses and noted the degree to which the solution and rationale included the ideas of networking, group processes, and implications of an individual's work on the group's work practices. In the second question, the subjects were provided three possible solutions that came from three different virtual team members who were introduced in the section before the scenario. The names of these virtual team members were Mary, Kathy, and Sam. We asked the subjects to rank the solutions in order from best to worst solution and to provide the rationale for the ranking order. We noted the rankings in order to compare them to the answers for the same question in the Post-Test.
The subjects then read the second NT serer scenario and answered one question. This question asked him or her to provide a free response solution. We asked the subject to say the solution aloud and to give the rationale for the solution. We recorded the solutions and noted the degree to which they included the ideas of networking, group processes, and implications of an individual's work on the group's work practices.
We had all subjects go through the Pre-Test without the aid of resources so that we could assess the prior knowledge with which they began the modules.
The subjects in Group A then continued to the Meeting Maker module and completed it in entirety. They were allowed to check their answers, read the feedback, and use any of the resources and references provided in the module. We did this to see if the references and resources, particularly the chart that compared the benefits and limitations to various solutions and the expert information that was provided in the feedback to checking the answers would teach subjects how to think strategically about the scenarios and solutions.
The subjects in Group B continued to the Meeting Maker module, but were not allowed to refer to the chart that compared the benefits and limitations to various solutions. Furthermore, they were not permitted to "check" their answers, which prevented them from gaining the feedback of experts. They were, however, allowed to see the definitions of words and terminology that was specific to the Meeting Maker application in order to understand the scenario. We permitted this because this information was pertinent to understanding the problem itself and the study would be flawed if the subject misunderstood or did not have enough information to understand the problem itself. Clarifying the problem statement does not influence the ways in which they solved the problem.
In the Post-test, both groups did the same parts of the NT server module as they did in the Pre-Test. We recorded the solutions and paid attention to the same things as in the Pre-Test in order to assess the differences, if any in their solutions and rationales.
We hoped to see that Group A, who had all of the resources available to them, would have more descriptive, more discussion of networking, group processes, and implications of individual work on the group's work practices, and rankings that were closer to those of the experts, than Group B. This would suggest that the resources, feedback, and references helped in learning strategic knowledge of the application's use.