1
ANNOTATED BIBLIOGRAPHY
800-Level Paper: Annotated Bibliography
I chose the field of instructional design for this annotated bibliography. Even with this focus, the types of peer-reviewed studies were very diverse and gave insight into a number of different contexts and applications of needs assessment and analysis tools, techniques, and processes.
Article 1
Corn, J. O. (2009). Investigating the quality of the school technology needs assessment (STNA) 3.0: A validity and reliability study. Educational Technology Research and Education, 58, 353-376.
Author and Purpose
The author, Jenifer Corn from North Carolina State University, identified shortcomings in previoustechnology needs analysis tools in use by K-12 schools in North Carolina and worked to develop and test the validity of a new survey tool.
Context, Participants, and Data Collection
A large sample of 1,918 educators, teaching assistants, staff, and administrators from 32 elementary schools, nine middle schools, and eight high schools in North Carolina completed the author’s new needs assessment survey.
Logical Progression
The author focused on the identification and compilation of gaps, and did not address identifying causes and so the study does not include a needs analysis. The author used a pilot test to confirm the 86 survey items, in four constructs and 12 sub-constructs, that appeared in the final assessment tool.
Findings
The author found that a tool that systemically included survey items asking about technology support, technology training, how technology is used, and the impact of technology in their classrooms, appeared to be an effective approach to gauge the overall needs of K-12 educators.
Conclusions
The researcher found that incorporating and applying the results of the study’s literature review, research, and pilot test, created a new assessment tool that could reliably help identify the technology needs of K-12 teachers. While the sample is relatively large, including additional middle school and high school educators may further support the validity of this needs assessment instrument.
Article 2
Eastmond, J. E. (1991). A needs assessment of primary school pupils of the Yi Nationality in the Sichuan province of China. Educational Technology Research and Education, 39(2), 102-105.
Author and Purpose
The author, J. Eastmond from Brigham Young University, was a member of a four-person research team sent to China to conduct a needs assessment and analysis of the impoverished Yi ethnic population of China. The team focused on learning the educational needs of the community in an effort to learn and prioritize these needs and to suggest solution strategies.
Context, Participants, and Data Collection
The Yi population of China were once enslaved and while now free did not have access to the same educational resources as other ethnic groups in China. The researchers used the Worldwide Model of Needs Assessment to identify the educational needs of the Zhao Jui village of central China. This model suggests including a wide variety of community participants in large, day-long, group discussions. The first focus group included Yi members as well as representatives from the majority Han population and a second group included only Yi members, overall, 30 Yi representatives participated in each focus group.
Logical Progression
The author’s needs assessment asked about, discussed, and gathered the educational needs of the mixed Yi and Han group and the specific needs of the second Yi focus group. The research team then analyzed these needs, was able to consolidate and prioritize them, and began to offer potential solution strategies.
Findings
The team found that the higher priority educational need of the community was for school age children to effectively learn to read and write Han Chinese. The second priority was to increase elementary school attendance.
Conclusions
The international context of instructional design needs assessment and analysis was interesting, and eye-opening considering the need of this population. Another interesting aspect was the focus groups each assuming that their primary need was for their young adults to attend higher education. The research team found that the more fundamental need was not the assumed needs of young adults but the needs of younger,school-aged children.
Article 3
Kaufman, R. (1977). Needs assessments: Internal and external. Journal of Instructional Development, 1(1), 5-8.
Author and Purpose
Kaufman describes two perspectives of a needs assessment in this article. An internal and an external viewwhich can both provide different starting points during a needs assessment, both can be advantageous depending on the context.
Context, Participants, and Data Collection
Kaufman’s context is educational environments, with internal perspectives of a needs assessment represented by educators and administrators inside of the school or school district (internal to the system). External perspectives on a needs assessment originate from outside the school or school district and are from the outside environment (external to the system).
Logical Progression
Often a needs assessment lead by internal educators is limited by the organization and are locked into “the way it is usually done.” However, a needs assessment lead by those from outside the system can be met with resistance by those internal to the organization.
Findings
Change is inevitable, and a needs assessment to trigger that change can originate from inside or outside the organization.
Conclusions
Needs assessments lead by internal system educators or lead by external entities can both have advantages and disadvantages in terms of successful change. The process that drives design, development, implementation, and evaluation can be thought of as a system with inputs that can be provided by both externally and internally originated needs.
Article 4
Kaufman, R., & Watkins, R. (1996). Cost-consequences analysis. Human Resources Development Quarterly, 7(1), 87-100.
Author and Purpose
Kaufman and Watkins collaborate again in this article where they present the use of a Cost-Consequences Analysis approach to determining the gaps between expenditures and results in an organization.
Context, Participants, and Data Collection
The context and participants for this approach can be virtually any organization that seeks to assess its needs. Data collection can be accomplished with surveys, interviews, and artifact and data analysis. The authors focus on economists in this paper, and use definitions and financial terminology that will relate to that audience.
Logical Progression
A Cost-Consequences Analysis approach is similar to a traditional financial Return on Investment analysis, and both techniques could be effective tools to use during a needs assessment. However, a Cost-Consequence Analysis is less intensive than a Return on Investment analysis and can yield quicker results.
Findings
The steps in a Cost-Consequences Analysis generally map to Kaufman’s Organizational Elements Model, or auditing relates to Inputs, cost-efficiency relates to Processes, cost-effectiveness relates to Products, cost-benefits to Outputs, and cost-utility assessment and analysis relates to Outcomes.
Conclusions
Economists understand Return on Investment and the impact of a needs assessment based on cost and revenue. The authors relate Kaufman’s Organizational Elements Model and Cost-Consequences Analysis to monetary indicators to help economists understand the systemic OEM approach as well as the impact of their decisions and outputs on their systems, organizations, and society.
Article 5
Kumar, D., & Altschuld, J. (1999). Evaluation of interactive media in science education. Journal of Science Education and Technology, 8(1), 55-65.
Author and Purpose
Kumar and Altschuld describe an evaluation in this paper, or an analysis of a product or program that already exists. However, the reader can begin to see elements in their description of a science education program that could also be described as the three levels in their future needs assessment model.
Context, Participants, and Data Collection
This paper describes a project for Vanderbilt University and their science education program which was a partnership with the university’s college of arts and sciences, college of education, and local K-12 teachers. The authors used document review and developed an interview process to gauge how well the use of interactive video was meeting the needs of science teachers.
Logical Progression
The researchers created transcripts of each interview for further analysis and used experts to review the transcripts to help find common themes. Some of these common idea threads included the need to better overcome the teachers’ fear of learning science and the need for more hands-on activities.
Findings
Interestingly, the authors admit that they could have used a survey data gathering approach, and that this would have been easier than the interview approach they used. However, the authors also felt that the rich data collected via the interviews was well worth the additional effort.
Conclusions
The authors also found that critical to the success of the program was systemically considering the needs of the program’s clients, the needs of faculty and instructors, and the need for administrative, technical, and other infrastructure supports. These ideas, or the consideration for levels within a system or organization, likely helped inform their future three-phase needs assessment model.
Article 6
Magliaro, S. G., & Shambaugh, N. (2006). Student models of instructional design. Educational Technology Research and Education, 54(1), 83-106.
Author and Purpose
The authors of this study, Magliaro and Shambaugh, conducted a long-term study of the final class projects of their instructional design graduate students that explored how the learners developed their personal instructional design models.
Context,Participants, and Data Collection
Data was collected from the research projects of 178 graduate students in the authors’ 12 sections of a master’s levelinstructional design course over the course of seven years. The students were tasked to create their own instructional design models based on several known models that they studied during their class (such as models from Dick and Carey, Gagne, and Morrison, Ross, and Kemp).
Logical Progression
The authors describe that a mental model is a human construct used to explain and make sense if the world. Students created their own mental models based on this perspective, and defined for themselves a means to methodically create effective instructional designs.
Findings
The authors looked for commonality among the 178 models and found what can be described as a need analysis was included in only 59% of the models and a needs assessment was included in only 60% of student generated models.
Conclusions
I would have hoped that the number of students who recognized the importance of a needs assessment and analysis early in their design models would have been higher. That so few of them recognized the importance of a needs assessment and analysis in their design models was disappointing. I’m glad to have taken this class to better understand how important this step is in any instructional design model, process, or project.
Article 7
Tessmer, M. (1990). Environmental analysis: A neglected stage of instructional design. Educational Technology Research and Education, 38(1), 55-64.
Author and Purpose
Tessmer describes what he feels is a neglected area of needs assessment and analysis, specifically the need to also analyze the context of a potential instructional intervention. Investigating the environment that the instruction will occur is a critical part of a project design and is often neglected by instructional designers (as seen in Magliaro & Shambaugh, 2006).
Context, Participants, and Data Collection
Tessmer recommends that once an instructional designer realizes that a situation requires an instructional intervention, then an environmental analysis should become an early and critical design step. This analysis can be conducted via interview oran environmental survey form for small-scale projects. However, for larger projects more detailed site surveys and project team interviews may be required to collect data.
Logical Progression
An instructional environment analysis is required when considering that the context of the instruction will have an impact on the overall system. Designers shouldask about both the physical factors of the environment where the skill will be performed and the environment where the instruction will be performed.
Findings
Instructional and support environments, patterns of use, how the contextual environment will contribute to or hinder instruction, and the environmental resources available, are all important aspects to consider early in the needs assessment and analysis.
Conclusions
An environmental analysis is an important, and often overlooked early step in instructional design and should be considered to be a critical part of a needs assessment and analysis.
Article 8
Tessmer, M., McCann, D., & Ludvigsen, M. (1999). Reassessing training programs: A model for identifying training excesses and deficiencies. Educational Technology Research and Education, 47(2), 86-99.
Author and Purpose
The authors of this paper, which includes Tessmer and reflects his instructional evaluation background, focus on an aspect of needs assessment and analysis that we have not covered in class; a needs assessment that identifies not only gaps but excesses.
Context, Participants, and Data Collection
The needs assessment approach proposed in this paper can be applied in a wide variety of instructional and non-instructional contexts. However the focus is the assessment in instructional contexts, two specific examples include an engineer training program (N = 29 field engineers) and a medical office manager training program (N = 201 office mangers). The authors served as consultants and used surveys, interviews, and existing document, artifact, and data analysis to collect data.
Logical Progression
The authors propose a CODE process, which is an acronym for Criticality, Opportunity, Difficulty, and Emphasis. A needs assessment that results is a situation where a task is found to be high in criticality (an important business need), low in opportunity (limited opportunities to practice), high in difficulty (intrinsically hard subject matter), and low in emphasis (limited time for training), would indicate a training deficiency, or a gap. However, a needs assessment that results is a situation where the task is found to be low in criticality (not an important business need), high in opportunity (many opportunities to practice), low in difficulty (intrinsically easy subject matter), and high in emphasis (more than sufficient time for training), would indicate a training excess.
Findings
The authors found the results of their examples to be valid indicators for the use of their CODE approach and used it to indicate in these two cases where training resources should be redeployed from areas of excess to areas of actual need (performance gaps).
Conclusions
The search for not only gaps but also excesses during a needs assessment is interesting. While the authors specifically say their CODE process is best “… used where students or workers have already received training and are purportedly using the skills they have learned” (Tessmer, McCann, & Ludvigsen, 1999, p.97), the CODE process can likely be applied in both aconfirmative evaluation situation and in a needs assessment and analysis.
Article 9
Watkins, R.,Kaufman, R. (1996). An update on relating needs assessment and analysis. Performance Improvement, 35(10), 10-13.
Author and Purpose
Ryan Watkins was a graduate student when he wrote this paper together with Roger Kaufman, together they further explain and explore Kaufman and Valentine’s Organizational Elements Model.
Context, Participants, and Data Collection
The Organizational Elements Model, or OEM, can be applied in a wide variety of contexts to improve human performance by identifying and examining an organization’s needs. The authors make an important distinction between a needs assessment and a needs analysis, or the need to collect data from participants to find the gaps, and then to analyze those gaps for their causes.
Logical Progression
The authors provide a systemic basis for the OEM model as they describe how an assessment would typically collect data related to outputs, or an activity’s ‘ends’. An analysis of those ends would involve an investigation into an activity’s ‘means’, or system inputs.
Findings
The authors guide HPT practitioners to define ‘needs’ as performance gaps as opposed to the more common definition of a need as a more generalized want or desire. Practitioners are also guided through the differences between a needs assessment and a needs analysis, and how both are critical to deriving the true needs of an organization.
Conclusions
The authors remind the reader that a system’s inputs are related to system’s processes, which are related to a system’s micro products, macro outputs, and mega outcomes.
Article 10
Watkins, R., Wedman, J. (2003). A process for aligning performance improvement resources and strategies. Performance Improvement, 42(7), 9-17.
Author and Purpose
Watkins and Wedman further build on Kaufman’s Organizational Elements Model with an overlay of needs assessment and asset assessment in an effort to help organizations understand and align their strategies and resources.
Context, Participants, and Data Collection
A needs assessment and an asset assessment can be generally applied in a variety of contexts, and can also use interviews, surveys, and analysis of existing data to collect information.
Logical Progression
These authors also differentiate between an assessment that identifies gaps and an analysis that explores the causes of those gaps, they also add an‘answering’ phase where the HPT practitioner is proposing solutions. This answering phase is similar to the third phase of Altschuld and Kumar’s approach.
Findings
Similar to other articles in this bibliography, these authors also identify the importance of both internal and external aspects and perspectives of an organization, as well as the need to consider micro, macro, and mega-level system outputs.
Conclusions
The authors propose that finding the gaps between ‘what is’ and ‘what should be’ at the micro, mega and macro-levels of an Organizational Elements Model constitutes a needs assessment. An asset assessment is part of the overall needs analysis and determines ‘what is’ and ‘what should be’ gaps in the Organizational Elements Model’s Processes and Inputs components. Knowing what assets are available, and comparing assets to needs can be used to begin to create potential solution strategies.