Assessment Item Writing Workshop

Item Writing Workshop

Assessment Item Writing Workshop

Ken Robbins

FDN-5560 Classroom Assessment

Dr Olson

Apr21, 2005

Agenda

Pre-assessment

PowerPoint

Assessment Item Writing Workshop

Classroom assessment is one of the tasks North Carolina’steachers detest. As a group we understand the significance of assessment, but creating an assessment to determine students’ success for a learning target is not a favorite activity. The major cause of this discomfort is the lack of testing and assessment training at the basic level. This workshop will attempt to remedy the lack of knowledge in basic item writing and provide specific skills for the participant. As teachers, we are easily enticed by the assessments provided by our textbook publishers because the tests are readily available and require minimum effort. But, as we will discover, textbook assessments are not the most appropriate measure of student achievement. To assess the objectives taught in individual classrooms, teachers should develop their own assessments specifically created to assess the learning targets of the lesson. Assessment is most effectively used to guide instruction and to provide an indicator if current teaching methods are relevantto the student population. One of the first tasks for every teacher is to prepare an assessment for objectives taught in class. The workshop will provide researched background evidence on textbook assessment deficiencies, a taxonomy for assessment based on Bloom’s taxonomy for the cognitive domain, and basic item writing guidelines to assist the classroom teacher in the development of suitable assessments.

LITERATURE REVIEW

There has been extensive research conducted in relation to individual classroom assessments and the deficiencies found in textbook publishers’ provided assessments. The GatewayAlternative School research (unpublished, Robbins, K. E., 2005) supported an original study conducted by David A. Frisbie, Diane U. Miranda, and Kristin K. Baker. This group researched textbook tests to determine the quality of publisher provided assessments. Their findings, published in Applied Measurement in Education 6(1), in 1993, determined that “ninety percent of the (provided) items were typically classified at the knowledge level” (p. 32). Their research only classified tests using the three lower levels of Bloom’s taxonomy and dismissed upper-level questions as insignificant because of the low percentages of the overall total. The Gateway research utilized the original six categories from Bloom’s taxonomy as demonstrated in Taxonomy of Educational Objectives. Benjamin S. Bloom (1956) and the others from the committee of the college and university examiners separated the cognitive domain into six categories; knowledge, comprehension, application, analysis, synthesis, and evaluation (p. 201-207). North Carolina has adopted the “Dimensions of Thinking” model adapted from Robert Marzano’s published research on the original taxonomy. This taxonomy utilizes nine categories derived directly from Bloom’s six categories (NCDPI, 1999). Even though the original six categories have been revised by a number of researchers, they provide a valuable reference for most current research on assessment taxonomy, and will be the focus for this workshop.

Research conducted at GatewaySchool (unpublished, Robbins K. E., 2005) involved some teacher prepared assessments and textbook tests utilized by the sample group of teachers. The team of Frisbie, Miranda, and Baker (1993) dealt exclusively with tests provided by textbook publishers. Research concluded that most teachers used publisher tests with confidence. According to the studies, teachers either used the tests in whole, or assimilated parts of the provided tests into their own teacher prepared tests. According to Gateway’s data, the majority of the questions were determined to be at the lower three levels of Bloom’s scale. In fact, ninety-six percent of the questions were in the lower three levels. Sixty-eight percent of the questions were determined to be at the knowledge level, fourteen percent at the comprehension level and fifteen percent at the application level. This appears to support Frisbie’s original research and suggests that additional teacher effort is required to enhance classroom assessments.

A number of different examples of taxonomy have been mentioned, but the one used for this workshop is will be A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives, a 2001 update of the original work. The updated version offers significant improvements in the guidelines for classifying questions into specific taxonomy levels. The revised version offers specific guidelines for each area, such as the one for the application level. “The Apply category consists of two cognitive processes: executing-when the task is an exercise . . . and implementing-when the task is a problem” (Anderson, 2001, p. 77). Today, our society demands that students provide evidence that they can apply the knowledge learned in school. To accomplish this feat and to ensure their future success, we must assess our students using the upper levels of the taxonomy scale. We can’t settle for Bloom’s original findings. In 1956, “Bloom found that over 95% of the test questions students encounter require them to think only at the lowest possible level” (Officeport, n. d.). This workshop will enable teachers to create assessments to develop talent and enhance cognitive processes in the student population. According to Bloom (1976) in Human Characteristics and School Learning; “modern societies no longer can content themselves with the selection of talent; they must find means of developing talent” (p. 17).

To develop talent in our students we must provide challenging assessments instead of the meaningless current assessments. One of the basic questions we must ask as teachers is: “Do the tests we currently utilize measure what we want our students to learn?” Textbook publishers sell the same textbook to school districts nationwide. The provided assessments are the same in California as in North Carolina. According to research, publisher assessments “do not cover objectives thoroughly, they only match the content of their unit objectives about half of the time, and they are dominated by items measuring at the knowledge level” (Frisbie 1993, p. 33). This workshop will provide the basic skills required to create a useful classroom assessment measuring the units’ learning targets. The first and most important step in developing an assessment is to define the learning target for assessment. In simple terms, a learning target is the knowledge or ability you wish the student to obtain. Once you have defined the learning targets, you can then create the assessment items and classroom instruction. Two major points of view are prevalent for the proper time to develop an assessment. Most teachers find themselves teaching a unit and then developing an assessment the day before the students are to finish the unit. This delayed action defies related research. One of the major points of view is to develop the assessment before the first minute of instruction to students. The proper steps to this viewpoint are; start with the required learning targets, develop an assessment to measure successful completion of the learning targets, and finally, design instruction to ensure coverage of the learning target as defined by the targets and the assessment (Popham 2005, p. 271). A slightly different approach published by Jerard Kehoe (1995)stresses the importance of developing the assessment immediately after each instructional period to ensure that the assessment shadows instruction provided by the teacher. For our use in the workshop, we willemphasize the importance of Popham’s method, with the understanding that a careful review should be accomplished after each instruction period to ensure that the assessment items were covered adequately.

This specific workshop will address multiple-choice (MC) item guidelines because MC items are easily used, easily scored and can be used to measure the higher levels of thinking required (Haladyna and Downing, 1989). The workshop will be based on the taxonomy of item-writing guidelines as developed by Haladyna, Downing and Rodriguez (2002). This team developed the taxonomy in a study of major textbooks on educational testing and research studies. The guidelines are attached (Atch 1) for further study and review. Each of the thirty-one items is significant for the classroom instructor and will be explained in detail during the workshop. The taxonomy is divided into five major categories. The first category is “content concerns” and addresses the actual content of the item, basing it on important aspects of the learning target, avoiding trick items, and using vocabulary appropriate for current students. The second category is “formatting concerns” which addresses the actual format of the items such as formatting the items vertically. The third category is “style concerns” and addresses areas of proper grammarand minimizing the amount of reading required. The fourth category is “writing the stem” and addresses areas such as including the central idea in the stem. The last category is “writing the choices” and addresses guidelines for writing both the proper answer and the distracters.

In addition to these thirty-one rules, we must remember that we are striving to increase the knowledge level of our students. To accomplish this increase, it is important to use Bloom’s taxonomy of the cognitive domain or an alternative when writing the actual items to ensure that the questions are at the appropriate level of the thought process. The levels are knowledge (recall), comprehension (interpreting), application (apply to new situation), analysis (break down), synthesis (combining into something new), and evaluation (judge) (University of Texas, 2004).

It is also important to develop questions that discriminate between knowledgeable and unknowledgeable students. This step may be difficult to assess before the items are answered by the students, but if the majority of students miss an item or answer the item correctly, it is of little use in discrimination (Kehoe, 1995). Itemsshould be reviewed after the assessment is complete to ensure that the assessment measures the learning targets, and can provide proper inferences about the students taking the assessment.

As teachers, we must have a method to assess our students’ learning success for objectives taught in class. The workshop providesvaluable evidence on publisher’s textbook assessment deficiencies. If provided assessments are used at all, do not use them intact. The workshop also provides a taxonomy for assessment based on Bloom’s taxonomy for the cognitive domain, and basic item writing guidelines to assist the classroom teacher in the development of suitable assessments. With this training, each workshop attendee should be able to create an assessment to measure student learning, and each assessment will measure what the teacher intends to measure, not what a publisher in California intends to measure.

Reference List

Anderson, L.W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., Raths, J., & Wittrock, M. C. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. New York: Longman.

Bloom, B. S. (1976). Human characteristics and school learning. New York: McGraw-Hill.

Bloom, B. S., & et al.(eds) (1956). Taxonomy of educational objectives. New York: David McKay Company.

Frisbie, D. A., Miranda, D.U., & Baker, K. K. (1993). An evaluation of elementary textbook tests as classroom assessment tools. Applied Measurement in Education, 6(1), 21-36.

Haladyna, T. M., & Downing, S. M. (1989). A taxonomy of multiple-choice item-writing rules. Applied measurement in education, 2(1), 37-50.

Haladyna, T. M., Downing, S. M. & Rodriquez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309-334.

Kehoe, J. (1995). Writing multiple-choice test items. Practical Assessment, Research & Evaluation, 4(9).

Kehoe, J. (1995). Basic item analysis for multiple-choice tests. Practical Assessment, Research & Evaluation, 4(10).

North Carolina Department of Public Instruction. (1999). Understanding North Carolina tests: Thinking skills level. Assessment brief, Vol 6, No 4.

OfficePort. (n.d.). Bloom’s taxonomy. Retrieved February28, 2005, from

Popham, W. J. (2005). Classroom assessment: What teachers need to know. 4th ed. Boston: Pearson Education.

University of TexasLearningCenter. (n. d.). Bloom’s taxonomy’s model questions and key words. Retrieved February 28, 2005, from