Evaluating Student Understanding of Popular Misconceptions

True/False

Good for:

· Knowledge level content

· Evaluating student understanding of popular misconceptions

· Concepts with two logical responses

Advantages:

· Can test large amounts of content

· Students can answer 3-4 questions per minute

Disadvantages:

· They are easy

· It is difficult to discriminate between students that know the material and students who don't

· Students have a 50-50 chance of getting the right answer by guessing

· Need a large number of items for high reliability

Tips for Writing Good True/False items:

· Avoid double negatives.

· Avoid long/complex sentences.

· Use specific determinants with caution: never, only, all, none, always, could, might, can, may, sometimes, generally, some, few.

· Use only one central idea in each item.

· Don't emphasize the trivial.

· Use exact quantitative language

· Don't lift items straight from the book.

· Make more false than true (60/40). (Students are more likely to answer true.)

Matching

Good for:

· Knowledge level

· Some comprehension level, if appropriately constructed

Types:

· Terms with definitions

· Phrases with other phrases

· Causes with effects

· Parts with larger units

· Problems with solutions

Advantages:

· Maximum coverage at knowledge level in a minimum amount of space/preptime

· Valuable in content areas that have a lot of facts

Disadvantages:

· Time consuming for students

· Not good for higher levels of learning

Tips for Writing Good Matching items:

· Need 15 items or less.

· Give good directions on basis for matching.

· Use items in response column more than once (reduces the effects of guessing).

· Use homogenous material in each exercise.

· Make all responses plausible.

· Put all items on a single page.

· Put response in some logical order (chronological, alphabetical, etc.).

· Responses should be short.

Multiple Choice

Good for:

· Application, synthesis, analysis, and evaluation levels

Types:

· Question/Right answer

· Incomplete statement

· Best answer

Advantages:

· Very effective

· Versatile at all levels

· Minimum of writing for student

· Guessing reduced

· Can cover broad range of content

Disadvantages:

· Difficult to construct good test items.

· Difficult to come up with plausible distractors/alternative responses.

Tips for Writing Good Multiple Choice items:

· Stem should present single, clearly formulated problem.

· Stem should be in simple, understood language; delete extraneous words.

· Avoid "all of the above"--can answer based on partial knowledge (if one is incorrect or two are correct, but unsure of the third...).

· Avoid "none of the above."

· Make all distractors plausible/homoegenous.

· Don't overlap response alternatives (decreases discrimination between students who know the material and those who don't).

· Don't use double negatives.

· Present alternatives in logical or numerical order.

· Place correct answer at random (A answer is most often).

· Make each item independent of others on test.

· Way to judge a good stem: student's who know the content should be able to answer before reading the alternatives

· List alternatives on separate lines, indent, separate by blank line, use letters vs. numbers for alternative answers.

· Need more than 3 alternatives, 4 is best.

Short Answer

Good for:

· Application, synthesis, analysis, and evaluation levels

Advantages:

· Easy to construct

· Good for "who," what," where," "when" content

· Minimizes guessing

· Encourages more intensive study-student must know the answer vs. recognizing the answer.

Disadvantages:

· May overemphasize memorization of facts

· Take care - questions may have more than one correct answer

· Scoring is laborious

Tips for Writing Good Short Answer Items:

· When using with definitions: supply term, not the definition-for a better judge of student knowledge.

· For numbers, indicate the degree of precision/units expected.

· Use direct questions, not an incomplete statement.

· If you do use incomplete statements, don't use more than 2 blanks within an item.

· Arrange blanks to make scoring easy.

· Try to phrase question so there is only one answer possible.

Essay

Good for:

· Application, synthesis and evaluation levels

Types:

· Extended response: synthesis and evaluation levels; a lot of freedom in answers

· Restricted response: more consistent scoring, outlines parameters of responses

Advantages:

· Students less likely to guess

· Easy to construct

· Stimulates more study

· Allows students to demonstrate ability to organize knowledge, express opinions, show originality.

Disadvantages:

· Can limit amount of material tested, therefore has decreased validity.

· Subjective, potentially unreliable scoring.

· Time consuming to score.

Tips for Writing Good Essay Items:

· Provide reasonable time limits for thinking and writing.

· Avoid letting them to answer a choice of questions (You won't get a good idea of the broadness of student achievement when they only answer a set of questions.)

· Give definitive task to student-compare, analyze, evaluate, etc.

· Use checklist point system to score with a model answer: write outline, determine how many points to assign to each part

· Score one question at a time-all at the same time.

Oral Exams

Good for:

· Knowledge, synthesis, evaluation levels

Advantages:

· Useful as an instructional tool-allows students to learn at the same time as testing.

· Allows teacher to give clues to facilitate learning.

· Useful to test speech and foreign language competencies.

Disadvantages:

· Time consuming to give and take.

· Could have poor student performance because they haven't had much practice with it.

· Provides no written record without checklists.

Student Portfolios

Good for:

· Knowledge, application, synthesis, evaluation levels

Advantages:

· Can assess compatible skills: writing, documentation, critical thinking, problem solving

· Can allow student to present totality of learning.

· Students become active participants in the evaluation process.

Disadvantages:

· Can be difficult and time consuming to grade.

Performance

Good for:

· Application of knowledge, skills, abilities

Advantages:

· Measures some skills and abilities not possible to measure in other ways

Disadvantages:

· Can not be used in some fields of study

· Difficult to construct

· Difficult to grade

· Time-consuming to give and take

Grayson H. Walker Teaching Resource Center
The University of Tennessee at Chattanooga
Mail Code 4354
401 Hunter Hall
615 McCallie Ave.
Chattanooga, TN 37403-2598
(423) 425-4026
(423) 425-4025 (fax)

The table below presents both pros and cons for various test item types. Your selection of item types should be based on the types of outcomes you are trying to assess (see analysis of your learning situation). Certain item types such as true/false, supplied response, and matching, work well for assessing lower-order outcomes (i.e., knowledge or comprehension goals), while other item types such as essays, performance assessments, and some multiple choice questions, are better for assessing higher-order outcomes (i.e., analysis, synthesis, or evaluation goals). The italicized bullets below will help you determine the types of outcomes the various items assess.

With your objectives in hand, it may be useful to create a test blueprint that specifies your outcomes and the types of items you plan to use to assess those outcomes. Further, test items are often weighted by difficulty. On your test blueprint, you may wish to assign lower point values to items that assess lower-order skills (knowledge, comprehension) and higher point values to items that assess higher-order skills (synthesis, evaluation).

Item Type / Pros / Cons
Multiple Choice
(see tips for writing multiple choice questions below) / · more answer options (4-5) reduce the chance of guessing that an item is correct
· many items can aid in student comparison and reduce ambiguity
· greatest flexibility in type of outcome assessed: knowledge goals, application goals, analysis goals, etc. / · reading time increased with more answers
· reduces the number of questions that can be presented
· difficult to write four or five reasonable choices
· takes more time to write questions
True/False
(see tips for writing true/false questions below) / · can present many items at once
· easy to score
· used to assess popular misconceptions, cause-effect reactions / · most difficult question to write objectively
· ambiguous terms can confuse many
· few answer options (2) increase the chance of guessing that an item is correct; need many items to overcome this effect
Matching / · efficient
· used to assess student understanding of associations, relationships, definitions / · difficult to assess higher-order outcomes (i.e., analysis, synthesis, evaluation goals)
Interpretive Exercise
(the above three item types are often criticized for assessing only lower-order skills; the interpretive exercise is a way to assess higher-order skills w/ multiple choice, T/F, and matching items) / · a variation on multiple choice, true/false, or matching, the interpretive exercise presents a new map, short reading, or other introductory material that the student must analyze
· tests student ability to apply and transfer prior knowledge to new material
· useful for assessing higher-order skills such as applications, analysis, synthesis, and evaluation / · hard to design, must locate appropriate introductory material
· students with good reading skills are often at an advantage
Supplied Response / · chances of guessing reduced
· measures knowledge and fact outcomes well, terminology, formulas / · scoring is not objective
· can cause difficulty for computer scoring
Essay / · less construction time, easier to write
· encourages more appropriate study habits
· measures higher-order outcomes (i.e., analysis, synthesis, or evaluation goals), creative thinking, writing ability / · more grading time, hard to score
· can yield great variety of responses
· not efficient to test large bodies of content
· if you give the student the choice of three or four essay options, you can find out what they know, but not what they don't know
Performance Assessments
(includes essays above, along with speeches, demonstrations, presentations, etc.) / · measures higher-order outcomes (i.e., analysis, synthesis, or evaluation goals) / · labor and time-intensive
· need to obtain inter-rater reliability when using more than one rater

The table below presents tips for designing two popular item types: multiple choice questions and true/false questions.

Tips for Writing Multiple Choice Questions / Tips for Writing True/False Questions
· Avoid responses that are interrelated. One answer should not be similar to others.
· Avoid negatively stated items: "Which of the following is not a method of food irradiation?" It is easy to miss the the negative word "not." If you use negatives, bold-face the negative qualifier to ensure people see it.
· Avoid making your correct response different from the other responses, grammatically, in length, or otherwise.
· Avoid the use of "none of the above." When a students guesses "none of the above," you still do not know if they know the correct answer.
· Avoid repeating words in the question stem in your responses. For example, if you use the word "purpose" in the question stem, do not use that same word in only one of the answers, as it will lead people to select that specific response.
· Use plausible, realistic responses.
· Create grammatically parallel items to avoid giving away the correct response. For example, if you have four responses, do not start three of them with verbs and one of them with a noun.
· Always place the "term" in your question stem and the "definition" as one of the response options. / · Do not use definitive words such as "only," "none," and "always," that lead people to choose false, or uncertain words such as "might," "can," or "may," that lead people to choose true.
· Do not write negatively stated items, as they are confusing to interpret: "Thomas Jefferson did not write the Declaration of Independence." True or False?
· People have a tendency to choose "true," so design at least 60% of your T/F items to be "false" to further minimize guessing effects.
· Use precise words (100, 20%, half), rather than vague or qualitative language (young, small, many).
· Avoid making the correct answer longer than the incorrect answer (a give-away