Planning Tests and Assessments

Four Steps in Planning an Assessment

  1. Deciding its purpose
  2. Developing test specifications
  3. Selecting best item types
  4. Preparing items

Step 1: Decide the Purpose

What location in instruction? Or, the role of time in assessment!

  1. pre-testing
  2. readiness
  3. limited in scope
  4. low difficulty level
  5. serve as basis of remedial work, adapting instruction
  6. pretest (placement)
  7. items similar to outcome measure
  8. but not the same (like an alternative form)
  9. during instruction
  10. formative
  11. monitor learning progress
  12. detect learning errors
  13. feedback for teacher and students
  14. limited sample of learning outcomes
  15. must assure that mix and difficulty of items sufficient
  16. try to use to make correction prescriptions (e.g., review for whole group, practice exercises for a few)
  17. diagnostic
  18. enough items needed in each specific area
  19. items in one area should have slight variations
  20. end of instruction
  21. mostly summative –broad coverage of objectives
  22. can be formative too

Step 2: Develop Test Specifications

  • Why? Need good sample!
  • How? Table of specifications (2-way chart, "blueprint")
  • Prepare list of learning objectives
  • outline instructional content
  • prepare 2-way chart
  • or, use alternative to 2-way chart when more appropriate
  • doublecheck sampling

Sample of a Content Domain (such as this course)

  1. trends/controversies in assessment
  2. interdependence of teaching, learning, and assessment
  3. purposes and forms of classroom assessment
  4. planning a classroom assessment (item types, table of specs)
  5. item types (advantages and limitations)
  6. strategies for writing good items
  7. compiling and administering classroom assessments
  8. evaluating and improving classroom assessments
  9. grading and reporting systems
  10. uses of standardized tests
  11. interpreting standardized test scores

Sample Table of Specifications (For content from this course)

Sample SLOs (you would typically have more) / Bloom Levels
Remember / Understand / Apply / Analyze / Evaluate / Create
Identifies definition of key terms (e.g., validity) / X
Identifies examples of threats to test reliability and validity / X
Selects best item type for given objectives / X
Compares the pros and cons of different kinds of tests for given purposes / X
Evaluates particular educational reforms (e.g., whether they will hurt or help instruction) / X
Create a unit test / X
Total number of items

Spot the Poor Specific Learning Outcomes (use previous table of specifications)

Which entries are better or worse than others? Why? Improve the poor ones.

  1. Knowledge
  2. Knows correct definitions
  3. Able to list major limitations of different types of items
  4. Comprehension
  5. Selects correct item type for learning outcome
  6. Understands limitations of true-false items
  7. Distinguishes poor true-false items from good ones
  8. Application
  9. Applies construction guidelines to a new content area
  10. Creates a table of specifications
  11. Analysis
  12. Identifies flaws in poor items
  13. Lists general and specific learning outcomes
  14. Synthesis
  15. Lists general and specific content areas
  16. Provides weights for areas in table of specifications
  17. Evaluation
  18. Judges quality of procedure/product
  19. Justifies product
  20. Improves a product

Why are These Better Specific Learning Outcomes?

  1. Knowledge
  2. Selects correct definitions
  3. Lists major limitations of different item types
  4. Comprehension
  5. Selects proper procedures for assessment purpose
  6. Distinguishes poor procedures from good ones
  7. Distinguishes poor decisions/products from good ones
  8. Application
  9. Applies construction guidelines to a new content area
  10. Analysis
  11. Identifies flaws in procedure/product
  12. Lists major and specific content areas
  13. Lists general and specific learning outcomes
  14. Synthesis
  15. Creates a component of the test
  16. Provides weights for cells in table of specifications
  17. Evaluation
  18. Judges quality of procedure/product
  19. Justifies product
  20. Improves a product

Step 3: Select the Best Types of Items/Tasks

What types to choose from? Many!

  1. objective--supply-type
  2. short answer
  3. completion
  4. objective--selection-type
  5. true-false
  6. matching
  7. multiple choice
  8. essays
  9. extended response
  10. restricted response
  11. performance-based
  12. extended response
  13. restricted response

Which type to use? The one that fits best!

  1. most directly measures learning outcome
  2. where not clear, use selection-type (more objective)
  3. multiple choice best (less guessing, fewer clues)
  4. matching only if items homogeneous
  5. true-false only if only two possibilities

Strengths and Limitations of Objective vs. Essay/Performance

Objective Items

  • Strengths
  • Can have many items
  • Highly structured
  • Scoring quick, easy, accurate
  • Limitations
  • Cannot assess higher level skills (problem formulation, organization, creativity)

Essay/Performance Tasks

  • Strengths
  • Can assess higher level skills
  • More realistic
  • Limitations
  • Inefficient for measuring knowledge
  • Few items (poorer sampling)
  • Time consuming
  • Scoring difficult, unreliable

Step 4: Prepare Items/Tasks

Strategies to Measure the Domain Well—Reliably and Validly

  1. specifying more precise learning outcomes leads to better-fitting items
  2. use 2-way table to assure good sampling of complex skills
  3. use enough items for reliable measurement of each objective
  4. number depends on purpose, task type, age
  5. if performance-based tasks, use fewer but test more often
  6. keep in mind how good assessment can improve (not just measure) learning
  7. signals learning priorities to students
  8. clarifies teaching goals for teacher
  9. if perceived as fair and useful

Strategies to Avoid Contamination

  1. eliminate barriers that lead good students to get the item wrong
  2. don’t provide clues that help poor students get the item correct

General Suggestions for Item Writing

  1. use table of specifications as guide
  2. write more items than needed
  3. write well in advance of testing date
  4. task to be performed is clear, unambiguous, unbiased, and calls forth the intended outcome
  5. use appropriate reading level (don’t be testing for ancillary skills)
  6. write so that items provide no clues (minimize value of "test-taking skills")
  7. a/an
  8. avoid specific determiners (always, never, etc.)
  9. don’t use more detailed, longer, or textbook language for correct answers
  10. don’t have answers in an identifiable pattern
  11. write so that item provides no clues to other items
  12. seeming clues should lead away from the correct answer
  13. experts would agree on the answer
  14. if item revised, recheck its relevance

Planning assessments 1