Development of a “Lean” Evaluation System

  1. Definition of Evaluation
  2. Performance Criteria for a “ELE Freshman Design Course”
  3. Desired qualities of the student at the end of the course
  4. Strong documenters
  5. Consistently collects critical information in a single document, time-stamped and places the synthesis of ideas, with special labels for easy reference in the future.
  6. Learning ownership
  7. Team players
  8. Students will be effective, reliable team players who (within their role) value ideas of others yet effectively aids in concensus
  9. Strong life visionaries
  10. Highly connected
  11. Has investigated and used a number of campus resources, involved themselves in multiple activities, events, and organizations, established strong friendships and a few relationships with faculty and staff.
  12. Eager learners
  13. Responsible
  14. Self-regulating
  15. Problem solvers
  16. Students will be able to identify, formulate, open-ended problems well, propose multiple solution paths, implement solution methodologies using appropriate resources, valid results, and communicate the process effectively.
  17. Performance tasks and evaluation tools for an ELE.

Performance Task / Eval System Tools
Exams
Homework / Portfolios
Quizzes / Performance Reviews
Presentations / Rubrics
Projects / Scoring Sheets
Poster / Peer Evaluation* do not use
Team Problem Solving
Attendance
Participation
Electronic Discussion Participation
Reflective Learning Journals
Peer Assessment
Learning Journals / *innovative idea – use of external evaluator
Essays/Papers
Written Plans
  1. Qualities of good vs. bad evaluation system and rankings.

Qualities of Existing Eval Systems / Qualities of ELE Eval Systems
Biased / Unbiased 3,2,2 7
Unreliable / *Reliable 4,3,1,3 11
Implicit criteria / *Explicit criteria 4,3,5,3,4,1, 20
Limited Number of Task Types / Diverse task types 2,1,1,2 6
Faculty Centered / Learning centered 4,2, 6
Misaligned with disciplinary practices / Authentic 4,4,1, 9
Unfair / Fair 5, 5
High Stakes / Spread stakes 1,3, 4
Poorly correlated with learning outcomes / *Highly correlated with learning outcomes 5,5,5,2,5,1 23
Lead to circumvention (cheating) / Promotes integrity 1, 1
Game Playing / *Motivational 5,2,3,2,3,4 19
Low level learning focus / *High level learning focus 4,4,1,2 11
One-dimensional / Multi-dimensional
  1. Performance tasks and the outcomes they can target.

Performance Task / Team players / Highly connected / Problem solvers / Documenters
Exams / X
Homework / X / XX / X
Quizzes / X / X / X
Presentations / XX / X
Project*s / XX / XX / XXX
Poster / XX / X
Team Problem Solving / XXX / XXX / XX
Attendance / X
Participation / X
Electronic Discussion Participation / X / X / X / X
Reflective Learning Journals / X / XXX
Peer Assessment / X
Learning Journals / X / XX / XX
Essays/Papers / XX
Written Plans / X / X

Issues in Measuring Learning Outcomes

  1. repeatability in measurement (having inter/rater reliability among the various measures)
  2. measures are relevant, meaningful and accurate (on target) to all stakeholders
  3. meeting professional standards
  4. fairness and non-cultural bias
  5. measuring long term behavior is harder than short term outcomes
  6. clearly identifying criteria
  7. efficient
  8. clearly identifying purpose, context and use of the measurement
  9. measuring movement
  10. timing
  11. measuring affective components
  12. variety of measurement tools for calibration
  13. using right tool for right measurement
  14. user friendly
  15. communication to the student about the measuring of the outcome
  16. clarifying the learning outcomes
  17. alignment of the evaluation system to the performance criteria
  18. capturing students at their peak performance
  19. picking the appropriate performance task to capture the performances in peak mode
  20. alignment of the measuring system with the performance criteria and the evaluation system
  21. measuring formative elements of the outcome