Lievens, F., Buyse, T., & Sackett, P.R. (2005). The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. Journal of Applied Psychology.442 – 452.

  1. Parallels between higher educational admissions and personnel selection
  2. History of cognitive measures of ability and achievement
  3. i.e. – GRE, SAT, ACT (Education) vs. Job knowledge & Cognitive abilities (Personnel select)
  4. Interest in exploring more non-cognitive predictors
  5. What kinds?
  6. Personality
  7. Interpersonal skills
  8. Alternative presentation formats
  9. Letters of recommendation
  10. Personal statements
  11. References
  12. Why?
  13. Lower adverse impact
  14. Broadening of the criterion domain
  15. Employment setting - beyond task performance
  16. OCBs
  17. CWB
  18. Education setting – beyond GPA
  19. Social skill
  20. Citizenship
  21. Lifelong learning
  22. Growing interest in SJTs
  23. Employment
  24. SJTs have significant criterion related validities
  25. Incremental validity over cognitive and personality tests
  26. Appear job related (more favorable response to them)
  27. Lower adverse impact than traditional cogtests
  28. Education
  29. Tacit knowledge inventory (SJT) had incremental validity over GMAT
  30. SJT predicted performance on leadership and perseverance but not GPA
  31. US Army Project A
  32. Cog measure most valid predictors of task performance
  33. Personality measure were best to predict effort, leadership, and CWB
  34. MAIN POINT: There is a need to tap such various constructs as important for performance in differing jobs with SJTs
  1. The present study
  2. IVs: Cognitive ability, science knowledge, work sample (reading comp), video SJT for doctor/patient interpersonal skills
  3. Monitored school performance of 4 cohorts for 5 years
  4. HYP #1: Cog predictors will be significantly related to GPA
  5. HYP #2: If GPA is based on interpersonal skills and medical science courses, SJT measuring interpersonal skills will be significantly related to GPA
  6. HYP #3: If GPA is based on interpersonal skills and medical science courses, SJT measuring interpersonal skills will explain incremental variance over cognitive predictors
  7. HYP #4: Interpersonal SJT will be significantly related to grades in courses that stress interpersonal skill development
  8. Results
  9. Internal consistency of SJT = .39
  10. Why so low?
  11. “multidimensional situations and response options” inherent to SJTs.. (this is confusing to me)
  12. GPA criterion was stable across the 4 year span
  13. .72 - .78 correlation
  14. H1 supported: cogscores correlated w/ GPA @ .52
  15. H2 supported: difference score correlations for the SJT and criterion significant as years in school increased
  16. H3 supported: In universities that valued courses about interpersonal skills in computing GPA, SJT accounted for 1%, 2%, 6%, and 7% incremental variance in scores for each of the respective school years
  17. H4 supported: r = .21 SJTs have more validity in more interpersonally oriented courses
  18. Contributions of study
  19. Incremental validity of SJT over cognitive test
  20. Prior research on educational SJTs were done in lab settings (this one wasn’t)
  21. Criterion data obtained over multiple years (not cross-sectional data)
  22. There’s still great utility for traditional cognitive predictors
  23. SJTs can be a useful/valid complement to traditional tests
  24. Future directions
  25. Needed research in area of adverse impact
  26. Less g-loaded the better (apparently)
  27. Effects of practice/coaching on SJT scores and validity
  28. Is predictive validity of video SJT better than written SJT?
  29. Limitations
  30. This happened in Belgium (they eat a lot of chocolate, drink more beer than usual, and their medical school admission procedures are quite different from the rest of the world, therefore this may not quite generalize)
  31. Data was only collected for 4 years
  32. Did not actually measure job performance after school
  33. Low internal consistency
  34. So there apparently is validity here without much reliability??? Hmm..