REPORT OF RESULTS ON COMMERCIAL COACHING

Barbara Griffin PhD

2013

These results have been published in peer reviewed academic journals1 and presented to teachers at forums for career advisors held at Monash University and the University of Western Sydney. Please do not reproduce the material without permission.

BACKGROUND

Current medical student selection in Australia draws on a large body of national and international research. This significant evidence base indicates that best practice in selection takes into account a range of skills, including cognitive and academic ability as well as non-cognitive skills and abilities. Typically, these are assessed by a combination of the ATAR score or a University Grade Point Average (GPA), the Undergraduate Medical and Health Sciences Admissions Test (UMAT) or the Graduate Australian Medical Schools Admission Test (GAMSAT), and interviews, either with a panel interview or a Multiple Mini Interview (MMI). Our research has focussed on selection into undergraduate medical degrees.

RESEARCH STUDIES

Results reported here are from the analyses of data from two main sources:

  1. Data from applicants to th University of Western Sydney School of Medicine – an undergraduate medical degree. There are approximately 3000 applicants each year and from these about 350 are shortlisted for interview.
  2. Data from 432 Year 12 students from 31 high schools across Sydney and Melbourne.
  • 58.5% were females
  • 51% of the participants were enrolled in a ‘selective’ high school

______

1Griffin, B. (in press). The ability to identify criteria: its relationship with social understanding, preparation, and impression management in affecting predictor performance in a high stakes selection context. Human Performance.

Griffin, B., Yeomans, N. & Wilson I. (in press). Studnets coached for an admission test perform less well throughout a medical course. Internal Medicine Journal.

Griffin, B., Carless, S., & Wilson,I. (2013b). The Undergraduate Medical and Health Sciences Admissions Test: What is it measuring? Medical Teacher.

Griffin, B., Carless, S., & Wilson,I. (2013a). The effect of coahing on selection test performance. Medical Teacher, 35, 295-300.

Griffin, B., Harding, D. W., Wilson, I. G., & Yeomans, N. D. (2008). Does practice make perfect? The effect of coaching and retesting on selection tests used for admission to an Australian medical school. The Medical Journal of Australia, 189(5), 270-273

RESULTS

The data analyses aimed to answer a number of key questions:

  1. How widespread is commercial coaching by applicants to medical school?
  2. Slightly more of the high school students (56.2%) attended commercial coaching compared to those who were actually offered an interview (between 51 and 53% most years)
  3. However, rate of coaching attendance are significantly higher (up to 74%) among students from academically selective high school.
  4. Do coached students do better at selection interviews?
  5. Results since the beginning of this research project in 2006 have consistently shown that there is NO statistical difference in the overall interview (MMI) scores of those who have attended commercial coaching and those who have not.
  6. Furthermore, on some of the short MMI interview stations, coached students perform significantly worse than those who had not been coached.
  7. What is the UMAT Test?
  8. The UMAT is published and administered by the Australian Council for Educational Research (ACER). Refer to for more details.
  9. UMAT test results are incorporated into the selection decision for most undergraduate medical degrees in Australia and New Zealand, and for a number of other health-related degrees in some universities.
  10. The UMAT is designed to assess three types of cognitive ability: ‘Logical reasoning’, ‘non-verbal reasoning’ and ‘understanding people’. Our results showed that scores were related to other validated measures of verbal reasoning, numerical reasoning, abstract reasoning and emotional intelligence (‘understanding people’ section only).
  11. Does it matter which coaching company?
  12. There is NO difference in UMAT score according to which coaching company students attend. In other words, no company appears to be any better than the others.
  13. Do coached students do better at UMAT ‘logical reasoning’ test questions?
  14. Coaching has NO effect on performance in the logical reasoning component of the UMAT. In fact, as can be seen in the graph below, students at non-selective public schools performed significantly worse in this part of the UMAT if they had coaching.
  15. Note that these results were the same after controlling for ATAR. In other words, even when we consider students who did equally well in their Higher School Certificate, coaching still had no effect.
  1. Do coached students do better at the UMAT ‘understanding people’ test questions?
  2. Coaching has NO effect on performance in the ‘understanding people’ component of the UMAT. The small difference illustrated in the graph below were not statistically significant. The analysis also controlled for ATAR scores.
  1. Do coached students do better on UMAT ‘non-verbal reasoning’ test questions?
  2. After controlling for ATAR scores, there was a significant but small positive effect for coaching on this type of question, but ONLY for those who attended a government selective high school.
  1. Does coaching have a ‘long-term’ influence on performance in medical school?
  2. A major concern with coaching is that it trains participants to answer test questions (“test wiseness”) but cannot alter the underlying ability of the construct the test measures. For example, even if coaching were to result in a student getting a higher score on an intelligence test, the student’s actual level of intelligence is likely to remain unchanged. The risk for coached students is that they will not have the real ability to cope with a demanding university degree or manage in their later professional life (when no coaching is available).
  3. We compared the academic performance of medical students who had attended UMAT coaching for selection with the performance of those who had not attended coaching. As can be seen in the graph below, for the first cohort of students, those who had attended UMAT coaching had POORER end-of-year exam results for every year of their medical degree. Furthermore, 60% of those who failed their exams had attended coaching. This effect was reduced in later years, after the university altered their selection metrics in a way that significantly reduced the likelihood of an applicant who had benefited strongly from coaching being selected.

EXPLANATION

Coaching familiarises students with possible test questions and attempts to reach basic rules and methods for answering questions. We believe the results above may therefore be explained by:

  • Interviews: interviewers are trained to identify faked/rehearsed answers and questions are subtly altered every year so that a rehearsed answer will not be sufficiently specific to score well.
  • Logical reasoning test questions: require students to synthesize information and draw conclusions about facts. The UMAT test developers have endless possibilities for new questions and coached students risk misapplying rules.
  • Understanding people questions: there is plenty of evidence beyond our research supporting the fact that it is difficult to train people to identify emotions and interpret social information if they do not have the underlying ability.
  • Non-verbal Reasoning: it may be possible to learn “rules” for solving some of these items. However, our results indicate this is only likely under conditions of sustained practice. Anecdotal evidence suggests that when there is a large group of students in one high school all applying for medicine a “norm” develops that motivates individuals to engage in many hours of practice. We suspect it is the practice not the coaching itself that may produce an advantage, albeit a small one, in this section of the UMAT.

CAUTION

Students need to consider that:

  1. The small positive effect of coaching for one third of the UMAT test questions may be outweighed by the potential for disadvantage on other types of questions.
  2. The need for sustained practice of test items distracts them from potentially more productive HSC study.
  3. Universities use the UMAT in different ways and can adjust these to reduce any advantage of coaching, which may in turn maximise the potential disadvantage of coaching.
  4. Medicine can be a stressful career and those who artificially raise their test scores risk being a poor fit even if they do succeed at being selected.

BIO

Barbara GriffinB.Psychology (Hons), PhD

Barbara is an organizational psychologist and Associate Professor in the Department of Psychology at Macquarie University plus Adjunct Associate Professor in the School of Medicine at the University of Western Sydney. Apart from teaching into the Master of Organisational Psychology program, she runs a substantial research program investigating person-environment fit across the employee career cycle, including projects on selection, incivility, adaptation during career transition, and culture assessment. In each of these she has addressed issues of national and international importance, obtained significant competitive research funding, and published in leading academic journals. Barbara has led the team that developed, and continues to implement, the Multiple Mini Interview for the UWS School of Medicine, including an interviewer training package for over 150 interviewers each year.