Selection Effects on Applications and Admissions to Medical Education with Regular and Step-wise Admission Procedures

Christina Cliffordson[(]

University of Göteborg and Trollhättan/Uddevalla University, Sweden

Abstract The purpose of this study is to examine step-wise admission to two medical programs with regard to selection effects. By using data from a large-scale longitudinal project, the influence of background variables on regular and step-wise admission is estimated. Characteristics of both applicants and those admitted are investigated using descriptive statistics and the probabilities of admission and application are examined using logistic regression analysis. The results indicate that self-selections were generally greater for the step-wise groups compared to the regular groups. On the contrary, the selections resulting from the procedure were generally lower for the step-wise groups. However, the results vary between different step-wise designs, indicating self-selections to be dependent upon the type of instrument used in the first step, and selections resulting from the procedure to be dependent on the number of steps and/or, more probably, the type of instruments used and the number of those selected in each step.

Introduction

For a long time, the Swedish government has emphasised its desire to increase social, national and gender diversity in higher education. A well-established central unified admissions system aims to restrict the influence of characteristics other than school performance in determining admission. With the exception of Art programs, admission to most programs is based solely on grades from upper secondary school. Thus the system is clear and easy for the students to grasp, which is another aim. In the 1990s, the Swedish Scholastic Aptitude Test (SweSAT), a test battery similar to the SAT in the US, was introduced as a voluntary complement to grades, primarily to enable certain applicants lacking the necessary grades to gain access to higher education and thus, hopefully, reducing the influence of social background.

However, in spite of policy declarations and a number of specific measures, the 1990s have not seen any great changes to the prevailing, socially-skewed recruitment patterns in higher education. A number of high-status programs, such as medical education, still recruit predominately upper-middle class students (National Agency for Higher Education, 2002a). It is thus evident that different kinds of selection mechanisms still play a powerful role in determining applications and admissions to such programs.

So called step-wise admission routines, which include special aptitude tests, such as interviews and different written tasks, were introduced on a trial basis as an alternative to the regular admission (grades or SweSAT-scores) for professionally-oriented programs in which certain personal characteristics or special competencies are required, e.g. in medical, social work and teacher education. For such programs, it was argued, not least by faculty in medical schools, that high grades or test scores do not automatically provide the best prospective practitioners. An alternative set of admission tests might, it was further argued, reduce the influence of background variables and open up prestigious professional schools to less academically gifted students from different social and national backgrounds.

In recent years professional programs have obtained authorisation from the National Agency of Higher Education to use certain additional grounds for admission (SFS 1996:984). Students elect to be assessed either via the step-wise procedure, or in accordance with the regular system. So far, the number of institutions that have applied for authorisation is not large. The interest in using step-wise admission is evident in many medical faculties. Universities are, however, more reluctant to implement step-wise procedures for a number of reasons. The costs incurred in developing and implementing these tests are high. Additionally, the Agency has set relatively strict conditions for the granting of authorisation, such as the provision that no more than two thirds of the places available in any program may be filled using such a procedure (National Agency for Higher Education, 2002b). In addition, surprisingly little is known about how step-wise admission actually works. Compared to the rich body of research on how grades and SweSAT scores work as admission criteria with regard to background variables, so far, little information is available regarding step-wise admissions. Thus, it remains an open question as to the type of students institutions select via step-wise admissions, and if indeed “better” students from a wider social spectrum are admitted.

The purpose of this study is to examine step-wise admissions to medical programs at two universities with regard to self-selection and other kinds of selection effects due to background variables (age, gender, socio-economic, educational and national backgrounds, grades, SweSAT scores and number of SweSAT sessions) on the admission process. The questions addressed are: 1) What are the characteristics of the applicants and the admitted students in the step-wise and the regular groups? and 2) What is the probability that, with regard to background variables, students a) apply for medical programs through the step-wise and the regular admission groups respectively and b) are admitted?

Selection effects in regular admission procedures

Previous research has demonstrated that grades from upper secondary school and SweSAT scores provide good measures for predicting performance in higher education (Carroll, 1982; Gustafsson, 2000; Schmidt & Hunter, 1998).

For predicting study success, grades from lower school levels are often considered to be the most valid instruments for selection to further education (Gustafsson, 2000). Grades are based on teachers’ long-term assessments of school-based performance. However, since the introduction of grades as the sole basis for admission to higher education in Sweden, their usefulness as a selection instrument has been questioned (Henrysson & Wedman, 1995). With regard to group differences, about 10 percent of the differences in grades can be explained by socio-economic background in favour of the upper-middle class, a relation that seems to have remained quite stable during the 1990s. The relation between grades and national background has been shown to favour students born in Sweden. Furthermore, studies have shown that there is an apparent relation between gender and grades in favour of females (Gustafsson & Westerlund, 1994; Gustafsson, Andersson & Hansen, 2000). In the 1990s, the earlier norm-referenced grades were replaced by goal-related grades, a shift that increased the criticism of grades as selection instruments.

Relations similar to those obtained between grades and socio-economic and national backgrounds have also been shown for SweSAT scores. Results presented by Reuterberg (1998) show substantial socio-economic differences in favour of the upper-middle class. Furthermore, Reuterberg and Hansen (2001) showed that there are selection effects due to national background in favour of test-takers born in Sweden. Like the SAT (Wilder & Powell, 1989; Willingham & Cole, 1997; Rosser, 1989), the SweSAT (Reuterberg, 1997; Stage, 1992) shows gender differences in favour of male test-takers. Such group differences can be interpreted as a consequence of bias in the test. An alternative explanation is that self-selection effects cause differences in test scores. It has, however, been shown that self-selection effects cannot fully explain the group differences in scores (Mäkitalo & Reuterberg, 1996; Reuterberg, 1998; Reuterberg & Hanson, 2001). Furthermore, Cliffordson (2004) has shown that there are self-selection effects due to age and grades, in that students with higher grades tend to take the test at an earlier age.

Admission by use of step-wise procedures

As mentioned above, it is often argued that, when used alone, grades and SweSAT scores fail to provide satisfactory information about the applicant's motivation for professional studies or suitability for the profession. In addition, such an admission procedure tends to select overly homogeneous groups of students with regard to scholastic aptitude and social and national background (Areskog, Holmberg & Ulfberg, 1992; Hindbeck, Hagenfeldt & Åberg, 1992; Holmberg 1995; Powis, Neame, Bristow & Murphy, 1988). Thus, it is argued, admission to medical education ought to be designed to predict not only study-related skills but also study motivation and potential professional aptitude.

In a step-wise procedure, each selection instrument has its own properties as regards both selection effects and costs. The design of the procedure is therefore important. The first step is often based either on grades from upper secondary school and/or SweSAT scores. The second step often includes various written tests that assess motivation, personality traits and biographical information etc. The final step is normally an interview aimed primarily at assessing personal qualities important for professionals.

Written tests, aimed at measuring personal characteristics as indicators of suitability for the profession are seldom standardised for selection purposes and thus little is known about their properties as selection instruments. However, they have strong face validity and give powerful signals to faculty and prospective students that personal qualities are taken into account. In the step-wise procedures implemented at the two universities studied, written tests and interviews were used to assess personality traits such as creativity, cognitive flexibility, social competence, independence, patience, and, in addition, mental energy, empathy, devotion, and communication skills (Askling, Ottosson & Ulfberg, 1991; Holmberg, 1995). The selection of variables was based on assumptions about the influence of personal characteristics on success both academically and in the profession.

The interview is a controversial selection method. It has been the subject of a considerable amount of research, especially in employment settings, but also in selection for medical, nursing and social work degree programs (e.g., Campion, Palmer & Campion, 1997; Cliffordson, 2002; Edwards, Johnson & Molidor, 1990; Mårdberg, 1999). There are two broad types of selection interviews: structured and unstructured. All studies have supported the superiority and the use of structured interviews (Campion et al., 1997). Some structuring components however, appear to be more important than others. Training the interviewers is one method of ensuring that all of the components are correctly implemented and may lead to a reduction of the bias that can arise from a number of sources, including, although not exclusively, rater tendencies, stereotyping and interviewer characteristics and background. Edwards et al. (1990) have shown that reliability and validity improves as structure is added, but that semi-structured interviews were almost equally reliable and valid if they met certain criteria such as standardised questions, anchored rating scales and the use of board interviews.

The admission procedure at the two universities studied

The step-wise admission to the two medical programs (A and B) have similar, although not identical, designs (Table 1).

The procedures for university A. From the Autumn semester 1993 to the Spring semester 1998 (design A:I), about 50% of the students were admitted by a step-wise and 50% by a regular procedure. In the first step, all applicants in the step-wise group were ranked according to their grades and/or SweSAT scores. Six times as many as the number finally accepted were selected for the next step; 60 % from the group with top grades and 40 % from those with highest SweSAT scores. Step 2 included a written motivation for the applicant’s choice of professional career, a biographical description and an essay. About 50 % were selected for the next step. Finally, step 3 consisted of a semi-structured interview conducted by two interviewers, one male and one female. One came from the faculty (doctor/teacher) and the other was a lay-person with experience in assessing personal qualities. Based on the interview, about 30% were finally admitted.

From the Autumn semester 1998 the design was modified (design A:II). The second step, which included the written tasks, was excluded. The selection in step 1 was based solely on the SweSAT scores, and twice as many as the number finally accepted were selected for the next step, which included the interview, that was conducted as described above.

The procedure for university B. At university B, the design remained unchanged throughout the period studied, split in a 50/50 proportion between the step-wise and the regular procedures. In the first step, the applicants were ranked according to their SweSAT scores, with additional points given as credits for previous higher education studies. Three applicants for every available place were selected to participate in the next step. Step 2 included a written motivation for choice of professional career, a personal biography and an essay. About 60% of these applicants were selected for the next step. In step 3 the applicants were interviewed twice, one interview being conducted by a doctor/teacher and the other by a psychologist. A final ranking of the applicants was based on the semi-structured interview, and about 50% of them were selected for admission.

(Insert Table 1 about here)

This set of designs makes it reasonable to present the following hypothesis:

·  Self-selection effects are mainly dependent on the first step and the applicants’ knowledge about the procedure generally. Accordingly, as females generally have higher grades, design A:I would favour females to a greater extent than design B. As the SweSAT scores show gender differences in favour of male test-takers, design A:II would favour males to a greater extent compared to design B.

·  The use of step-wise admissions is based on the assumption that motivation and suitability for the profession are only marginally related to grades and SweSAT scores. Thus, the selection effects due to grades and SweSAT scores should be relatively moderate for the step-wise groups compared to the regular groups.

·  Other kinds of selection effects are dependent on the overall procedure and the number of applicants selected in each step. Thus, with fewer steps and with a lower number of selected applicants in each step, the final results of the selection procedure should be more dependent upon the self-selection effects generated by the content of the first step. That is, for design A:II the self-selection effects would be relatively more important for the final results compared to design A:I (more selected and more steps).

Method

Participants and procedures

The study is based on data extracted from a large scale longitudinal project, in which information has been collected from different official registers covering every individual in Sweden born during the years 1972-1984 (about 1.4 million individuals). For this study, individuals born in the years 1974-1982 (922 169 individuals) were examined. Among them a group of 6 586 applicants for medical programs at the two universities included in the current study during the years 1993 (autumn semester) to 2000 (autumn semester) could be identified. This group formed part of a larger one of 7 648 individuals who applied for medical programs at all six of Sweden’s medical schools. As individuals generally apply for admittance at more than one institution on more than one occasion, the difference between applicants to the two universities and the total number of applicants to all six universities during the period is relatively small.