The Effects of Individual, Institutional, and Market Factors on
Business School Faculty Beliefs About Grades
Timothy Cairney
Assistant Professor, School of Accountancy
Georgia Southern University
Christopher Hodgdon*
Assistant Professor, School of Business
University of Vermont
Sewon O
Assistant Professor, School of Business
Texas Southern University
ABSTRACT
This study examines gatekeeper and frame-of-reference (norm versus criterion) grading beliefs of U.S. business school faculty using a web-based survey. We report that faculty in the business courses with a more theoretical pedagogy have a stronger gatekeeper and greater norm-referenced beliefs than do the faculty of the courses with a more skills-oriented pedagogy. We find that the individual attributes examined indicate that teaching load is associated with greater criterion-referenced grading beliefs. We also find that male faculty have greater gatekeeper and norm-referenced grading beliefs, and that faculty who experience more pressure from departmental grading policies revert to more criterion-referenced beliefs. Furthermore, we find that faculty in schools that service more national/international firms tend to have greater norm-referenced beliefs, and faculty whose graduates tend to go to small employers (i.e., regional and local firms) exhibit greater criterion-referenced beliefs, suggesting that larger firms would like colleges to rank students to allow for a more informed choice for their large employment needs.
*Corresponding author: School of Business Administration, The University of Vermont, Burlington, VT 05405, (802) 656-5774, (802) 656-8279 (fax),
The authors thank John Dyer for research assistance, and acknowledge the helpful comments from the 2006 AAA meeting and the Georgia Southern University College of Business symposium series. Summaries of survey responses are available from the authors.
11
The Effects of Individual, Institutional, and Market Factors on
Business School Faculty Beliefs About Grades
INTRODUCTION
This study examines business faculty beliefs about the purpose of grades, and considers the question of why grading beliefs differ among business faculty. To examine this question, we first measure the grading beliefs of faculty by assessing two dimensions: (1) what reference group faculty use to determine grades (frame-of-reference beliefs), and (2) what grading standards the faculty use (gatekeeper beliefs). We then propose and test reasons why these beliefs differ.
In related research, Barnes et al. (2001) survey a broad, multi-discipline sample of university faculty about their grading beliefs and find that faculty gatekeeper grading beliefs differ systematically among disciplines but frame-of-reference beliefs do not. For instance, they report that physics faculty have more of a gatekeeper (defined in the next section) belief than do business faculty. Barnes et al. (2001), however, did not consider the separate and distinct disciplines within schools of business. Because of the wide range of subjects offered, we believe that business faculty do not readily form a homogenous group. Thus, our study investigates the grading beliefs of faculty from different business disciplines.
Our hypothesis is that grading beliefs are influenced by the environment of the faculty member. Our framework for the environment of grades comes from Lunneberg (1978), who notes that inconsistent and unreliable grades reduce their worth in at least three important ways. First, faculty use grades to provide students with feedback, and prior studies suggest that individual faculty attributes are related to this feedback. Thus, we hypothesize that individual faculty attributes (e.g. gender) will impact grading beliefs. Second, universities use grades for assessing teaching and program effectiveness so we hypothesize that institutional differences (e.g. departmental grading policies) will impact faculty-grading beliefs. Third, external users examine grades to identify capable students so we hypothesize that differences in the “marketplace” for a school’s graduates influence business faculty grading beliefs.
To investigate these hypotheses, we asked a nationwide sample of business school faculty to complete a web-based survey. The survey results suggest that different disciplines within the business school exhibit systematically different beliefs about what their grades communicate. Further, our results show that individual attributes and institutional influences are more (less) able to explain variations in frame-of-reference (gatekeeper) beliefs. On the other hand, the “marketplace” factors have greater (less) influence on gatekeeper (frame-of-reference) beliefs. Overall, we conclude that the diverse objectives sought by the various business disciplines are reflected in their different grading beliefs and, as such, a proper evaluation of student performance requires consideration of grades from all courses.
PRIOR RESEARCH AND VARIABLE DESCRIPTIONS
Barnes et al. (2001) propose that between-discipline differences in grades can be explained by the cross-discipline differences in the beliefs faculty hold about the purpose of grades. Their study, however, combines all business faculty into a single group and does not recognize that managing human resources, for example, is a more subjective discipline than finance or accounting. The management grades, therefore, are more likely to reflect the lower specificity of course content compared with the highly structured content of finance or accounting.
Like Barnes et al. (2001), we use Biglin (1973) to describe disciplines as either “hard” or “soft,” (hard is more related to physical nature), as “pure” or “applied” (pure is more theory driven) and as “life” or “non-life” (life deals more with live entities). Barnes et al. (2001) relate course objectives to these classifications and describe courses whose content devolves from theoretical constructs (e.g. physics and mathematics) as the hard, pure, non-life disciplines, and the courses whose contents pursue the development of situational skills (e.g. fine arts and education) as the soft, applied, life disciplines. The Barnes et al. (2001) study reports differences in grading beliefs among the Biglin (1973) classifications.
Within business school disciplines, economics and finance are more theory-driven while management and marketing relate more to the development of the skills that can be applied in various situations. Casual conversations with colleagues from these disciplines support this view of business courses, although most consider the courses to be more complex than just this two-factor classification. Although many may disagree with the simplistic Biglin (1973) classifications, we find consistent approaches in schools of business and therefore use the Biglin (1973) scheme to provide a perspective on the differences among business disciplines.[1]
Grading Beliefs
We measure grading beliefs along two dimensions: the frame-of-reference and the gatekeeper dimensions (Barnes, 1997). The frame-of-reference labels one extreme as “norm-referenced” and the other extreme as “criterion-referenced.” Norm-referenced graders believe that a student’s grade has meaning only when compared to the performance of other students. Thus, faculty who are more norm-referenced believe that the purpose of grades is to evaluate each student’s performance relative to other students in the same class. Criterion-referenced graders, on the other hand, believe that grades are most meaningful when used to evaluate whether the student has met an external standard for performance (i.e., certain set criteria). Faculty who are more norm-referenced may “curve” grades and faculty who are more criterion-referenced may assign an “A” to all students, provided they have all met the established criteria (Barnes, 1997; Geisinger and Rabinowitz, 1982).
The second dimension of faculty beliefs about grades is the gatekeeper belief, the extent to which the professor acts as the profession’s advocate in the educational process. Strong gatekeeper beliefs among faculty indicate that a significant component of the educator’s job is to serve the business community by separating out the most capable students. On the other hand, a weak gatekeeper belief emphasizes student improvement and is consistent with advocating students’ capabilities. As students develop professional skills, the grades reflect their improvement and indicate to users whether their efforts were successful.
Barnes et al. (2001) find support for differences in grading beliefs among academic disciplines. Their university-wide results suggest that those who teach mainly facts (e.g., the hard, non-life disciplines) have higher gatekeeper beliefs compared to those who emphasize personal development (e.g., the soft, non-life disciplines). They do not find statistical differences between disciplines for the frame-of-reference measures (norm vs. criterion reference). We break out the individual business disciplines and expect differences among business disciplines to exist in a predictable manner.
Influences on Grading Beliefs
In this section, we describe the individual attributes, institutional factors, and external (market) factors that are hypothesized to influence grading beliefs. We examine the literature for variables that reflect Lunneberg’s (1978) various stakeholders in university grades (i.e., faculty, institutions, and external users).
We expect that faculty attributes will influence grading beliefs beyond Biglin’s (1973) classifications of faculty members’ chosen disciplines. Barnes et al. (2001) report that grading beliefs are associated with how the faculty view their role as teachers (e.g., as imbuing students with a love of learning, or as a teacher of facts). Thus, we include a test of whether the role (ROLE) influences grading beliefs. Related to this, we ask whether teaching load (LOAD) influences grading beliefs because teaching load can differ according to rank. Gender may influence whether a faculty member is initially employed at a research or teaching institution (Collins et al., 2000), and we therefore test for gender (GNDR) effects.
University, business school, and departmental factors constitute institutional influences. Thus, we examine whether departmental grading policies (POLY) influence grading beliefs, reflecting Addy and Herring’s (1996) report that such policies contribute to grade inflation.
External (i.e., “market”) influences may also impact grading beliefs from two directions. First, Albrecht and Sack (2000) suggest that accounting faculty should be acquainted with the primary employers of their students, so we ask whether the type of employer (EMPR) influences grading beliefs. For instance, it may be that smaller accounting firms do not have the resources to provide additional training to new hires, and as a result may provide feedback to faculty who would then be encouraged to adopt a greater gatekeeper belief. A second direction of influence is how faculty view the competitive environment of their school. The existence of competition among schools can influence how graduates are compared to graduates from other schools (Kemelgor et al., 2000; Bailey and Dangerfield, 2000). Consequently, we investigate the influence on grading beliefs of faculties’ perceptions of how grades from their courses compare with grades from other universities. If a faculty member believes “top-tier” schools have tougher grading standards than “low-tier” schools, this would be consistent with a hierarchy view (HEIR) of schools, in contrast to the view that similar courses across schools have equal standards.
SURVEY AND SAMPLE
Our survey questions are based on the Faculty Beliefs about Grades Inventory (FBGI), developed and verified by Barnes (1997) and published in Lester and Bishop (2000). The FBGI consists of 24 Likert-scaled questions, which Barnes tested on a broad base of university faculty.[2] We also include additional instructor, institutional, and “market” oriented questions that seek to identify influences on grading beliefs. The survey was pre-tested by 26 undergraduate business faculty at a southeastern public university. After pre-testing, a cover letter and link to the web-based survey was emailed to a representative sample of university business school faculty across the nation in May of 2005. The month of May is the last month of many semester-based institutions so it is assumed that faculty are most aware of their grading and evaluation beliefs at this time. The link to the survey was left open for 29 days.
The sample selection process is as follows. The American Accounting Association includes seven geographic sections, from each of which one state was selected as representative of the section. Next, from each state we identified AACSB-member universities that were judged to represent a broad distribution of university sizes (major state institutions to smaller universities) and types (public and private). Then, the names and email addresses of faculty from all business departments were copied from the business school web pages to an email address list. In total, 2,876 faculty email accounts were selected.
The first request went out on May 2, 2005, and the second request was sent to the whole sample on May 23, 2005. The second requests went to the whole sample because anonymity was guaranteed to respondents. From the first email request, 77 were returned with faulty email addresses, indicating a base sample of 2,799 faculty from 36 universities. The first emailing resulted in 200 completed surveys (a sample response rate of 7%). The second request provided an additional 75 responses for a total response rate of 10%. There are no significant differences between early and late responses. In untabulated results, we evaluated the response rates by Carnegie classification universities. The more teaching-oriented universities (Carnegie classifications 2 and 3) had total response rates of 15% and 20% respectively. The more research-oriented Carnegie classification universities (classifications 4 and 5) had weaker response rates. Although the combined overall response rate is low, we believe our results may be more indicative of the teaching-oriented universities.
We replicated Barnes (1997) factor analysis on the responses and identified two factors which are equivalent to Barnes’ Frame-of-Reference and Gatekeeper attributes.[3] The two factors explain 40% of the variance (Barnes, 1997, was 39%) and have reliability alphas of 0.78 and 0.86 for the gatekeeper and frame-of-reference factors, respectively (Barnes, 1997, reports alphas of 0.83 and 0.86 and Barnes et al., 2001, report alphas of 0.73 and 0.83). Hair et al. (1997) suggests that alphas greater than 0.80 indicate reliable scales, so we believe the instrument to be a valid measure for our sample.
GRADING BELIEFS OF PARTICIPANTS
Table 1 presents Gatekeeper and Frame-of-reference scores, by business disciplines. As previously noted, Barnes et al. (2001) classifies business faculty into a single group and compares that group to other disciplines, whereas this study examines whether faculty beliefs about grades differ among disciplines within business schools. This more detailed analysis proves to be informative.
(insert Table 1 here)
The main observation is that the mean GATE and the mean FOR scores are different among the disciplines. The overall mean GATE is 4.51 and mean FOR is 3.93, consistent with Barnes et al. (2001). However, an anova test (not shown) indicates that the mean GATE and the mean FOR scores are different among the disciplines (F-statistic is 4.08 for the GATE test and is 3.96 for the FOR test). Further, a ranking of the disciplines by GATE scores is consistent with the conjecture that the “theoretical” disciplines exhibit stronger gatekeeper beliefs (Barnes et al., 2001). For instance, economics and finance have higher GATE scores while and management and marketing have lower GATE scores. Waller-Duncan multiple range tests (not shown) indicate that economics (high) and marketing (low) are separated from the rest of the disciplines. It is also interesting to see that the GATE scores have a narrower standard deviation than the FOR scores. The standard deviation of GATE also indicates that the marketing discipline has the most variability of gatekeeper opinions among Business disciplines, and economics has the most variability of FOR scores among the disciplines.