Paper presented at the European Conference on Educational Research, University of Lisbon, 11-14 September 2002

Guarding and Preserving the autonomy of the individual professional, Controlling for potentially dysfunctional consequences of performance

measurement in organisations in higher education

By Christine Teelken & Geert Braam

University of Nijmegen

Netherlands

Abstract

As a consequence of the managerialism, organisations in higher education increasingly have to function business-like in competitive settings. This implies that their needs in terms of management and, therefore, measurement systems have changed. However empirical results (Marheim Larsen & Gornitzka, 1995; Rasmussen, 1995;Trow, 1994; Taylor, 2001; Teelken, 1999, 2001) show that the focus on performance management may have unintended and potentially dysfunctional consequences. The question is whether higher education organisations are prepared for the consequences of this increased performance management and measurement.

This paper examines to what extent management systems for higher education have deficiencies with respect to their performance measurement systems. On the basis of Smith (1995), eight potentially dysfunctional consequences are distinguished. Combining new public management literature and strategic management control literature, we suggest the balanced scorecard as strategic management system to overcome these inadequacies. In the context of the new managerialism it offers a suitable framework to implement performance management and measurement in higher education institutions. In a systematic and structured way and on a continuous basis it may effectively and efficiently embed performance measurement in the wider organisational context. However, the number of empirical studies of balanced scorecard's impact on performance is limited, which underlines the necessity for future empirical research in this area.

1. Introduction

Higher education systems in Western Europe have undergone major processes of change. Governments of many countries have sought to increase the throughput of students by creating more ‘marketlike’ educational situations. The community wants ‘value for money’ (Pounder, 1997). Simkins (2000) speaks of a change of focus, were the general university output is more important than other measurements for success. Organisations in higher education increasingly have to function business-like in competitive settings. Due to the influences of new public management (NPM) and managerialism universities are increasingly held accountable for the quality of their performance (Deem, 1998; Roberts, 2001) and consequently have to control and improve the quality of their output (Deem, 1998, 2001; Haley, 1995; Politt & Boeckaert, 2000). Various authors (Hood, 1995; Maassen, 2000; Sizer & Cannon, 1999; Barry et al., 2001; Parker & Jary, 1995) agree on the fact that there is a greater need for measurement of performances specifically in higher education organisations. Parker and Jary (1995) argue in their article ‘The McUniversity’ that as a consequence of the managerialism employees use their autonomous rights (of ‘donnish dominion’) by introducing and refining control mechanisms. Outmoded practices or poor performances can thereby be made visible, discredited and replaced by more autonomous forms of work. At the same time increased bureaucracy becomes necessary (p. 325).

However empirical results (Marheim Larsen & Gornitzka, 1995; Rasmussen, 1995; Trow, 1994; Taylor, 2001; Teelken, 1999, 2001) and the conceptual study of Smith (1995) show that the resulting focus on performance management may have unintended and potentially dysfunctional consequences. For instance, as professionals engage in behaviour directed towards increasing (or maintaining) their own autonomy, while management implements control systems are designed to control that behaviour, conflicts may arise that lead individual professionals to behave suboptimal and unproductive. The question is whether higher education organisations are prepared for the consequences of increased performance management and measurement.

This paper examines to what extent management systems for higher education have deficiencies with respect to their performance measurement systems. Combining new public management literature and strategic management control literature, we suggest the balanced scorecard as a strategic management system to overcome these inadequacies. In the context of the new managerialism it offers a suitable framework for implementing performance management and measurement in organisations in higher education in a systematic and structured way.

The remainder of the paper is organised in three sections. Section 2 outlines the potentially dysfunctional consequences that increased performance measurement may have. Section 3 develops a comprehensive strategic control framework for dealing with these inadequacies in a structured and systematic way. The last section provides concluding remarks and suggestion for further research.

2.Potentially dysfunctional consequences

Defining performance measurement is no easy task as it is a very broad concept, which can be aimed at the preliminary setting of goals or for evaluation and it can be used for internal or external purposes. Examples of performance measurement are targets, standards, waiting lists, exam results, graduation rates and university research ratings (Politt & Bouckaert, 2000). It can have many different appearances and can direct itself at the level of the individual employee, the level of the department or faculty or for organisations as a whole. In a context of increased managerialism performance measurement in higher education organisations has changed in a threefold way: it has become more intensive, more external oriented and more elaborate, involving more complex trajectories.

The most important danger of increased performance measurement is an overconcentration on output (counting the number of cases) at the expense of the outcomes (satisfied clients), which may lead to less attention for risky, unattractive but nevertheless important functions. External legitimacy and quality measurement are becoming goals in itself. See for example Teelken (1999, 2001) who argues that the quantitative performance indicators of Dutch secondary schools emphasise and may even increase (social) differences between schools. The most important indicators for the quality of an educational organisation can hardly be quantified, and are therefore often not included in the information provided by the public organisation. Trow (1994) points at the great danger of external quality assessment by quantitative measurements as part of the managerial ethos. He warns for a paradoxical result that assessment of quality of university work leads to its decline, because so much energy is spent on bureaucratic reports and because the quantitative nature of the results has a simplifying tendency. His research suggests that departments and individuals shape their activities to what counts in the assessment. This can lead to the impoverishment of university life, which is always more complex and varied than quantitative methods can assess.

A second danger is that one can only measure what can be measured. Measurement activities often lack a logical basis and are constructed for reasons of pragmatism or legitimacy. As direct, actual activities and individual services are easier to measure than indirect, abstract activities and collective services, unimportant features (e.g. the number of copy cards used by employees) may therefore be easier to measure than important ones (the actual quality of education and research).

A third danger is that the output of some employees is much easier to measure than that of others. Research in the National Health Service of the UK shows a clear pattern: the more powerful groups of stakeholders have been able to postpone or deflect the tide of measurement more effectively than other groups. Activities of nurses and receptionists in health services have been measured far more intensively than the quality of clinical decision-making by doctors. Indeed, performance measurement of teachers, police, social workers, social security clerks is much more widespread than for MPs or ministers (Pollitt & Boeckaert, 2000).

Taylor (2001) analyses the efficacy of performance indicators, in raising the effort level of university academics through a raise in pressure and in turn minimising inefficiency within the institutions. She bases herself on the earlier research of Cave & Hanney (1990) and Goedgebuure et al. (1990) when she explains that although performance indicators are intended to trigger an improvement in academic effort and achievement, they could also lead to undesirable consequences such as behaviour which may lead to good scores on the indicator but which is dysfunctional to the institution (Cave et al., 1991, in: Taylor, 2001). The findings are consistent with Smith (1995), who shows that the implementation of a system of performance indicators in most public sector organisations is likely to have both intended, functional and unintended, dysfunctional consequences. Smith (1995) identifies eight possible consequences of the publication of performance data that are not necessarily intended, and which are likely to be dysfunctional:

  1. Tunnel vision;
  2. Suboptimization;
  3. Myopia;
  4. Measure fixation;
  5. Misinterpretation;
  6. Misrepresentation;
  7. Gaming; and
  8. Ossification.

In the following section each of these eight consequences is applied to the specific situation in higher education, and suggestions for limiting their detrimental impact are made.

  1. Tunnel vision refers to emphasis on quantified effects at the expense of unquantifiable aspects of performance.

Quantification of objectives is specifically difficult in higher education institutions, because its goals are so diverse and complex. For example, the quality of research is determined by the number of publications and by the status of the journal these publications appear in. Such a scheme makes it attractive for researchers to rehash data and conclusions from previous research in order to create several similar publications, rather than embarking on new, innovative research, for which possibilities for publication are uncertain.

To avoid too much quantification, Smith recommends to decrease the focus on performance measurement, and suggests nurturing a sense of shared values instead. Other recommendations involve scanning the environment for unanticipated externalities and embedding the performance measurement system in a broader environmental monitoring system.

  1. Sub-optimisation is the pursuit of a narrow local objectives, at the expense of the objectives of the organisation as a whole.

This danger is particularly present in the case of strictly developed managerial discretion. It can be very difficult to match the targets of the devolved units with the top level objectives for the organisation as a whole. If rewards schemes are directed at the individual and output is the result of team efforts (which is often the case in the public sector) it may be difficult to motivate certain members of staff and free-riding behaviour can arise. For many public sector activities, the rigid model of control implicit in market or bureaucratic designs will often be inadequate.

The problem of sub-optimisation is not necessarily soluble. Instead, there is a trade-off between the beneficial incentive effects of a formal control mechanism and the dysfunctional consequences of sub-optimisation.

  1. Myopia (short-sightedness) can be induced by performance indicators stimulating the pursuit of short-term targets at the expense of legitimate long-term objectives.

Performance indicators are often based on a snapshot of organisational activities and offer an imperfect reflection of the efficacy of current management. Current performance indicators may say more about the performance of previous management, than about the long-term objectives of an organisation. Myopia can be induced by short-term career perspectives of many workers, specifically in the case of fixed term contracts.

The increasing focus on predetermined output measures determines to a certain extent the danger for myopia. A solution suggested by Smith is trying to measure processes instead of outcomes.

  1. Measure fixation means emphasis on measures of success rather than the underlying objective.

Determining clear objectives in public organisation is difficult and this is probably even more so in higher education organisations. Even if such objectives can be identified, performance measures do not normally capture all dimensions of the associated objective. In such cases, managers may be encouraged to enhance the measure instead of the actual objectives of the organisation. Examples of such interference in higher education are lowering examination norms in order to obtain higher graduation rates.

Instead, measure fixation can be reduced by measuring clients’ satisfaction. The evaluation forms which students have to fill in at the end of their courses is an example of such investigations, which is probably more informative with respect to the quality and content of a course than the actual exam results.

  1. Misinterpretation means that although the available data are a perfect representation of reality, the interpretation of the signals emerging from the data is often extremely complex.

Bounded rationality might cause systematically misinterpretation by the controller of the data and consequently, wrong policy signals can be sent to the provider of the data. This unintended consequence is quite likely in the UK education system. All state schools have to publish the results of their examination results. However, the interpretation of raw performance data is complex and can therefore be misleading. Similar developments are going on in the Netherlands, where secondary schools have to publish their exam scores and other figures in a choice guide and on the internet. Higher education organisations results are published once a year by means of the Elsevier journal based on surveys among students and professors.

  1. Misrepresentation implies the deliberate manipulation of data so that reported behaviour differs from actual behaviour.

Public organisation employees will be tempted to manipulate data in order to show the organisational performance in a more advantageous light. The scope for misrepresentation can be particularly broad in this context because of the complexity and the dual character of higher education organisations with their ambiguous goals (Bleiklie, 1998). Moreover, the data used are under the direct control of the accountable staff. Misrepresentation is potentially dysfunctional because it can be misleading and result in misallocation and inequitable treatment of staff and clients. A possible solution may be giving staff ‘ownership’ of the data by integrating the performance data collection into the administrative information systems which staff use to run their own units and using strict guidelines.

  1. Gaming can be seen as the deliberate manipulation of behaviour to secure strategic advantage. The sender causes desired messages to be sent.

In contrast, misrepresentation is a strategy intended by the sender to manipulate the recipient by affecting the set of data available to the recipient. There are several ways to reduce the risk of gaming: by using a range of performance indicators and by maintaining some kind of uncertainty about which indicator will be used to influence rewards. Gaming is specifically likely in the case of longer term career perspectives for managers, as opposed to myopia, which is more likely in short term perspectives.

An example of gaming in higher education is lowering your exam standards in order to receive more favourable evaluations from the students.

  1. Ossification refers to organisational paralysis brought about by an excessively rigid system of performance evaluation.

It seems that bureaucratic performance measurement schemes can inhibit innovation and therefore lead to an inability to respond to new circumstances. New threats and opportunities may be ignored by managers, focussing instead on performance measurements and targets.

Ossification can be minimised by ensuring that managers are rewarded for behaviour that anticipates new challenges, even if these activities are not consistent with current performance indicators.

Analysis of consequences

The eight potentially dysfunctional consequences of increased performance management described can be classified in four groups. These first three consequences - tunnel vision, suboptimization and myopia - can be derived from a lack of congruence arising from divergence between organisational objectives and measurement schemes. The fourth and fifth consequences - measure fixation and misinterpretation - can both be explained by the inability to interpret and measure complex phenomena with validity (and reliability). The question is whether these measurements fulfil their aim. The sixth and seventh consequences - misrepresentation and gaming - refer to the inability to process data correctly. While misrepresentation leads to distortions in reported behaviour, gaming is the manipulation of actual behaviour; often in order to keep expectations and targets for the future low. The sixth and seventh consequence are closely related to the fifth consequence. To send a message which is ‘credible’ to the recipient despite manipulation means that it is not easily doubted (Birnberg et al.,1983). The last dysfunctional consequence – ossification - can be related to the relatively inflexible bureaucratic structure of organisations in higher education. Although Smith (1995) suggestions are helpful, they are presented in an adhoc fashion and lack consistency. The next section deals with developing a framework that overcomes these deficiencies.

3.Framework to improve performance measurement and management

To overcome the potentially dysfunctional consequences described in the previous section, this section provides suggestions for minimising these inadequacies. Second, it describes a strategic control framework for implementing these suggestions in a coherent and systematic way to ensure that they adapt successfully to changing requirements.

The first three consequences - tunnel vision, suboptimization and myopia - imply that functional departments and individuals have to be aligned. For organisational performance to be more than the sum of its parts, individual interests must be linked and integrated.

The second group of undesirable consequences - measure fixation and misinterpretation - refers to the organisational inability to translate the strategic objectives into operational terms. Performance indicators do not represent strategic goals and critical success factors directly. Preparing them requires time and effort which could be better used elsewhere. Moreover, performance measurement is still largely focussed on co-ordination and control (Atkinson & McCrindel, 1997). On the other hand, lack of useful performance information hampers the management of performance in a number of areas.

The third group - misrepresentation and gaming - stresses the importance that all employees have to understand the strategic goals of the organisation to execute it, i.e. their involvement has to conduct their actions in ways that contribute to the success of the organisation. However, the processes that performance management induces, directly and indirectly influence and change individual behaviour and corporate culture. Performance measurement may be very confronting because explicating the results of a strategy and actions implies that, on a deeper organisational level, the underlying norms and values are made clear and have to be complied with. Maybe in the previous culture it was not done to criticise somebody on his or her performance. A successful performance management system implementation therefore implies cultural change. Generally, it will be important to understand the interactions between the various critical variables as the presence of counteracting and reinforcing variables implies the need to provide synergies or complimentaries to enhance organisational performance. Within the context of strategic performance measurement studies mainly report linkages between managerial performance practices, organisation’s strategy and organisation’s competitive environment, as they frequently counteract or reinforce each other (e.g., Govindarajan & Gupta, 1985; Simons, 1987 & 1990; Govindarajan, 1988; Dent, 1990; Shrank & Govindarajan, 1994; Langfield-Smith, 1997; Ittner & Larcker, 1997; Chapman, 1997; Bouwens & Abernethy, 2000). Literature that reports linkages with organisational and cultural change is limited yet.