1

The Higher Education Academy Annual Conference July 2006 – Session Paper

MEASURING IMPACT OF A NATIONAL AWARD SCHEME: THE CASE OF THE NATIONAL TEACHING FELLOWSHIP SCHEME

by

Anthony Rosie, Margaret Johnson and Philip Frame

INTRODUCTION

Recognition of the achievements of individual scholars in learning and teaching is the focus of a number of national award schemes. Probably the most established are the Carnegie Awards in the USA and the 'Triple M' awards in Canada. Other national awards include Australia (from 2006), England and Northern Ireland (from 2000), New Zealand and Sweden. While there are differences between these schemes in individual countries, there is also considerable overlap, e.g. criteria for assessment that consider impact on learners, colleagues, reflective practice and evidence of commitment to scholarly inquiry in learning and teaching. See Appendix 1 for outline of different national schemes. Perhaps the most telling and important evidence of overlap is the creation of a multinational scholars forum involving national award winners. To date this group has met twice: (2003 (USA) and 2005 (Canada) with a third meeting linked to the HE Academy annual conference 2006 in the UK. The dialogue amongst these national scholars/fellows in respect on learning and teaching has involved a consideration of their impact within nations, the nature of quality enhancement, and the first steps towards an international approach to learning and teaching.[1] The three of us as NTFs from 2001 have worked together on exploring the experience of being a national teaching fellow and our output is set out in Appendix 2.

THE IMPACT OF NATIONAL AWARD SCHEMES

All the national schemes for recognition of individual staff sit alongside other forms of funding awards and rewards for learning and teaching within particular countries. These might include institutional awards for learning and teaching, subject and discipline-based funding, funding council or recognised national body funding. In different national contexts there may be greater or less emphasis on charities and voluntary organisations as providing a funding stream for learning and teaching. Employer organisations have played important roles in different countries in funding aspects of learning and teaching practice. It is salutary to remember that whilst only 10% of the world's population lives in either the USA, North America or Europe, these are the very countries with the highest concentration of funding for individual awards. This does raise the question as to how the work of inspirational teachers outside the wealthiest countries can be recognised. It also raises the question as to how award holders and many others can contribute positively to the educational experience for the greatest part of the world's population. By and large governments in less developed countries (LDCs) cannot afford to educate to degree level the great number of their young people who are eligible for higher education. This raises at least the following three questions.

1. What are the conditions under which successful impact can be achieved by the work of an individual award winner both within their institution and national HE sector as a whole?

2. How transferable are either these conditions or these impacts within broadly similar national settings?

3. Are there any tentative suggestions that we can make for the wider transferability of conditions for success? This paper develops a model which contributes to answering question one, and provides an outline comment on what might be required to answer question two. Question three, though, is beyond the scope of this paper but can , we would suggest, be taken up within the multischolars forum referred to above.

THE NATIONAL TEACHING FELLOWSHIP SCHEME (NTFS) AND EDUCATIONAL CHANGE

The details provided in the handout, and referred to above, show the pattern of award holding between categories and the changes to the scheme since its inception in 2000. The scheme has formed the individual strand for rewarding outstanding teachers, alongside a formula funding approach for provision to institutions (institutional strand) as well as funding for disciplines via the Subject Centres (subject strand). The presence of guaranteed funding at the level of the institution, subject discipline, as well as opportunities for individuals means that there is a mix of national and institutional drivers for exploring and measuring impact.

Funding Council strategies[2] show an interesting shift in strategic approach. HEFCE (2003) outlines and emphasises quality enhancement, which is also very much the focus of the Higher Education Academy. Quality enhancement is seen as distinct from quality assurance. This debate was summarised in Middlehurst (1997) with quality enhancement being identified as a key means of bringing about transformational change to existing practice, that is, innovation is tackled in new ways, including explicit risk taking. In contrast, quality assurance by and large establishes that what is claimed is correct, that what is provided is fit for purpose and that it achieves efficiency and effectiveness. National teaching fellowship holders are required to demonstrate excellence in quality enhancement, normally the preserve of educational development, and are therefore required to be concerned with productive and meaningful change. The careers of national teaching fellows also shows that most have substantial and respected involvement in quality assurance within their institutions and nationally. In fact there appears to be no artificial divide between assurance and enhancement in these careers, and, following Middlehurst's pyramid, we can see the upper levels of quality enhancement subsuming the quality assurance base. Quality assurance alone does not develop or deliver innovation, or exciting and rewarding programmes of study. Similarly quality enhancement activity without careful consideration of assurance issues, including efficiency and effectiveness, is of little value. HEFCE (2006) brings together both forms of quality monitoring with an emphasis on continuous improvement. This shift of emphasis is developed in the approach we adopt in this paper, because we find a tension here and it is this tension we which to explore. On the one hand, innovation is frequently discontinuous at first and rests on bringing new elements together in unexpected ways. On the other hand, its harmonisation and take up, through embedded practice, achieves a line of continuity and improvement which stands in a direct contrast to the discontinuous leaps that innovation so often requires: hence the tension and hence our interest.

APPROACHES TO EDUCATIONAL CHANGE AND MEASURING IMPACT

Bate (1998) draws a distinction between first order and second order change which is akin to Middlehurst’s distinction between quality assurance and quality enhancement. First order change refers to cultural maintenance and development (essentially "trying to do what you do best, better - and more often" (1998: 35). Second order change refers to transformational change or qualitative discontinuous step changes of the culture itself (Rosie et al, 2005).[3] Most NTFs are involved in both continuous, incremental change and in discontinuous transformational change. However, the measurement of impact needs to be sensitive to the type of change being attempted.

Our suggestion is that successful NTF activity is a major contribution to a J-curve effect. While the J-curve is a common model in change management with the early stages of an innovation moving along on a plane and probably with people (including funders) wondering what the impact will be, the curve then takes up an upward thrust and considerable impact is achieved (see figure 2). This may at first seem a fanciful reading of change management. After all, if such was in fact the case, would not the history of higher education be one of discontinuous revolution? In fact that is the clue we take in this account – the possible contribution of studies of revolution to the analysis of educational change within institutions. What we take here as a heuristic is a well established feature of functionalist sociology and while we are not necessarily adherents to this approach, we do see some strengths in exploring the lines of expectation versus achievement as it has appeared in some sociological accounts. Our interest arises for the following reasons: the model is conceptual and capable of appropriate realisations in different contexts; it is measurable and therefore it helps in defining impacts; it can be expressed in terms of shareable outcomes so that a dissemination strategy becomes an integral part of the account.

FROM EXPECTATIONS AND ACHIEVEMENT TO THE J-CURVE

The J-curve model of revolution was first introduced by James Davies (Davies, 1962; 1969; 1971. Also see Gurr (1971).[4] In the study of revolutions Davies, Gurr and others see a revolution as an outcome of a conflict between expected satisfaction of needs and actual experience of satisfaction. Paraphrasing Davies and using different terminology:

Transformational change is most likely to occur when a largely continuous period of objective and planned innovation is followed by a sharp positive impact. The all important effect on the minds of staff and students is of a need to hasten implementation and impact which rises in relation to expectations. (based on Davies, 1962, p. 5 cited in Kimmel, 1990, p.75)

Of course for Davies revolution involves a negative so that for him the precursor is a prolonged period of economic and social development (educational development) which is followed by a sharp reversal. The reversal occurs when expected needs rise and they can no longer be met.

This produces the J-curve of revolution - see figure 1. In this model once an intolerable gap appears between needs and expectations revolution breaks out. Davies applied this model to a number of world revolutions with a downward J-curve.

Figure 1: Davies J-curve model of revolution. Source: Davies (1969, p. 548)

Of course from our point of view if we treat this in terms of educational development and innovation the J-curve is a disaster waiting to happen. It is the nightmare of every risk taker in educational development and learning and teaching. It refers to when staff and students’ expectations run high but actual satisfaction does not meet expectations. It is a commonplace of marketing approaches, particularly when it is felt that students as consumers will not buy in unless needs satisfactions are met. What is of interest here is to think what is missing from Davies's model which needs to be built in to the lived experience of learning and teaching in higher education.

First, the J-curve model concentrates solely on needs and expectations at an uncertain mass level. It is an aggregate of psychological dispositions which are not located within the structure of the relevant society, or in our case, institution. What relationship there is is primarily ideological. Thus, there is no sense of impact from external influences. Nor is there any detailed conceptualisation of ‘satisfaction’ which is why marketing approaches often present a different world from the complexity of learning in higher education. There is also an assumption that all change prior to the outbreak is continuous and there is no scope for introducing discontinuous change. We therefore need to build a model that incorporates all these elements.

In Figure 2 we bring three lines together. The work of the individual innovation is the blue line which proceeds on a regular plane before taking a significant upward curve. This line does not exist in isolation.

Figure 2: J-Curve for Successful NTF Innovation

The institution, or the sector, depending on the level of generality employed, is faced with continuous improvement as both a desirable and a requirement. It is very much the line of efficiency and calculable change. We have placed it at 45 degrees. In our model the line of innovation proceeds below the lines of expectation at first. This is based on our reading of how NTF winners have responded to winning. Most have been surprised and delighted; they did not expect it. They often feel their work is 'below the surface'. In fact national teaching fellows do not occupy a single line or space. Their project, inevitably like all projects starts off on a continuous plane but what gives the work its impact and coverage is the involvement of the Fellow in the lines of institutional change and role in continuous improvement. Many national teaching fellows have been involved in large projects[5] which are based in institutions and departments. Thus the fellow already has a place defined in terms of the line of institutional change. For example, three fellows appointed in 2000 were leaders of FDTL projects and this pattern is repeated in subsequent years.

For NTFs there may be a gap between the work they embark on with their project and the line of institutional change as in figure three. It is certainly true that in a number of cases NTF winners have been marginalised and we are likely to find the position of figure three.

In figure three the project holder engages in a project which certainly provides for improvement - the end point is higher than the starting point. But the line is not drawn upwards by interaction with institutional step changes. Our data from analysis of questionnaires and interviews with winners from 2000-2002 and questionnaire data from winners in 2003 and 2004 suggests that if this pattern occurs it does not happen because the national teaching fellows wish it to do so. Their wish is for a stronger upward movement. In some cases the Fellows have gone outside their department either to work more extensively across the institution (winners who have moved into whole institution educational development), or they have aligned themselves more strongly with their discipline through the Higher Education Academy Subject Centre network so the upward impact of their project has a different source of impetus. A number of national teaching fellows have been or continue to be Subject Centre Directors or Associate Directors. It therefore matters what focus we take to explore impacts. It is easy to see the work of a national teaching fellow within their institution only in terms of figure three when in fact the impact of their work is at discipline level or is located cross institutionally. In such a case figure 2 may give a more accurate measure of impact. So, we have to find ways of combining the two experiences.

THE STEPS OF DISCONTINUOUS CHANGE

The size and nature of the step changes may seem arbitrary. Of course they may assume different ‘space’ and impact in different institutions. We suggest that many steps arise from complex interactions in policy environments. The ‘steps’ we might tentatively suggest for consideration are derivable from the HEFCE strategic plans, from institutional missions and strategies. Thus we might suggest the following as possible candidates: (i) shift from external review to audit and often a move away from formal validation to continuous review processes, (ii) international students and recognition that world-wide learning takes different forms, (iii) diversification of funding streams, (iv) strategies for supporting students after tuition fees increase in September 2006. Each of these topics impact on most UK HE, with their correlates in other national HE systems, yet each can be realised in distinctive ways. How learning and teaching for undergraduates (the single most significant aspect of core business for all UK higher education) links to and underpins strategies here will vary across institutions.