Tension and slippage: the status and impact SERA, November 2005
of educational research in IcelandMacdonald, Kaldalóns and Jónasson
Tension and slippage: the status and impact
of educational research in Iceland
Allyson
Ingibjög Kaldalóns
Iceland University of Education
and
Jón Torfi Jónasson
University of Iceland
Scottish Education Research Association
Annual Conference, Perth, November 2005
Abstract
The status accorded to educational research may be as much a result of the activities of researchers as it is a reflection of the way in which research and development systems initiate, fund, use and reward educational research.
Labaree (2003) has written about the status of educational research in the academy and about the problems educational researchers face in ‘living with a lesser form of knowledge’ (Labaree 1998). From 2003-2005 an evaluation of educational research in Iceland was carried out under the auspices of the Icelandic Centre for Research focussing on:
- University-based educational research,
- Institute-based research,
- Development work in schools, and
- Research and evaluation in education in the private sector.
Several themes emerged from the evaluation. Here we will consider some assertions made by Labaree and evaluators of educational research and the extent to which they are borne out by the Icelandic findings. The status of researchers within the scientific community, and in relation to practitioners and policy-makers, is discussed. Further we consider dilemmas raised by the concepts of ‘coordination’, ‘cooperation’ and ‘collaboration’ in attempts to enhance the impact of educational research.
Tension and slippage: the status and impact
of educational research in Iceland
Allyson ,
Ingibjög
Jón Torfi
November 2005
Introduction
An evaluation of educational research in Iceland was carried out from 2003-2005. The findings could be grouped into six themes. In this paper we focus on two, thestatus of the researcher and the impactof knowledge gained from educational research.
We consider whether it is possible for researchers to maintain and develop high status and high impact with different target groups – can researchers write peer-reviewed articles for the scientific community, do policy-makers read research, and do practitioners find a useful relationship between development work or education and training to research? What is to be gained or lost by working towards higher impact with one or all of the target groups or towards higher status?
The evaluation was carried out under the auspices of a working group appointed by the Icelandic Centre for Research and a smaller management group which met between meetings of the working group. It was funded by he Icelandic Centre for Research, the Ministry of Education, Science and Culture, the Iceland University of Education, the University of Iceland, the University of Akureyri and Starfsafl. Early on it was decided that the evaluation should:
- Describe the scope and nature of research and innovation in education and draw attention to areas lacking in research,
- Consider the status of researchers with different target groups, working environments and research processes,
- Draw attention to the value and relevance of research for policy-making,
- Draw attention to the value and relevance of research for school and other educational activities, and
- Consider the conditions necessary for innovation in schools and the private sector and the relationship of innovation to educational research.
A summary of the key findings of the evaluation can be found at:
Methods
Four separate studies constituted the core for data-gathering and report-writing and focussed on:
- University-based educational research,
- Institute-based research,
- Development work in schools, and
- Research and evaluation in education in the private sector.
Data about university-based research was gathered by interviews, through access to databases on researcher activity and coding of individual publications covering a five year period, from 1998-2002. In particular information was obtained about research and researchers in three universities:
–University A (2300 students in all), which offers undergraduate teacher education and graduate level educational studies. The function of this university is, by law, the centre for teacher education and research.
–University B (one department, in Faculty of Social Science in largest university which has over 9000 students), which offers graduate level teacher educationand educational studies.
–University C (one department of six, new university, about 1500 students), which offers undergraduate teacher education and graduate level educational studies.
Information was also obtained for the same time-period from several institutes – at national and local level and attached to universities – through interviews and coding of publications.
Data on school development projects was collected through a questionnaire and interviews at preschool, compulsory school and secondary school levels.
Finally information was gathered about educational research and development in industry through interviews and coding of publications
A framework for analysis of status and impact
Labaree (2003) has written about the status of educational research in the academy and the difficulties of graduate students making the transition from one worldview to another, from the school or classroom to the world of research. He has also talked about the problems educational researchers face in “living with a lesser form of knowledge” (Labaree 1998).Educational knowledge is considered to soft and applied rather than hard and pure. It may be difficult for researchers to establish causal claims; education is the “softest of the soft fields of inquiry.”Labaree (1998) has discussed the nature of knowledge production in university education departments and the value judgements used when talking about educational research. Some of these we considered important in understanding the results of this evaluation.
We have summarised some of the terms invoked when discussingqualities of educational research(Table 1). The summary draws on discussions by Labaree (1998, 2003) and others.In this paper we will present some results from the evaluation and then discuss some implications by presenting different scenarios for raising impact or status.
Table 1Qualities invoked in discussions on the status and impact of research
Soft / HardApplied / Pure
Looks at specific problems / Speculative, an inquiry approach
Of interest to “consumers” / Of “academic” interest
Fewer formal opportunities for peer review / Clear procedures for peer review
Distributed disciplinary base / Well-classified/specific disciplinary base
Egalitarian researcher culture / Elite researcher culture
Low status within the university / High status within the university
Few indicators of progress
within the field exist / Clear indicators of progress
within the field exist
Less use of statistics; more use of
open-ended methods / More use of statistics
Appears to have an accessible terminology / Draws on a specific terminology
Distributed disciplinary base / Well-classified/specific disciplinary base
There are several positive consequences of the nature of educational knowledge, according to Labaree (1998). The production of useful knowledge is not necessarily a bad thing – responsiveness and usefulness can make it easier for funders to justify investment. Educational knowledge is relatively free from consumer pressures and university education is built on social rather than individual ends. The absence of disciplinary boundaries can offer research a certain freedom, though this is not necessarily an advantage if significant others, such as funding agencies, think primarily in terms of disciplines. The social organisation of educational research is relatively egalitarian, though this can pose problems for elite researchers, especially if they are associated with particular disciplines. Finally we can mention that “Education, however, is largely accessible to outsiders and therefore vulnerable to discursive critique from nonexperts” (Labaree, 1998, p. 11). This can work both ways though, for it also means that educational researchers have easier access to the public than others working in more prestigious but somehow more closed areas, such as cutting-edge biochemistry.
Labaree (1998) has also pointed out several negative consequences. These include low status within the university, weak authority with education and educational policy-making, a pressure to transform education into a hard science, a pressure to transform education schools into pure research institutions and a sense that the field is never getting anywhere. We have seen that the support given to research in the Iceland University of Education, whose primary mission is teacher education, is less than that given to the other two universities under study, the University of Iceland and the University of Akureyri. University researchers feel that they often write for policy-makers but that little attention is paid to their work. The introduction of the productivity assessment scheme at the universities studies has increased pressure for research that is “hard science” (for example, international peer-reviewed publications) rather than soft and applied (for example, development projects in schools). There are more incentives for pure research than applied research (points assigned to different activities and publications). An apparently high rate of rejection from national competitive funds has discouraged several of the more active researchers.
We will now consider some of the results in terms of the qualities in Table 1.
Results and discussion
The evaluation focussed on three main target groups (Figure 1), though in the final analysis we separated practitioners into those working in schools and those working in education and training in the workplace.
Figure 1Key target groups in the production of knowledge through research
Research by university researchers was classified in one of six ways (Table 2) and modes of publication in four ways (Table 3).
Researchers in Universities B and C indicate that they spend more effort on basic research (Table 2, 46%, 51%) than on applied research (26%, 33%) and more time on basic research than University A (28%), which has teacher education as its major function. By this measure about half the output of B and C is intended for the scientific community. These figures are corroborated by the reporting mechanisms used where we see that very little published work is peer-reviewed (Table 3, 6%, 15%, 16%) and that University A, tasked particularly with teacher education, has a particularly low output (6%). By comparison, in areas other than teacher education/educational science at University B, about 30% of the output is peer-reviewed (Inga Dóra Sigfúsdóttir et al., 2005).
The effort put into applied research is similar for all three universities (Table 2, 32%, 26%, 33%), but it is likely that much of this work is reported in publications that are not peer-reviewed.
If we consider evaluation, development work and advisory work as indicating work intended to be of direct relevance to policy-makers and practitioners then we see that University A (41%) has a higher proportion of publications in this area than B (23%) and C (33%).
All three universities report on about half their work at conferences (Table 3) and again much of this work is not peer-reviewed.
Table 2Nature of university research carried out from 1998-2002 as classified by university researchers themselves
Nature of research / Univ A% / Univ B
% / Univ C
%
Basic research / 28 / 46 / 51
Applied research / 32 / 26 / 33
Evaluation / 20 / 7 / 14
Development project / 14 / 11 / 13
Advisory work / 7 / 5 / 6
Review of research / 15 / 16 / 16
Total / 116%* / 110%* / 133%*
*It was possible to categorise some research more than one way.
Table 3Reporting mechanisms used by educational researchers 1998-2002 as accounted for in databases of publishing activities
Reporting of activity1998-2002 / Univ A
% / Univ B
% / Univ C
%
Peer-reviewed (e.g. journals) / 6 / 15 / 16
Not peer-reviewed (e.g. reports) / 35 / 27 / 22
Talks and posters / 53 / 51 / 55
Other – books/chapters/theses / 6,0 / 7 / 9
Researchers were asked whether their work was intended for the scientific community, for policy-makers, for practitioners or for the general public (Table 4). It was clear that the educational researchers in the private sector do not emphasise writing for the scientific community, instead writing mainly for policy-makers and practitioners. Not unsurprisingly researchers in institutes write primarily for policy-makers who commission many of the studies. In Table 4 the research team coded the institute and private sector documents but university researchers themselves coded their own documents. The results that showed that around 60% of university writing was targeted at the scientific community and practitioners, in and of itself not a surprising outcome. What was surprising was that university researchers felt that in 44% of their writings there was a message for policy-makers (Table 4). At the same time it was clear from interview data that academics felt that policy-makers did not make use of their research and policy-makers felt that research findings were generally not available in a form that was useful and accessible. Research results were often used indirectly in policy-making, depending in part on the type of research being read or used by consultants.
Table 4For whom do researchers think they are writing?
Producers(1200 documents)
Targets* / University research
% / Master’s research
% / Institute research
% / Private sector
%
Scientific community / 58 / 100 / 6 / 20
Policy-makers / 44 / 25 / 95 / 65
Practitioners / 61 / 56 / 31 / 68
General public / 7 / - / 10 / 9
* More than one target possible
These results of the evaluation indicate that:
•Educational researchers feel that they write for a range of audiences and that their work should have an impact for the scientific community, policy-makers and practitioners. Their work is generally unlikely to be peer-reviewed, the ‘gold’ standard of research, indicating that others in the scientific community will not assign high status to their work. By the same token the impact of their work within the scientific community is limited as many of their results will not reach an international audience.
•Much educational research is considered to be applied and/or aimed at practitioners. It appears in non-refereed publications and much of it is presented at conferences, many of which do not publish proceedings and/or they are not peer-reviewed. Reports and conference talks generally do not reach wide audiences, especially if the research itself has been commissioned and contains sensitive information. The status of latter publications can be ambiguous, carrying considerable weight with some and being written-off as ‘bought’ by others.
Tension and slippage
As a follow-up to these results we decided to take a systematic look at what the implications might be if attempts were made to change the status and/or the impact of educational research.
To carry out this analysis we use the qualities suggested by Labaree and others as shown in Table 1. Further we work from an initial position of what is considered to be low status and low impact research.
StatusLow / High
Impact / Low / A
/ B
High / C / D
Figure 2Options for changing the status or impact of educational research. These include moving from quadrant A to B, C or D.
In Figures 3a, 3b and 3c the consequences of attempts to move between quadrants in terms of what might happen and what might be lost.
Towards / Hard – experimental research?Pure – basic research, disciplinary based?
Being speculative
More “academic” interest – publishing in journals?
Detachment from the field under study – experimental?
Drawing on a specific terminology – disciplinary based?
An accessible terminology
A distributed disciplinary base
An egalitarian culture
Nearness to the field / < < Risk losing
Figure 3aOption AB: Increase status only
Towards / Applied research?Soft science – based on “educational principles”?
Problem-solving
Being of interest to practitioners – publish in teacher journals?
Being relevant to the field under study – action research?
Connections with development projects
Procedures for peer review
Speculation
Elite researchers
Development of new concepts and terminology / < < Risk losing
Figure 3bOption AC: Improve impact only
Towards / Links between research and developmentA wider range of skills used in EDR
Innovative peer-review procedures
New indicators of progress
More cooperation in definition and collection of data
Adapting reportingto different audiences
Revision of incentive schemes
Targeted research
Freedom of choice
Established relationships
Nearness to the field
Fuzzy reporting procedures / Risk losing
Figure 3cOption AD: Increase status and improve impact
What is likely to happen? Practitioners and policy-makers want research to have more impact on practice and most researchers would like that too. But the incentives in the system and imminent changes probably mean that university-based researchers will drift to more status, towards more peer review, more basic research, more cooperation in the definition and collection of data and more of a disciplinary approach. While this may improve the quality of the research, we should guard against losing our nearness to the field, an accessible terminology and a distributed disciplinary base.
Coordination, cooperation and collaboration
We turn now to look at the dynamics of coordination, cooperation and collaboration and how this might affect the impact of educational research.
First, do researchers work together now? Over 70% of university publications are by individual authors (up to 90% in one of the universities). About 17% of institute publications are clearly by individual authors and in 42% the institute is given as the author, with similar figures for educational projects in industry. Do researchers work on common projects?From 50-66% of projects are individual. Researchers are more likely to be working with people in other organisations than in their own.
Coordination requires and even creates meeting places, for example, virtual meeting places which are non-personal and make few demands on researchers. Journals are examples of such coordination – they provide an efficient way of finding out what others are doing; researchers can access the meeting-place whenever they wish.
Real meeting places can also be non-personal; making more demands on the individual, but still no commitments are required. Conferences like these are an example – they too can be non-personal; we can choose what we want to know more about, but must meet at the same place and time as others if the conference as a concept is to work.
We also find coordination within spheres of activity e.g. specialised journals. We need to think whether we want more coordination? Do we need coordination in the way researchers approach the grassroots? In the way we go into schools with our questionnaires, do interviews and observe what takes place? Schools are often the site of many research projects – would it help to introduce more coordination?