Stories from Commissioned Research
Pat Thomson and Helen Gunter
Summary of paper presented at the British Educational Research Association Annual Conference, University of Warwick, 6-9 September 2006
This is not a traditional paper. We were members of a symposium and chose to present some preliminary findings from our observations and conversations with our colleagues in the research community in education. As a result of the paper and the session conversation we present a summary of the issue, the trends we have identified and some recommendations for the field to consider.
There are lots of stories about the conduct of commissioned research – horror stories, stories in which the forces of evil triumph over good. These urban legends of ‘heroic researchers’ and ‘dim-witted commissioning agents’ are of course incomplete and one-sided; we do however hear more of them, and the problems of commissioned research, than of trouble free and/or highly influential projects. This may be because such narratives become a trope for the way in which academic work is becoming more regulated and less’ free’. However, most reports on the (dire or otherwise) state of research have not investigated the conduct of commissioned research while generally endorsing its continuation or expansion, thus allowing such stories to proliferate without substantiation or deconstruction.
It is now over a decade since Gray’s report for the ESRC on research in education, and he notes stories by researchers ‘finding themselves severely constrained by their sponsors’ and that ‘contractual restrictions on publications are also likely to inhibit the development of rigorous research’ (Gray 1998: 30-31). We can find no further systematic strategic research into these matters, and yet one commissioner, the DfES (2006), reports that the budget for 2006-2007 for ‘externally commissioned research’ is £4.5 million (p24) which will be with ‘our partners to provide evidence on pivotal policy questions’ (p4). Given the resources, the commitment to work with researchers, and the varied perceptions of the relationship between commissioner and researcher, we decided to re-open the debate about commissioned research at a symposium at the BERA annual conference, 2006.
This is timely as the banner headline for the THES on December 1st 2006 is ‘Ministers vilify researchers’ (Baty and Shepherd 2006) and across a two page spread (Shepherd 2006) there are reports of the experiences of researchers who have ‘been subjected to concerted campaigns of “vilification”, have had their work publicly rubbished and have been subjected to repeated personal criticisms’ (Baty and Shepherd 2006, p1). Reports are provided from educational researchers such as Peter Tymms and John MacBeath, with different experiences. While both have found their work attacked, Tymms talks about how the Centre that he directs have found it difficult to win further contracts, while MacBeath is clear that ‘the climate has shifted’ where the government has learned its lesson and so ‘the row is over now’ (Shepherd 2006, p6-7). These stories confirm two things in regard to our BERA paper: first, that there are stories of particular experiences that need to be more systematically researched; second, there are different experiences where some researchers do not face difficulties, while others do. Furthermore, our interviews show that there are other issues than the end of report vilification of results, and that there are pre-project as well as process management issues that are not necessarily in the public domain.
We present a brief summary of our paper and conclude with a set of ideas for further action.
Thomson and Gunter: summary of paper from BERA 2006.
Our initial intention was to solicit stories from our colleagues via BERAmail and then to follow up with a random set of individual interviews: appeals via the list and on the BERA website produced no responses, so we adopted a ‘snowballing’ approach to data production. We interviewed 11 people whom we knew had conducted commissioned research and took notes of each conversation (we have not included ourselves). We have sorted out the major themes from these interviews rather than focus on ‘cases’: nine of our respondents were keen for us to ensure anonymity for fear of repercussions. That this fear exists is noteworthy. We present the information we have, not as a thorough piece of research, but rather as the beginning of a process – and we return to this point in conclusion.
Some of the people we interviewed were long term commissioned researchers, where others were early career and had less experience. The projects they had worked on ranged in size from about 10 thousand to well over a million pounds. The agencies discussed included: DfES, QCA, TDA, GTC, BECTA, Royal Navy, local authorities, NCSL and Academies and Specialist Schools Trust.
Four interviewees reported having no problems in the conduct of commissioned research, six reported varying degrees of problems and one said that they had never done a piece of commissioned research that was problem-free. We were told that it was necessary to understand the nature of the process – the audience, the role and responsibility of the sponsor, and the need to contribute to policy. Three people told us that a confident researcher can ‘control’ the commissioner, making productive changes in the tender documents. Another said that if you used mixed methods, write carefully, are flexible and check data rigorously then there won’t be any problems. Four reported finding the work enjoyable and reported impacting on policy and practice: a further six talked about how some of their research was of this order.
The themes we identified were:
· Commissioning
Some projects were not put out to tender: we were told of some very large sums involved. Some agencies had very little transparency in the tendering process with criteria for selection being vague. There are tensions between appropriate use of professional networks and nepotism/cronyism. There was a suspicion that politically sensitive research had deliberately gone to researchers outside the UK. We were told of one example of a project changing specifications between the tender document and the initial contract.
· Contracting
One person said it was important to anticipate problems and ensure that they are covered in the contract. But we were also told of a series of problems related to continued commissioner modifications to the contract around milestones, different versions of documents and confidentiality of data. This took a week of the project manager and two days of time for two PIs to sort out. We were told that there were often variations to the contract requested during the project and it was necessary to negotiate tradeoffs (if we do this now, we do something else later). There was also he instance of a commissioner failing to produce data required for the project.
· Research instruments
Commissioners are prone to ‘interfere’ in the design of research instruments, requiring particular questions to be asked and others not. In one instance a commissioner used ‘track changes’ through a questionnaire – the changes were inappropriate and not discussed beforehand. Some reported no interference with instruments.
· Reference groups
Two people described a series of problems with reference groups which included confrontational meetings and emails/letters. There were two instances of steering group meetings being cancelled because the commissioner was unhappy with the material to be presented at the meeting.
· Data analysis
In one case a commissioner wanted to change the weighting of survey data because they wanted to say the results were representative (this was not the research design).
· Risk management issues
We were told of: two instances where projects which were finished early; three initiative ended before the evaluation was completed; one example of policy change before the pilot study was fully completed; two projects held up due to election blackouts; and three projects were commercial in confidence dealing by the commissioner held up other projects.
· Micro- management
The problem can emerge during a project or be inbuilt from the beginning. It was suggested that this was because: commissioned research is a relatively new phenomena and commissioning staff often have to learn how to do it on the job; projects are often managed by junior staff whose jobs are on the line if the project doesn’t deliver; it’s about experience - researchers who have a track record of delivery are not micromanaged.
· Findings
Our informants had experienced: being asked for findings ahead of time with no adjustment in the work schedule; emails asking for 6 bullet points; being asked to write a press release for the commissioner; the commissioner overclaiming emerging findings; requirements to drop some evidence; and several examples about the nature of the findings and their wording.
· Reports
There were: several instances of commissioners amending and editing reports prior to publication without checking; a commissioner claiming authorship; several instances of refusal to publish reports; one example of a project that ‘was buried because it did not fit government policy’; considerable evidence of pressure to write in bullet points; one instance where a report was written three times and then the commissioner rewrote it.
· Ethics
We had information that one commissioner now requires that data not be confidential to them.
· Publications
No-one reported being prevented from publishing after the final report was accepted although there was considerable discussion of the process and timing of reporting interim findings at conferences – and being prevented from doing so.
· Broader implications
One person was particularly concerned that the failure to publish, and manipulation of results, meant that there is then a bias in knowledge production. As other researchers use commissioner ‘manipulated’ publications as ‘evidence’, there is a skewing about what is known, if the primary data is not available for re-interrogation.
Our informants had their own perspectives on how such issues came to be. Most people agreed that the more politically sensitive a project the more likely it was that the project would be closely managed, and could face problems with early use of emergent findings, in publication and even being cut short. Several people suggested that evaluations in particular were commissioned in order to justify further funding, and anything short of ‘good news’ was bound to be disappointing and contentious (one researcher had a counter experience where bad news’ was accepted). One person was also concerned that political pressures often led to speedy research which was less rigorous than it might be.
Our conclusions
It is clear to us that a more thorough study is needed. While this set of stories is from a small number of non-representative researchers, they were from a mix of people with different standpoints and histories. There seems to us to be sufficient spread of concerns to warrant an investigation to see how far their experiences are common to the UK research community.
We are interested in how we might secure funds for a study to:
· update the ESRC study funded more than a decide ago when ‘evidence’ informed policy development was less the norm
· generate recommendations for universities as contract managers
· determine any implications for the BERA code of ethics and whether a complaints process is desirable
· investigate the possibility of an independent and national code of ethics for commissioned research
References
Baty, P. and Shepherd, J. (2006) “Ministers vility researchers” Times Higher Educational Supplement. 1st December 2006. p1.
DfES (2006) Analytical Strategy 2006. Nottingham: DfES Publications.
Gray, J. (1998) “An episode in the development of educational research” in: Rudduck, J. and McIntyre, D. (eds) Challenges for Educational Research. London: PCP.
Ranson, S. (1998) “The future of educational research: learning at the centre” in: Rudduck, J. and McIntyre, D. (eds) Challenges for Educational Research. London: PCP.
Shepherd, J. (2006) “An increasing number of researchers face censure over studies critical of Labour policy, raising fears about freedom”. Times Higher Educational Supplement. 1st December 2006. pp: 6-7.
1