Draft - Not for unauthorised quotation

Improving the usability of educational research: Guidelines for the reporting of empirical primary research studies in education

Mark Newman

Diana Elbourne

Evidence for Policy and Practice Information and Coordinating Centre (EPPI- Centre)

Social Science Research Unit (SSRU)

Institute of Education

University of London

Marilyn Leask

Teacher Training Agency

ESRC Teaching & Learning Research Programme Annual Conference

Cardiff 22/24 November 2004

Roundtable Discussion Paper

Contact details:

M. Newman

18 Woburn Square

London WC1H 0NR

Tel: +44 (0) 207 612 6575

Fax: +44 (0) 207 612 6400

E-Mail:

Introduction

There is considerable rhetoric about increasing the usefulness of educational research for a wide range of audiences including researchers, academics, practitioners, policymakers and users of educational services (Slavin, 2003; Sleeter, 2004; OECD/CERI, 2002). This requires the development of new ways of working for both researchers and practitioners including efforts to make the details of research more accessible (Leask, 2004; Slavin, 2004).

For this reason, the EPPI centre ( has been funded by a range of bodies to work in collaboration with educators to develop ways of making research findings accessible to teachers, teacher educators, policy makers. As well as developing models for research synthesis or systematic reviews, the centre conducts and supports the production of systematic reviews. In contrast to traditional forms of literature review, systematic reviews are characterised by the use of structured and explicit processes (Gough and Elbourne 2002) to identify, evaluate, data extract and synthesis findings from primary studies that have addressed a particular research question.

The quality and comprehensiveness of systematic reviews are highly dependent on the quality and comprehensiveness of the reporting of, for example, the contexts, aims, and methods used in the primary studies. A common concern of those conducting EPPI reviews has been that fairly straightforward information necessary for understanding and synthesising individual studies such as sample sizes and characteristics, location and duration of studies is not reported in published articles. This paper reports on the findings of a scoping project that set out to investigate this phenomenon more systematically, and to produce a draft set of ‘reporting guidelines’ which authors might use in reporting primary empirical studies so that readers of articles can more accurately assess the usefulness of the findings. The project was funded by the TTA as part of a series of initiatives intended to ensure research findings are more accessible to and useable by teacher educators (Leask, 2004).

What is the extent of inadequate reporting/?

The first stage of the study investigated the empirical evidence of under reporting. Data from any primary study that is included in an EPPI Centre Systematic Review is entered onto a single database (EPIC). A sample of studies was drawn from the database in March 2004. All studies on the database were included except those that had been used either in a systematic review of ‘young people and healthy eating’, or those put onto the database as part of a large-scale review of special needs education undertaken by the University of Colorado. The studies were excluded to avoid any skew in the sample toward ‘health’ ‘youth’ and /or Special Educational Needs journals.

Data are extracted onto the database using a standardised set of questions (EPPI Centre 2003). Sixty-six of these questions are designed to extract information about the aims and rationale, design, context and methods used in the study. For 32 of these questions the response ‘Missing/not stated’ is available to reviewers. Not all of these 32 questions are applicable to each study because some relate to the use of specific study designs, for example questions about randomization procedures. The denominator (i.e. the number of studies to which a question was applicable) therefore ranges from 125 to 489 studies.

Table 1: Information not reported in published studies on the EPIC database (for questions where missing/not stated not a possible answer)


The results of this analysis are shown in table 1. The table shows the total number of studies for which the answer to a particular question was ‘missing’ or ‘not stated’, the denominator (i.e. number of relevant studies for a particular question), and the proportion of the eligible studies for which the data was missing. Questions where more than 25% of eligible studies did not provide the information have been highlighted in the table. The data indicate that some aspects of the studies were consistently under reported Issues to do with the sampling strategy and the sample itself appear to be generally inadequately reported. This information is crucially important to understanding any research and its omission limits the value of any research considerably.

This is not a random sample of educational studies nor can it be said to be representative of all types of educational research. However it is worth noting the EPPI Centre review process requires that two reviewers extract data independently before reaching agreement on a single master copy, which is uploaded onto the database. This suggests that it is unlikely that any questions will have been labelled ‘missing’ or ‘not stated’ due to reviewer error.

What guidance is currently provided for authors by journals?

The second stage of the project consisted of investigating the reporting guidance provided to authors by journals publishing research in education. There are a large number and variety of such journals. In order to focus on generic, high profile journals, a sample of journals was constructed. The sample was constructed from (1) ‘general education’ journals cited on the above database; (2) a list of ‘prestigious’ education journals supplied by one large University Education Faculty Office; and (3) a similar list provided by a former member of the Education RAE panel. The sample consists of 11 journals cited in either list 1 and 2, 1 and 3, or 1,2,and 3. Guidance for authors was obtained either from a copy of the journal or from the publishers’ website. A list of the journals examined and a summary of the guidance provided by each journal can be found in Appendix A.

The table in appendix A shows the number of times a journal has been cited on the EPIC database, the name of the journal and the guidance provided by the journal to authors. With one exception, all of the journals listed provided only guidance on ‘style’ to authors. By ‘style’ we mean presentational issues such as font size, reference format, table formats etc. The guidance for authors given in the ‘Journal of Curriculum Studies’ specified that where an article was reporting original research a structured abstract containing the information listed in the table should be used. Interestingly this was not specified for the main text. All American Education Research Association Journals require authors to follow the American Psychological Association Style Manual (APA 2001), which is currently in its 5th edition and is 439 pages long. Interestingly only 13 out of the 439 pages discuss the reporting of research methods. The manuscript checklist contained in the style manual does not refer to research methods at all.

This is not a random sample of journals and is only a small fraction of the journals in the field (there are over 800 different journals listed in the EPIC database, for example), and therefore the results, while indicative, cannot be assumed to be representative of all journals in the field.

Development of the guidelines for authors

On the basis of the two investigations reported above it would seem reasonable to conclude that there is indeed a problem of inadequate reporting of at least some important aspects of primary educational research and that at least some prestigious and well known ‘general education’ journals do not provide guidance to authors about what information about the research study being reported should be included. Such guidance has been produced for a number of different research designs in the field of ‘healthcare’ (Altman et al 2001, Bossuyt et al 2003, Campbell et al 2004, Des Jarlais et al 2004, Stroup et al 2000). In the fields of Education and Psychology such guidance appears to have been limited to the call for the use of structured abstracts (Hartley 2003, Mosteller et al 2004). Given the lack of advice for educational authors and the developments in other disciplines, it does seem reasonable to assume therefore that the development of such guidance might be of use to authors and editors of educational journals. Papers that reported the information suggested by the guidelines would be more useful to the reader.

The draft author guidelines given in appendix B were developed from the EPPI-Centre ‘Guidelines for extracting data and quality assessing primary studies in educational research’ (EPPI 2003). These data extraction guidelines were developed based on the emerging ‘science’ of systematic reviewing and the cumulative experience of reviewers conducting EPPI-Centre education systematic reviews since 2000.

The purpose of these draft author guidelines is to help authors prepare manuscripts that report their research study in a way that will enable readers, reviewers and editors to interpret and contextualize the study, and to assess its strengths and weaknesses. This enables the user to assess the applicability, quality and implications of a study relative to their own purposes. As such, the guidelines aim to promote more ‘complete’ reporting of important information about the aims, methods, contexts, results and conclusions of primary empirical research studies in education. The guidelines are intended to be relevant to the reporting of any kind of primary empirical educational research using any type of research design i.e. studies which collect and analyse data with the intention of answering a specific research question. The detailed information reported for any particular design will vary.

At this stage the guidelines have been presented simply as a checklist or a minimum set of headings which can act as reminders about areas to consider. These are DRAFT guidelines, which are presented as a basis for further discussion and amendments. We are currently engaging in a series of consultation activities with education communities both as producers of research and potential users of research. The intention is to try and obtain feedback and input about the use of author guidelines as well as their content. We will be glad to hear people’s views either in the roundtable discussion at the Conference or through individual written submissions.

References

Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D et al (2001). The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Annals of Internal Medicine;134:663-94

American Psychological Association (2001) Publication of the American Psychological Association (5th Edition). WashingtonDC. American Psychological Association.

Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, de Vet HCW for the STARD group. (2003) Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD initiative. Annals of Internal Medicine ; 138: 40-44.

Campbell MK, Elbourne DR, Altman DG for the CONSORT Group. (2004) The CONSORT statement: extension to cluster randomised trials. British Medical Journal ;328: 702-8.

Des Jarlais, D et al. (2004)Improving the Reporting Quality of Nonrandomized Evaluations of Behavioral and Public Health Interventions: The TREND Statement. American Journal of Public Health ; 94 (3 ): 361-

EPPI-Centre (2003) Guidelines for Extracting Data and Quality Assessing Primary Studies in Educational Research. Version 0.9.7. London: EPPI-Centre, Social Science Research Unit.

Gough D, Elbourne D (2002) Systematic research synthesis to inform policy, practice and democratic debate. Social Policy & Society. 1(3) pp225-236.

Hartley, J. (2003) Improving the clarity of journal abstracts in Psychology: The case for structure. Science Communication, 24, 3, 366-379

Leask, M. (2004) ‘Using research and evidence to improve teaching and learning in the training of professionals – an example from teacher training in England’, paper produced for the Higher Education Academy.

Mosteller F, Nave B, Miech EJ. (2004) Why we need a structured abstract in Education Research. Educational Researcher; 31 (1): 29-34

OECD/CERI (September 2002), Educational Research and Development in England, Examiners’ Report, London: OECD/DFES.

Olson, D. (2004) ‘The Triumph of Hope over Experience’ Educational Researcher 2004; 31 (1): 24-26

Sheffield, P. and Saunders, S. (May 2004) Scoping study related to development of a Web portal for users of education evidence basesReport presented to the Teacher Training Agency Contract Number: TTA32605P Leeds University: British Education Index.

Slavin, B. (2003) A Reader's Guide to Scientifically Based Research.Educational Leadership; Vol. 60 Issue 5, p12-17.

Slavin, B. (2004) ‘Educational Research Can and Must Address ‘What Works’ questions’ Educational Researcher ; 31 (1): 24-26.

Sleeter, C. (2004) ‘From the Vice –President’, Teacher and Teacher Education AERA Division K newsletter: fall, 2004, pp 2 and 6.

Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, Moher D, Becker BJ, Sipe TA, Thacker SB. (2000) Meta-Analysis of observational studies in epidemiology: a proposal for reporting. Journal of the American Medical Association ; 283: 2008-2012.

17 / How do groups differ eg SES, educational level / 205 / 52 / 25.4
18 / who collected data / 489 / 115 / 23.5
19 / Methods used to analyze data / 489 / 94 / 19.2
20 / Sex of individuals in sample / 486 / 90 / 18.5
21 / details of data collection method / 347 / 62 / 17.9
22 / Countries sample from / 487 / 87 / 17.9
23 / Authors trying produce representative findings / 489 / 76 / 15.5
24 / gender of learners / 340 / 44 / 12.9
25 / total no participants in the study / 485 / 60 / 12.4
26 / Which country / 489 / 59 / 12.1
27 / Why this time, these people / 489 / 37 / 7.6
28 / ages covered by sample / 343 / 23 / 6.7
29 / if prospective allocation what was unit of allocation / 157 / 10 / 6.4
30 / linked to theory, empirical studies / 347 / 19 / 5.5
31 / research question/hypothesis / 347 / 13 / 3.7
32 / data collection methods / 489 / 18 / 3.7
33 / If comparison between groups basis for division / 210 / 6 / 2.9
34 / If evaluation when were measurements made / 254 / 6 / 2.4
35 / Variables/ concepts measured/examined / 347 / 6 / 1.7
36 / Broad aims of the study / 347 / 3 / 0.9
37 / Study timing – How is this different to 13 / 347 / 3 / 0.9

APPENDIX A: Summary of guidance on reporting provided by

a sample of educational journals

Citations on EPPI-Centre database / Journal Name / Publisher / Guidance
27 / American Educational Research Journal / AERA / American Psychological Association Style Guidelines
15 / British Educational Research Journal / BERA/Carfax/
Taylor & Francis / Style Only
13 / Cambridge Journal of Education / Taylor & Francis / Style only
9 / Research Papers in Education / Taylor & Francis / Style only
8 / Learning and Instruction / Elsevier / Style only
7 / Journal of Curriculum Studies / Taylor & Francis / Style plus structured abstract for 'original research’ - 'Primary objective/hypothesis', research design' methods and procedures, selection and number of subjects, describe the experimental intervention main outcomes and results, conclusions
5 / Journal of Education for Teaching / Taylor & Francis / Style only + age/grade of students
4 / Research in Education / ManchesterUniversity Press / Style only
3 / Curriculum / Studies in Education / Style only
3 / Harvard Educational Review / Harvard Education Publishing Group / Style only
2 / Oxford Review of Education / Taylor & Francis / Style only

Appendix B: Guidelines for the reporting of empirical primary research studies in education

DRAFT for Consultation

Guidelines

STRUCTURED ABSTRACT

e.g. background; research question; methods; results; conclusions

INTRODUCTION

Study aim(s) and rationale

Consider:-

  • Broad aims of the study
  • Explain why was the study done: link to theory, why at this time, with this sample, in this context
  • Funding of study
  • Date study started and completed

Study research question(s)

Consider:-

  • Study research questions and/or hypotheses
  • Topic focus/foci of the study
  • Curriculum area if relevant

METHODS

Research design

Consider:-

  • Type of study design used e.g. questionnaire survey, ethnography, quasi-experiment etc.
  • The concepts or variables investigated

Methods - Sampling strategy

Consider:-

  • Population from which sample is drawn
  • How sample selected: methods of identification of population for sampling frame and then methods used to identify population from sampling frame.
  • Planned (a priori) sample size
  • How were people recruited into study?
  • Was consent sought, how and from whom?

Describing the sample

Consider:-

  • Number of participants
  • Age, sex, ethnicity, socio-economic status and educational level of participants including Special Educational Needs.

Describing the context

Consider:-

  • Type of educational institution,
  • Country in which study located
  • The learning environment (details will be study-specific)

Methods - Data collection

Consider:-

  • Types of data collected
  • Details of data collection methods or tool(s).
  • Who collected the data
  • Where were data collected
  • How reliability and validity of data collection methods or tools established

Methods - Data analysis

Consider:-

  • Which methods were used to analyse those data and why

Procedures for qualitative data analysis

Statistical approaches and tests

  • The reliability and validity of the analysis

RESULTS

The reporting of results will vary according to the study type and so only general guidance of particular relevance for secondary data analysis (as in systematic reviews) is given here.

  • Ensure that the results reported are consistent with the methods used
  • Report on all variables measured
  • Where data are in the form of numbers, consider:-

Providing both the numerator and denominator for categorical data

For continuous data provide sample number, and a measure of central ` location (e.g. mean) and variability (e.g. standard deviation)

Give exact p values where possible when tests of statistical

significance used

  • For studies where the data are text, show clearly how the results (e.g. themes) are derived from the data collected.

CONCLUSIONS

  • Differentiate between the results of the analysis and the conclusions drawn by the authors
  • Ensure that conclusions follow from the results.
  • Include a statement about the generalisability of the findings

Paper for TLRP Conference 2003 D3 121104