Research Methods; Their Design, Applicability and Reliability

Chapter 10.2

Research methods; their design, applicability and reliability

Gail Marshall

Gail Marshall Associates

Walnut Creek, USA

Margaret Cox

King's College London

London, UK

Abstract: A review of research and evaluation studies into IT in education shows us strengths and weaknesses. Both quantitative and qualitative studies conducted in the past have often been based on inadequate design, limited discussion of settings or measurement development and analysis. The limited consideration of different educational settings and the small sample sizes in many studies result in difficulties in generalizing the results. Few studies have been linked to specific learning activities using specific IT tools, further compromising generalisability of factors impacting innovation and implementation activities have seldom been addressed, making studies less than useful to a wider audience.Failure to acknowledge that some research questions, those based on Logo use, for example, are best answered by employing qualitative methods, has limited our understanding of IT’s impact on pupils’ learning. Similarly, the basic differences in epistemological theory and the consequent difference in research design and analysis have seldom been addressed by the research community.Suggestions are offered for improving the quality and applicability of future IT research studies.

Keywords: Research; reliability; research methods; theories; research goals; standards

10.2.1Introduction

This chapter reviews the strengths and weaknesses of different research methods used to measure the impact of IT in education. It includes reviewing the designs and approaches, and the relevance and applicability of quantitative and qualitative methods drawing on previous research evidence. Many previous research studies have relied upon methods which measure changes in education due to innovations, generally resulting in many cases of a limited applicability of the methods to the research goals. Such studies could have been linked more specifically to the learning activities promoted by specific IT tools and the relative emphases required to take account of different factors influencing the integration of IT in education. A basic premise of this chapter and of research into IT in education is that important questions are generated, and strategies that are appropriate and rigorous are deployed to answer those questions as unambiguously as possible. A major goal of research and evaluation is to advance knowledge.

'Evaluation' is often considered a more pragmatic aspect of educational research than the techniques employed in “classic” research studies. Evaluation has usually been conducted to answer questions of interest to funding agencies, education authorities and other stakeholders who demand answers to questions of “How?” and “Why?” and “Of what influence?” Evaluation results, for better or for worse, have often been influential amongst policy makers in determining the effectiveness of IT in teaching and learning. We therefore include some discussion about evaluation design and approaches, and the consequent results in this chapter in addition to the often more rigorous research techniques which have been used. Design and evaluation of IT tools are discussed in more detail by Reeves (2008) in this Handbook.

10.2.2Research Goals

Before considering the effectiveness and relevance of different methods which have been used to research IT in education we review the different goals of researchers, which have been influenced by technological developments, national priorities, expectations of educators, the types of IT tools available and so on. The major research goals have changed as a result of the growth and uptake of IT in education as discussed by Cox (2008) in the Handbook. The research goals reported on most extensively are listed in Table 2.1 below showing the types of research methods most often used to address these goals.

Table 2.1- Research Goals and relevant research methods

Research Goals / Quantitative / Qualitative
To measure the impact of IT on pupils’ learning. / Large experimental and control groups.
Pre- and post tests of learning gains.
Meta-analyses of quantitative data.
Large-scale (on-line) surveys. / Small groups of learners using specific IT tools either as an intervention or part of the natural class activities.
Observations, specific task performance, focus groups
Interviews of learners.
Pre- and post tests.
Uptake of IT by schools and teachers. / Large scale surveys.
Questionnaires of many teachers and schools. / Questionnaire surveys of teachers in schools.
Observations of classes.
Effects of IT on learning strategies and processes. / Pre- and post tests of specific subject processes. / Assessment of prescribed tasks.
Observations of pupil-computer interactions.
Effects of IT on collaboration, contextual effects etc. / Class observations, teacher interviews, questionnaires, documentation.
Attitudes towards computers in education. / Attitude tests of pupils and teachers. / Interviews and focus group discussions.
Effects of IT on pedagogies and practices of the teachers. / Large scale surveys of frequency of use, types of IT use, etc. / Class observations, teacher interviews, questionnaires, documentation.
Computer use by girls versus boys. / Large scale questionnaire surveys of computer use. / Class observations, pupil interviews, questionnaires.
Contribution of IT to enhancing access and learning for special needs. / Large scale surveys.
Questionnaires of many teachers and schools. / Small groups of learners using specific IT tools either as an intervention or part of the natural class activities.
Total operating costs and cost effectiveness. / Online surveys of all school staff, Large scale surveys. / Teacher interviews, school records, staff questionnaires.

As explained by Cox (2008), in the late 1970s and 1980s the major emphasis of researching IT in education was on investigating the specific impacts which an IT intervention might have on students' learning outcomes, their motivation and the impact on understanding specific concepts and skills. The methods used were mostly methods of subject based assessment which had been used to measure impacts of other educational innovations and interventions. This focus on the impact of IT on learning changed as more factors were identified. The types of interventions were dependent upon the priorities of the educational establishment, the way the curriculum was delivered and the types of IT tools developed at the time. These changing goals shown in Table 1 and discussed below were as a consequence of the growth of IT in education and the outcomes of previous research studies.

10.2.3To measure the impact of IT on learning

Since the mid 1960s one of the major goals of researching IT in education has been to examine an intervention (“What was the treatment?”) and examine the impact (“What was the impact?”). Explaining the variance between what occurred (or what did not occur) and the consequent impact should be part of any educational research project. As discussed by Pilkington (2008) and Reeves (2008), the original design for measuring the effects of an intervention was to expose a select group of pupils to using IT tools specifically designed for educational purposes, sometimes in a school setting, sometimes in a laboratory and to measure improvements in learning through pre- and post-testing. For examples of this type of research see the early editions of Computers & Education and similar journals. The various limitations of this type of research design included the researchers often assuming that the tests could be based on those which measured traditional learning gains. The intervention was often an additional extra to the pupils’ learning programme and the results could not be generalized to other schools and settings.

10.2.4Uptake of IT by schools and teachers

Once IT resources became cheap enough to be able to be purchased in large numbers, schools purchased networks of computers, clusters of stand-alone ones and some for administrative purposes. The increase in provision resulted in governments and local districts wanting to find out what the uptake was by individual schools and teachers. Many governments commissioned large-scale surveys of their schools to find out how IT resources were being used. A good example of the range of research instruments used for national surveys is given by Pelgrum and Plomp (1993, 2008). They coordinated several international surveys of the uptake of IT by schools, teachers, head-teachers etc. There have been many large-scale studies since the 1980s which have focused on measuring this level of resource use (e.g. Watson, 1993, Harrison, et al., 2002).

10.2.5Effects of IT on learning strategies and processes

Once there were sufficient IT resources in many schools and classrooms, there was an increase in research projects to measure in more detail how specific IT uses impacted upon learning strategies and processes, requiring different research methods as shown in Table 2.1 including audio-visual recordings of pupils’ IT uses, human computer interactions, and the devising of specific tasks which pupils had to complete relating to specific IT uses. The most reliable evidence of a positive impact of IT tools on learning strategies and processes as explained later was with this research approach where the pupils were assessed in great depth and detail regarding specific identified uses of IT (Cox & Abott, 2004).

10.2.6Effects of IT on collaboration and the learning context.

As a consequence of a large number of studies of pupils using computers in different class settings and the growing awareness of researchers about the importance of team work amongst pupils, research in IT in education expanded to include the goal of measuring what effects pupils’ collaborating had on their learning and team-work skills when using IT and conversely what effects the use of IT environments had on those collaborative skills. Since the rapid growth in the use of online learning then more sophisticated research tools have been developed, including online monitoring of pupils’ computer use, online assessment techniques and the need to take account of the different knowledge representations which such complex environments provide (Cheng, Lowe & Scaife, 2001)

10.2.7Attitudes towards computers in education

In spite of a large increase in IT resources in schools and informal educational settings, the research into uptake by teachers and pupils still showed that this was disappointingly low. It became apparent that the attitudes of teachers and learners significantly affected their willingness and abilities to use IT tools and thereby the level of benefit which could be achieved. As a consequence, there are now a large number of research studies into attitudinal and personality factors towards IT in education (Katz & Offir, 1988, 1993; Sakamoto, Zhao & Sakomoto, 1993; Gardner, Dukes & Discenza, 1993; Koutromanos 2004) in which attitude tests consisting of many questions about fear of computers, liking of technology, liking using them in schools, etc., have shown strong links between pupils’ and teachers’ attitudes and the effects on IT use and learning (see Knezek and Christensen, 2008 for more details about this research). Many researchers also claimed to measure attitudes of pupils by simply asking a few questions about whether they liked using computers or not, a strategy which does provide some useful evidence but is not so robust as using tried and tested attitude tests which try to measure the underlying feelings about IT.

10.2.8Effects of IT on pedagogies and practices of the teachers

Even though there was a gradual growth in IT use by teachers and pupils reported in many countries, the research evidence showed that the ways in which IT was used was very dependent upon the teachers themselves, what they believed to be important, how they selected the IT tools for their curriculum, how they organized the lessons and so on. Therefore many researchers focused on measuring those factors, mostly either through questionnaire surveys which only obtained evidence about level of use and types of IT use in different curriculum subjects (see for example Pilkington, 2008; Pelgrum & Plomp, 2008). In order to understand the effects of teachers’ beliefs and their consequent practices, it was necessary to conduct interviews with individual teachers, observe their behaviour in lessons and follow their practices over a period of time (Somekh, 1995, Castillo, 2006, Cox & Webb, 2004).

10.2.9Computer use by girls versus boys

In the early days of IT in schools, many researchers reported that especially with teaching IT or Computer Science as a subject, many more boys were making regular use of IT compared with girls, resulting in a range of studies into the rate of access to IT by girls and boys and types of IT use etc. The studies involved both large-scale questionnaire surveys as well as actual class observations, pupils’ interviews and focused questionnaires. These more detailed methods are able to find out whether girls use IT in different ways to boys and how the design and development of IT tools should take into account these differences (Hoyles,1989). See also Meelissen (2008) in this Handbook.

10.2.10Contribution of IT to enhancing access and learning for special needs

One important goal, which is sometimes overlooked by governments, is to measure the important contribution which IT use can make to pupils with special needs (Abbott, 1999, 2002). This is a very complex research area because of the contribution which IT can make to both physically and mentally disadvantaged pupils. Some IT devices may consist of hand-held manipulative toys which provide sound feedback when a task is performed correctly; others may involve sound output for blind pupils typing email messages on a computer; and others may involve providing safe IT environments for pupils who emotionally find it difficult to relate directly to humans. Measuring the effects of IT tools on pupils’ learning requires specifically designed measures to determine the specific characteristics of the IT device and how the device contributes to changes in skills and competencies.

10.2.11Total operating costs and cost effectiveness

A final goal which dominates many large-scale nationally funded research projects is to answer the questions; “Does IT provide a more cost effective solution to improve teachers’ and pupils’ performances? and “What are the total operating costs of IT in schools?” It is very difficult to provide clear irrefutable answers to the first question although some projects have tried to measure this by extrapolating financial benefits from the improvement in pupils’ learning (e.g. Watson, 1993). However national bodies and researchers are developing tools to answer the second question but they have to rely on teacher interviews, the accuracy of school records and questionnaire surveys of all staff (Becta/KPMG, 2006a, 2006b). See also Moyle (2008) in this Handbook. It is well known that it is very difficult to estimate the total operating costs of IT systems in educational establishments because account must be taken not only of the purchases of IT tools, systems, repairs, upgrades, online subscriptions and so on but also the costs of training the staff, miss-use, inappropriate uses of IT tools etc., while the IT environment and its role in education is constantly changing.

10.2.12Epistemological Theories and Research Design

A fundamental influence on the research designs to measure the impact of IT in education since computers were first introduced into schools in the late 1960s and early 70s has been the beliefs of researchers, educators and policy makers into how pupils learn (epistemologies). The Structure of Scientific Revolutions (Kuhn, 1970) contributed to an acceptance of different beliefs about how knowledge was organized and how investigations into the nature of all manner of things could be conducted. Thus, theorists began to re-examine traditional theories and methods in a wide range of disciplines. For example, Reese and Overton (1970) said that the epistemological foundations of researchers in the field of child development (i.e., how learning occurred and was organized) result in different and mutually exclusive strategies for research design and analysis. Dede (2008) in this Handbook elaborates how different theoretical perspectives influence the use of IT in teaching and learning. Briefly stated, behaviourists believe that knowledge is a copy of reality, that learning occurs from the outside in, occurs in incremental bits and is facilitated by repetition and reward. According to this philosophy instructional design including that using IT should be organized in pre-ordained steps.

Researchers working in the tradition of Comenius, (Kalas & Blaho, 1998), Piaget, (Sigel & Hooper, 1968), Bruner (1966) and Papert (1980) believe that pupils do not photocopy reality. Instead a mismatch between levels of mental maturity and instruction will result in “deformations” of what has been taught. Similarly, their research is based on the belief that pupils’ interactions, either physical or mental, are crucial components of learning, which, in turn, results in a partial or total reorganization of knowledge. Intrinsic needs, not reward and repetition, drive learning and instruction which should proceed in “spirals” (Bruner, 1966), or in the presentation of fully organized blocks of material. Serendipity, (Duckworth, 1972), capitalising on a spontaneous event that engages pupils’ interest and motivation, is also recognised as a viable foundation for instruction. The different beliefs about learning have had major influences on the educational design of IT environments and researching IT in education.

Some IT researchers design and conduct experiments based on a behaviourist epistemology. The early work of Bork (1980, 1985), Nishinosono (1989), Katz and Offir (1993) and many of the studies reviewed in the large scale meta-analysis by Niemiec and Walburg (1992) for example, report on investigations into the impact of IT on specific tasks through the analysis of test performances. Descriptions of the research might include resources, including costs associated with a “treatment” (Moonen, 2001), types of IT tools used but pedagogical methods are often only described very briefly. Pre- and post-tests are used to assess the extent to which changes (number of items correctly answered, for example), if any, occurred. Time; how much time pupils need to solve specific questions, is often an important variable. The methodology ignores questions about the conditions promoting changes in cognitive structuring because researchers worked within “the mind is a black box” framework. An example of such an approach is shown by the early work with pupils conducted by Katz and Offir (1988, 1993).The problem with this type of investigation is that we do not know whether the pupils understood all or part of the material used, or participated in the instructional activities, nor do we know if they understood directions or had the mental maturity needed to engage fully with the task. We do not know if teachers’ instructional practices were geared to all pupils nor if the time allowed for instruction was sufficient for all pupils to understand the tasks. Finally, we do not know what cognitive processes pupils used to solve the problems. Did all pupils use the same strategies? Did some pupils understand the problems but make simple mistakes? Could some pupils have solved the problems if more time were available?