Designing Studies to Measure the Implementation and
Impact of Technology in American Schools
by
Larry V. Hedges
Spyros Konstantopoulos
and
Amy Thoreson
The University of Chicago
Paper Presented at the Conference on The Effectiveness of Educational Technology: Research Designs for the Next Decade, Menlo Park, CA, February 26, 2000
1
Hedges
Abstract
Different and complementary research strategies are needed to measure the implementation and the impact of technology in American Schools. To measure the implementation of technology surveys of (ideally representative) samples of American classrooms are needed. The National Assessment of Educational Progress (NAEP) provides some idea of how such surveys can contribute to knowledge and provides important lessons about the development of targeted surveys to learn more than can be discovered from a general purpose survey such as NAEP. Some recommendations for in depth surveys of technology use are given. Assessment of the impact of technology from cross sectional surveys is difficult, as illustrated by analyses of NAEP data on technology use and achievement. Recommendations for longitudinal studies and designed experiments that might provide less ambiguous information will be given. However a major consideration is how quickly technology and technological competence is changing, which may make information rapidly obsolete. Consequently a major concern in the design of studies about technology is how to developing information in a timely fashion. New strategies may be needed to provide useful information for policy formation.
1
Hedges
In commerce and manufacturing, in multinational corporations and individual households, computer technology has fundamentally altered how business is conducted and how people communicate. In the field of education, computers have become a common fixture in this country’s schools. In 1980, less than 20% of elementary, junior and senior high schools in the U.S. were equipped with microcomputers. Less than a decade later, virtually all public schools had some computing capability (U.S. Bureau of the Census, 1989). Similarly, student access to computers has increased dramatically, from more than 60 students per computer in 1984 to approximately 6 students per computer in 1998 (U.S. Bureau of the Census, 1998). Important questions for educators and policy makers concern the availability, use, and impact of computers in American schools.
The results from a number of published studies on the relationship between computer use and academic achievement indicate that this technology can bolster student outcomes (Becker, 1994; Christmann and Badgett, 1999; Hativa, 1994; Kozma, 1991; Kulik and Kulik, 1987; Liao, 1992; Niemiec and Walberg, 1987; Niemiec and Walberg, 1992; Ryan, 1991; Van Dusen and Worthen, 1994). In their research synthesis on computer-based instruction (CBI), for example, Niemiec and Walberg (1992) calculated a positive average CBI effect on achievement of 0.42 standard deviations. Ryan (1991) computed a mean effect size of 0.31 in a meta-analysis of 40 published and unpublished studies on computer use and achievement in elementary schools. Most of the subject-specific research on computer use and achievement have examined performance in science and mathematics.
There is some evidence, however, that the access to computers and the academic benefits that can be derived from computer use are not the same for all students. Although monies from federally-funded programs such as Title 1 that are targeted to assist disadvantaged students are often used to purchase computers (Scott, Cole and Engel, 1992), high-income and White students tend to have greater access than low-income and Black students and non-English speaking students tend to have the least access (Cuban, 1993; Neuman, 1991; Sutton, 1991). Moreover, even when high-SES and low-SES schools have comparable student-to-computer ratios, students in low-SES schools are likelier to use computers for drill and practice exercises while their more affluent counterparts engage in more challenging activities (Cole and Griffin, 1987; Kozma and Croninger, 1992; Watt, 1982). A number of quasi-experimental studies of the computer-achievement relationship for students of different abilities have also been conducted. The results from these designs are mixed. Some studies show that even under the same treatment conditions, high-ability students receive greater benefits from learning by computer than their lower ability classmates (Hativa, 1994; Hativa and Becker, 1994; Hativa and Shorer, 1989; Munger and Loyd, 1989; Osin, Nesher and Ram, 1994) while others indicate that high- and low-ability students attain similar gains (Becker, 1992; Clariana and Schultz, 1988). However, the results from longitudinal studies of computer-assisted instruction prompted some researchers to conclude that computerized learning contributes to the increasing achievement gaps between high- and low-SES students and between high- and low-ability students (Hativa, 1994; Hativa and Becker, 1994; Hativa and Shorer, 1989).
1
Hedges
Considerable evidence indicates that even though teachers have had increasing access to computers for instruction, very few actually use them. In 1996, for example, the National Education Association reported that although 84% of all public school teachers said personal computers were available to them, only around 60% indicated that they ever used them (U.S. Census Bureau, 1998). Analysis of teacher data from NELS showed that about half of 8th grade math teachers have students who spend less than 10% of class time working on computers (Owens, 1994), while across subject matter, teachers average only about 4% of all instructional time with computers (Cuban, 1993). A survey of middle school math and science teachers in South Carolina (Dickey and Kherlopian, 1987) also showed that although 70% of these teachers had access to computers, almost half of those with access did not use them. Thus even though computer technology may be widely available, in general, it is poorly integrated into the classroom curriculum and is under-used (Maddux, Johnson, and Harlow, 1993; Becker, 1991; Ognibene and Stiele, 1990).
Most of the research on technology in schools indicates that computers have had little effect on teaching practices or classroom activities. Some authors (Cuban, 1993; Scott, Cole and Engel, 1992) have argued that computer use in schools simply follows the pattern of new technology such as radio and television when they were introduced. According to this view, the educational systems conservatism resists innovation, seeking to retain current goals and social organization. As a result, new technology is incorporated in old ways. Moreover, the sharp increase in the number of computers in schools is due to the efforts of those who profit from this expansion, such as hardware and software makers, not educators. These profiteers have been particularly successful by supplying goods and services for federally-funded programs for low-achieving minority students. These programs often feature computer systems with drill and kill software and are designed to replace teachers and control student behavior (Scott, Cole, and Engel, 1992).
A System of Studies to Provide Information on the Implementation and Impact of Technology in American Schools
The purpose of this paper is to consider the kinds of studies that are necessary to obtain the information needed to measure the implementation and impact of technology in American schools. I will argue that no single study or even genre of studies is adequate for the task of understanding and monitoring the uses and impact of technology. Instead a system of data collection, including studies of different types for different purposes is necessary. Little of what is recommended here is revolutionary. Indeed much of it is rather straight forward, even simple minded. However many of the things suggested here are not part of current practice. Therefore they may have particular utility precisely because they are feasible but not currently done.
A comprehensive program of assessment of technology must include a program of interrelated studies. The individual component studies would focus on different aspects of technology, use different methods, and have different purposes and time horizons. Some of the components (such as NAEP as a largescale survey component) are either already in place or studies that are already in place could be modified slightly to accomplish the purposes of a component of a technology information system. Other components would have to be created as wholly new systems. However regarding them all as components of a unified system could bring certain efficiencies and improve the overall quality of information about technology in education.
The system of assessment I envision should include as components four kinds of studies. The first component consists of large scale surveys of technology availability and use based on probability samples of schools and students. Such surveys provide the broad base of information about technology that are necessary to understand the extent of technology availability and use in American schools. They also provide information about the availability and use of technology in the homes of school children. They can also provide useful, albeit limited, information about the ways technology is used in classrooms in particular subject matters. Such studies are absolutely essential in providing the framework for designing more detailed studies of technology use. They are also essential for monitoring the inequality of access to technology and its use, the so-called digital divide which threatens to expand societal inequality.
1
Hedges
It is important to realize however that large scale surveys cannot provide all the information that is needed to understand the impact of technology in American education and inform national policy formation in this area. Large scale surveys cannot, with plausible cost constraints, provide detailed information on the detailed use of technology in American classrooms and the meaning of that technology use to teachers, students, and parents. Large scale cross sectional surveys cannot provide adequate information on cause and effect. In particular they cannot provide convincing information about the effectsof technology use on student achievement. Moreover, while large scale surveys with probability samples are the best tools available for understanding the current status of schools and their students, they are complex and therefore time consuming. In a rapidly changing environment like that of computer and information technology, large scale surveys may not be able to provide timely information about important trendsthe information provided may be obsolete by the time it is available. The other three components of the system are designed to address the information needs which surveys are ill suited to address.
The second component of the system I envision is a program of intensive studies of technology use in actual schools and classrooms. Such intensive surveys could be coordinated with a large scale survey, but would not have as extensive a sample of schools and would not necessarily be based on a probability sample. Such studies would be designed to provide detailed information about how technology was actually used in classrooms, who uses it, its relation to the broader curriculum, how it affects teacher and student roles, and the general level of satisfaction of students and teachers. Such studies would almost certainly involve interviews or classroom observations as well as conventional survey methods. Coordinating such intensive studies with large scale surveys would provide an opportunity to situate the intensive information provided within the broader national context.
The third component of the system would be studies designed to assess cause and effect. It is critical to assess not just that technology is used, but that it has effects on student achievement, attitudes, and behavior. Cross sectional surveys alone simply cannot be used make persuasive causal arguments in this area. Consequently it is necessary to do studies with designs that can provide evidence with higher internal validity. Such studies would include both longitudinal studies and randomized experiments. Longitudinal studies measuring many process variables can provide much less ambiguous evidence about cause than cross sectional studies. The measurement of intervening variables can also be an important benefit of such studies. However even longitudinal studies without random assignment do not rule out all rival hypotheses about cause. Randomized experiments provide the least ambiguous information about causal effects. Randomized experiments are both desirable and feasible in education and they should be seriously considered as part of a system of work to understand the effects of technology in American education.
The fourth component of the system I envision involves new methods of assessment and new research designs designed to provide very timely information about trends in the rapidly changing technology environment. It would involve a system of teacher-researchers in a network of schools distributed across the country. The purpose of the network would be to provide on-going feedback about technology issues. In principle the system could alert us to emerging trends that were not anticipated. Moreover it could be adaptive in the sense that it could be used not only to identify new data needs, but also to collect preliminary data about issues which were not previously identified. Moreover such a network of teacher researchers could provide useful formative information for the design of surveys or more intensive data collections at a later time.
The next four sections of the paper expand somewhat on each of the components outlined above, providing examples where available and relating the work to existing data collection efforts. In each case I argue that the component is feasible in terms of cost and the current state of the technical competence in educational research.
1
Hedges
Large Scale Surveys
Much of my insight about cross sectional surveys of technology use is based on recent analyses of the National Assessment of Educational Progress (NAEP), including the 1996 main assessment in mathematics and the 1998 main assessments in reading and writing (see Hedges, Konstantopoulos, and Thoreson, 1999). NAEP is an important component of the nations educational data collection system and its regular data collections based on cross sectional probability samples will likely remain so for the foreseeable future. NAEP already asks questions about technology availability and use in its student, teacher, and school questionnaires as part of the general background information collected. Therefore it is reasonable to assume that the cross sectional studies component of a technology information system would rely on NAEP for all or part of this component of the system.
NAEP is well funded and superbly executed with respect to sampling design and execution. It is the most extensive and valid source of data on what 4th, 8th, and 12th grade students in the United States know and are able to do. Therefore NAEP is well suited to its primary purpose of describing the patterns of academic achievement in America. Other cross sectional surveys will share its weaknesses but it is difficult to conceive of any that would provide superior measurement of academic achievement.
However well designed NAEP may be for its primary purpose of collecting achievement data, it is less adequate for its secondary purpose of collecting data about technology availability and use in American schools. A major shortcoming in NAEP for the purposes of providing information about technology is that the subject matter specific surveys in NAEP are not always seen as part of a coordinated system. The background information collected by NAEP (such as information about technology) is sometimes varied across subject matters and changes from year to year. However what is background for some purposes is foreground in others and such practices can limit the usefulness of the data provided.
Question Design Should be Principled Considering all NAEP Surveys as Components of a Single Data Collection System
The strategy for designing the computer use questions in NAEP is unclear. For example, the subject matter content (e.g., assessed subject matter or not), location (e.g., at home or at school computer use), and amount of time spent using the computer (time on task) could be used to provide one logical design framework around which computer use might be measured. No doubt there are wiser design frameworks, this one is intended as a simple minded example to illustrate the point. Questions designed around a more specific framework would yield more useful information about computer use and its relationship to achievement. In any event, more specific and consistent questions would be a valuable forward step. Note that there might be good reason to vary question wording across subject matter assessments (for example in a matrix sampled design). The argument here is that such variation should be thoughtfully designed to yield maximum information.
1
Hedges
For example, the 1996 mathematics assessment and the 1998 writing assessment student questionnaires ask how often students use a computer at home for schoolwork, and how often they use a computer when they do mathematics at school. The intent of each question was unclear. The first question may have been intended to elicit general information about computer use for all schoolwork in all subjects, not just mathematics. Such a question may well yield valuable information. However, we believe it would be useful to also ask (or to ask instead) how often students use a computer at home for schoolwork in mathematics (or some other construction that specifically targets at-home use of computers for mathematics or writing schoolwork. Such a question, in addition to the question about using computers for mathematics at school, would provide a more complete picture of how computers are being applied in mathematics learning.