Chapter 4.4
Observation Measures for Determining Attitudes and Competencies
Towards Technology
Renate Schulz-Zander
Michael Pfeifer
Andreas Voss
Institute for School Development Research (IFS)
University of Dortmund
Dortmund, Germany
Abstract:This chapter provides an insight into how observational measures can contribute for determining attitudes and competencies towards Information and Communication Technology. To get an idea of the capability of observational measures the first part of this chapter outlines the meaning of observational techniques as a tool of research and explains what is important when planning an observational study. For a better understanding in this context, several video- and audio-supported observation techniques are presented, as well as advantages and challenges of observational techniques in general. The second part of this contribution focuses on the methodology and on particular findings of ERIC-referenced empirical studies that researched attitudes and competencies towards IT by using observational techniques. After reviewing those studies and summarizing their findings, this chapter ends with the conclusions that observational measures as part of a mixed methods research design have the potential to deliver meaningful, unique data and findings. One of the major findings is that attitudes and competencies toward IT tend to be positive in most of the reviewed studies.
Keywords: observation measures, data triangulation, non-standardized research, attitudes/ competencies towards IT
1. Introduction
During the past quarter century major investments have been undertaken in most OECD countries to bring computers into classrooms. As a result of this effort a rapid improvement in student-computer ratios could be seen (see OECD, 2004; 2006). In most OECD countries primary as well as secondary schools have access to computers and educational software applications in their classes or in specially facilitated computer classrooms.
In the United States the No Child Left Behind legislation puts forward the need to continue these efforts in the IT sector and makes substantial funds available through a performance-based grant program (The White House, 2001). However, the grant program will also shift emphasis from ensuring access to IT to assessing the impact that technology has on classroom practices and on how much students learn at school (Dirr, 2003).
Until recent years empirical research on the effects of IT on students’ attitudes and competencies could not provide a clear picture about the relationship between IT access and student academic achievement. The research studies show controversial results: For example the 2006 OECD PISA report on computer attitudes declares an important effect on the extent of computer use and student performance in mathematics (OECD, 2006). Reanalysis of these results by the German PISA group showed that these effects are negligible if these analyses are controlled for context variables in a longitudinal data analysis approach (Senkbeil & Wittwer, 2006).
In order to address these controversies, alternative research approaches to determine the impact of IT on classroom practice and student achievement need to be considered. The challenge is to assess not only the extent of technology in the classroom but also the appropriate use of technology to facilitate student learning and to relate these observations to student achievement.
One family of approaches is observation techniques. The main advantage of observation techniques is that research objectives can be studied in their natural field settings (such as classrooms) and thereby provide a richer understanding of the topic under investigation. These techniques allow an alignment of classroom observation with the instructional context and the evaluation goals. According to Dirr (2003)observation protocols – or forms of observation techniques – serve as valuable tools in order to:
-Evaluate the effectiveness of a school program.
-Assess the performance of a teacher or a school.
-Provide feedback to teachers for professional development.
-Conduct basic research on classroom practices.
These functions make observation a useful component of educational research. For example, observation protocols can serve as a tool to measure whether the use of IT programsachieved anticipated changes in teaching practice and how these changes are related to student performance.
In this chapter we describe different observation techniques, present the rationale for observation techniques, and discuss their unique characteristics as well as advantages and challenges of this research approach. In addition we present examples of how observation techniques can be used to study IT attitudes and competencies and we report a synthesis of findings. We conclude the chapter by discussing how observation research techniques can be enriched and combined with quantitative research approaches,
2. Observation as an Approach to Researching IT Competencies and Attitudes
In general observation is regarded as a “primary” data assessment technique due to the closeness to everyday (non-scholarly) techniques in order to acquire information. In comparison to this “naive” kind of observation, the systematic form of observation under controlled conditions by means of structured observation criteria. In the following section different types of classroom observation measurement instruments are introduced and advantages as well as limitations of these data assessment techniques are discussed.
2.1 Use of Observation Techniques as a Tool of Research
Observation measures in educational research have been used for decades. Substantial research focusing on tracking classroom management and interactions that occur between students and teachers has taken place since the 1970s.
Observation measures for determining attitudes and competencies toward technology include a wide range of approaches, e.g. checklists, inventories, time interval ratings, holistic ratings, narrative descriptions, and logs. Some of these approaches have built on the prior work of others, but according to Dirr (2003) there is little evidence of any attempt to bring cohesion to these efforts.
Observation techniques allow a systematic, structured data collection process, using well-designed observation record forms. Compared with large-scale summative evaluation tools the main advantage of these techniques is that data are collected in their natural setting, thereby providing a richer understanding of the subject under investigation. In the instructional educational research context these techniques allow a researcher to view and monitor processes that lead to student comprehension. The data can be used to explain students’ performance.
Observational studies in general go through several stages. The Center for Development Information and Evaluation (1996) named a step-by-step list for observational techniques which can be used to judge quality. These steps are named in logical order of the data collection and analysis process.
Step 1: Determine the unit of instruction being studied
The first step is to decide which unit of instruction is to be studied.This consideration is especially important when the protocol involves third-party observers. Will they observe an entire class period, a segment, an entire day, an entire course or program? Different protocols are used to observe different units of instruction. Because of resource constraints, the observational approach has to be selective, looking at a few activities, events, or phenomena that are crucial to the research questions. A good example of this point is the study of Wecker, Kohnle and Fischer (2007). The research teamchose an observation method which focused on the activities on the computer screen and the users’ spoken comments regarding those activities (see 3.2).
Step 2:Select or develop an observation form
Observation record forms help to standardize the observation process and to ensure that all important items are covered. They also facilitate aggregation of data gathered from various sites or by various investigators. Closed response categories (yes or no answers; check boxes) help to minimize observer variation, and improve the quality of data. If the number of items in a form is limited then the observation process is more manageable. One important question concerning this step is how the prior work form of other scholars can be integrated into one’s own research questions.
Step 3:Specify a sample
Once the forms are ready, the next step is to decide where the observations will be carried out and whether it will be based on one or more schools or classrooms.A single site observation may be justified if the class or school can be treated as a typical case or if it is unique. However, single observations should generally be avoided. As a rule, several schools’ classes are needed to obtain a reasonable understanding of a situation (see Stigler & Gallimore, 2000). In SITES-M2 (Second Information Technology in Education Study - Module 2) for example the countries selected one or more of the primary, lower secondary, or upper secondary levels, with 4 cases for each level (Kozma, 2003). The cases were selected by a national panel for their “innovativeness” and for what could be learned from them about how technology is being used to support educational change.
Step 4:Decide on timing
Timing is a crucial point in the observational process, especially when events are to be observed as they occur. Classes normally follow a fixed sequence. Poor timing can distort findings (see Petko, Waldis, Pauli, & Reusser, 2003). Because most studies observe classes continuously with at least two observers that record their findings independently on a form and/or by the help of video-technology (see Schaumburg, 2001), the probability of missing the key-moments when events occur can be decreased.
Step 5:Conduct the field observation
As previously mentioned, an observation team can provide more comprehensive and reliable data than a single observer (see Schaumburg, 2001). If many classes are to be observed, non-experts can be trained as observers, especially if observation forms are clear, straightforward, and mostly close-ended.
Step 6: Complete forms
The observer should take notes as inconspicuously as possible. The best time for recording is during observation, however, is not always feasible because it may disturb the situation. In such cases, recording should take place as soon as possible after observation. For example, McInerney, Mc Inerney, Lawson, andJacka (1996) used a procedure where the instructor also took the role as an observer and recorded his observations in a weekly tutorial diary (see 3.2).
Step 7: Analyze the data
Observation data from close-ended questions can be analyzed using basic procedures such as univariate and bivariate statistics. Qualitative observations need to be coded in reference to specific categories with respect to the research questions. Finding these aspects and arranging them together in a system of codes is an important process. The “cyclical analytical process” includes viewing, coding, and analyzing the data aimed at transforming the video images into objective and verifiable information (Jacobs, Kawanaka, & Stigler, 1999). Usually the coding process is repeated several times because in viewing the videotaped material new aspects might occur important from the researcher’s perspective (also see Stigler, Gallimore, & Hiebert, 2000; Dirr, 2003; Petko, Waldis, Pauli, & Reusser, 2003; Horsley & Walker, 2003). For the coding itself software such as videograph ( or ATLAS.ti ( can be used.
Step 8: Check for reliability and validity
Observation techniques are susceptible to error and bias that can affect reliability and validity. Errors can be minimized by following some of the procedures suggested, such as checking the representativeness of the sample of schools or classes selected; using close-ended, unambiguous response categories on the observation forms, recording observations promptly, and using teams of observers at each site (see Wecker, Kohnle, & Fischer, 2007).
2.2 Video-andAudio-Supported Observation
In the early 20th-century, one can find the roots of present observation approaches. However, early 20th century approaches are now supported by high-end technical devices. Video was originally used as a stenographic recording method (Stevens, 1910 cited by Stigler, Gallimore, & Hiebert, 2000). The video supported observation was used more than 40 years ago for the first time in the context of educational research. Now this research approachreceives much attention (Leinhardt, & McCormick, 1996; Stigler, Gonzales, Kawanaka, Knoll, & Serrano 1999; Aufschnaiter, 2001; Ulewicz & Beatty, 2001; Petko, Waldis, Pauli & Reusser, 2003; Seidel, Prenzel, & Kobarg, 2005). Recording audio data is another observational method. Depending on the research questions a combined use of videotaping and audio-recording can be useful since multiple types of data enable the data to be triangulated and thereby increase confidence in the conclusions drawn.
A recent approach to recording observational data is to record user activities on a computer screen (Wecker, Kohnle, & Fischer, 2007). This approach can be adequate, for example, for investigating students’ competencies while using a computer. Compared to videotaping teachers’ or students’ behavior, screen recording captures all desktop activities. It can deliver more data about the individuals in detail due to the separately recorded data on each computer. In contrast to the observation with a single video camera this approach can deliver data from every single computer and its user(s). For capturing the activities on a computer screen, software has to be installed like Snapz Pro (MAC) ( or Camtasia and SnagIt (PC) ( If a microphone is installed near the computer the software is able to record the subjects’ spoken comments during their activities with the computer.
The latest approach to gaining data on attitudes is the implementation of Face Reader-Software ( This software automatically recognizes specific properties in facial images, including the following emotional expressions: happy, sad, angry, surprised, scared, disgusted, and neutral. The emotions are represented as bar graphs and as a continuous signal. An additional gauge display analyzes how negative or positive the emotion is. This software is currently not a substitute for a human coder but it might be a good support for precoding video data in respect of specific aspects.A human coder/researcher can select such precoded video sequences and analyze them more deeply in the context of the whole video. The classifications made by the Face Reader can be exported to analytical and database programs for further analyses.
2.3 Advantages and Challenges of Observation Research Techniques
Observational data deliver a much wider spectrum of information, which expands the researcher’s ability to analyze complex human interactions such as those in a classroom setting. When using a video-supported observation approach, the analysis of complex human interactions can be conducted more easily because the recorded video data can be analyzed again and again. Through this set of techniques unanticipated ideas and alternative analytic categories can also be discovered. The same set of video data might provide a focal point for interdisciplinary collaboration. In addition, data gained from video-supported observation allow the integration of qualitative and quantitative methods of analysis (Stigler, Gallimore, & Hiebert, 2000).
When conducting an observation in a classroom there are also challenges (Dirr, 2003). Each observer approaches the classroom with his or her own experiences and biases. As a result, two observers may focus on different aspects of the classroom and thus they may record different phenomena for the same lesson. Also every school environment provides a different context in which the observations take place. This variation can affect the record-keeping behaviors of the observers. One or more observers may have an effect on classes e.g., on the behavior of teachers and students. The observed lessons can vary from the “normal” situation and thus lead to a falsification of the data. Unless the observers and coders are given extensive training, reliability among their observed and rated data can be low.
Classroom observations, data recording, and data analysis are very time-consuming. Therefore the researcher has to carefully consider which data are needed with respect to the research questions.
3. A Synthesis of Empirical Research Results
The U.S. No Child Left Behind Act (The White House, 2001) emphasizes that the use of technology in the classroom can be a tool to improve academic achievement. Such legislation emphasizes the fundamental need to gain more knowledge about how integration of IT in classrooms can be realized. In this respect research on attitudes and competencies toward IT is a major topic for study. Observation techniques may provide a deeper insight into learner attitudes, compared to standardized questionnaires.
In the following section we present the results of an ERIC search of studies on attitudes and competencies toward information technologies that used observational techniques. Our major goal is to provide insight on what kind of observation techniques exist and how they are implemented in studies. Furthermore we will give examples of findings attained by the use of observation measures and advantages that accrued compared to other methodological approaches. We will synthesize the findings of these studies to put together a picture of what we can learn from using observation techniques in understanding teachers’ and students’ attitudes and competencies towards technology.
We followed Whitley’s (1997) approach to locate studies. As a first step we conducted a computerized search the ERIC database from their inception through abstracts available as of June 1994. We conducted the search using the term 'comput* and attitud*', in which '*' is a wild-card character that institutes a search for any word having the designated stem. As a second step the reference lists of prior literature reviews were examined for relevant sources. Finally, we searched the reference lists of articles located by the other means.
The ERIC inquiry results on this issue produced a few results based on the keywords observation/measurement, qualitative research, participant observation, data collection, and social research. The search also produced a small number of results for the keywords attitudes, students and observation. The largest number of hits was reached with the keywords computer-capabilities, attitudes, computer use in combination with a second set of keywords computer attitudes,computer literacy,computer uses in education, that were added to the search by using OR-operators.
3.1 Review of Researches on Attitudes toward Information Technology
Many studies have used the “traditional” observation-method, conducted by one or two persons who joined the observation group and filled out observation forms. This approach of investigating attitudes is still quite common although the influence of new observation technologies is continuingly growing.A more elaborate way to gain data on attitudes is to use a triangulation approach. This means the interpretation of the acquired data through observation methods by involvement of other qualitative data (e.g. interviews) or quantitative data (e.g. questionnaires) collected in parallel. Several studies used mixed methods in order to capture the richness, complexity, and interdependence of events, actions, and conditions in the real classroom. Observation methods were also applied to corroborate interview or survey data. Combining data from more than one source helps to detect possible biases in subjective reports. By using triangulation for the interpretation of data, the probability of getting objective, reliable and valid results is much higher than simply using a single approach.