School Age Readers Evaluating the Quality of Their Textbooks

School age readers evaluating the quality of their textbooks

Larissa Vassilchenko

University of Tartu

E- mail:

Paper presented at the European Conference on Educational Research

Edinburgh, 20-23 September 2000

Abstract

Educational publishing in Estonia has changed very rapidly during recent years. Several factors have an impact on the content of modern textbooks: a rapid growth of the volume of knowledge, a grate changes of information environment and realization of new curriculum.In such situation of transition period, when we have varied range of alternative instructional materials, the assessment of the quality of new textbooks acquires great significance.

In this paper we argue (on the base of results of our study of 1979-1999 years) that pupils of forms 5 -12 can and should participate in the complex evaluation of the quality of their textbooks bifore publishing.

Introduction

The two factors characterizing education today are a rapid growth of the volume of knowledge acquired at school and the process of that knowledge becoming quickly outdated. These factors have an impact on the choice of the content of subjects taught at school as well as on the content of new textbooks applicable in the conditions of a new information situation.

As shown by the results of twenty years of study conducted under the title “Students in Information Flow” (1979-1999), textbook quality is a major factor determining the teacher’s classroom activities and the extent of additional material used. The quality of preparatory texts introducing new topics is of utmost importance in this context. The first encounter with a new topic should be pleasant in order to promote motivation for study and readiness for independent work on additional material. If the introductory text fails to incite interest, the resultant lack of motiovation may be long and damaging.

Therefore, well-grounded assessment of the material to be included in the textbooks has not lost its significance but rather has gained in its ever-growing importance for a number of reasons. Firstly, a thorough assessment of the content of textbooks is dictated by the ever changing content of global education in the information era. Secondly, in recent years there is a tendency of writing alternative competing sets of textbook to teach a subject according to the approved programme. It happens that teachers and students will have to choose a textbook without a thorough knowledge of the content and quality of texts in the textbooks. On the other hand, the authors of textbooks are interested in a prompt feedback from students and teachers with the aim of a rapid publication of a new improved edition of the textbook tested in practice.

The method of assessment

Today there is much talk about a new generation of textbooks reviewed and tested by a multidisciplinary group of experts. This will provide for a good quality expertise on the texts in the textbook. The results published in a clear and accessible manner will make it easier to write textbooks of well-defined content and level of difficulty. It is advisable that the traditional reviews and comments on textbooks written by researchers, experts of teaching methods and teachers should be supplemented with the expression of opinion of those young people for whom the textbooks have been written. Their opinion can provide the authors with valuable information on difficult places in the textbook or questions requiring a thorough explanation in the classroom. The authors will be able to identify places in the textbook requiring improvement or rewriting. New technologies of information processing have made the experimental testing process and assessment of textbooks less time consuming, as the process can take place in the form of round-table discussion attended by representatives of the interested parties: authors of textbooks, academic experts, workers of publishing houses, teachers and students. Today, when most old textbooks are being replaced by new ones, this approach would provide for a new information environment with the minimal waste of time, creative effort and material resources.

We, in our search for study texts assessment methods that could quickly and easily be understood by students of different age groups, decided that the best method for study text quality assessment would be the method of self-evaluation because it is a method of wide applicability and easiness of relevant data collection and processing. In the late 1970s and early 1980s we conducted a series of pedagogical experiments on science texts with the aim of measuring the adequacy of student textbook assessments. The familiarity levels of isolated words (terms) and topics were measured on five-point scale using the following assessment criteria:

“++” “I know it very well, I can explain it at once without much thought.

“+” “I know it well. I can think it over and explain it.

“?” “I have an idea what it may mean, but I am not sure whether I am right

“-“ “I have heard it, but I do not know what it is, I cannot explain it.

“—“ “I hear it for the first time. I do not know it at all.

After we had applied the system of grading used in the Estonian schools (marks given from 1 to 5), we had to change the system of assessment grading and use less emotionally coloured signs, which made the statistical processing of the data a little problematic. We had to effect the change, because the students had associated some of the marks with unpleasant experiences and emotions and they did not want to use grades 1 and 2 (which mean “no pass” in the school system of grading). When they were asked to use signs while assessing the quality of text, we received a different picture, as the emotions of students no longer had impact on the results of the assessment. Our experiment has shown that this system of self-assessment of competence in different fields of knowledge can be understood even by students of the first form of the elementary school. We tested the method for preliminary assessment of vocabulary familiarity on the texts of the topic “Estonia and Estonians” in the new textbooks of the Estonian language for Russian students of elementary schools. A detailed description of the assessment method can be found in the collection of papers titled “Family and Textbooks” (Tartu, 1995). Levels of unbiased attitude in the answers of young experts were assessed in different types of oral interviews and written tests on the content matter of the texts they had assessed. It was found that 80% of the results of self-assessment of knowledge were reflecting the real levels of students’ knowledge. The differences (slightly higher grades than should be given) could be found only in giving the highest grades “+” and “++”, whereas the lowest grades “-“ and “—“ given 100 per cent reflected the level of knowledge. The lowest grades given to the knowledge of the meanings of words are of special value for teachers and authors of textbooks, as they point to the difficult paragraphs in the texts requiring a more careful explanation provided by the teacher and make a purposeful improvement of textbook texts possible.

The experimental work has shown that students of Forms 5-6 can assess the degree of familiarity of 600 key words in the text in 45 minutes. The grades given are different, depending on the student’s attitude towards the subject and his/her learning capacity. However, the vocabulary of the most frequent words in the textbook always contained terms that were considered difficult by most of the students. Therefore, we introduced the mean assessment value as the familiarity index to make rank lists of words from more familiar to less familiar. We were sorry to discover by the end of the school-year that some of the results clearly pointed to shortcomings in the textbooks. About 30% of the key words the authors had presented in bold were of very low familiarity index (in other words assessed with signs “- -“, “-“ or “?” or given grades 1-3). These assessment results show that the material had not been acquired by the students. When the same students were interviewed, the familiarity index did not change, which points to the stability of the index value. We informed the teachers of the results of our preliminary studies. The cause of the relevant terms having not been acquired by the students was too little consolidation of the concepts, as many of the terms of high level of abstractness occurred in the text only once or twice. This was also reflected in the word choice of the teacher who did not pay much attention to the words of rare occurrence in his/her explanations.

Now these textbooks have been abandoned. We can say that the method of assessment has been repeatedly tested and can serve as a reliable tool in testing new textbooks. Students of Forms 5-12 can tell the authors of new textbooks much about the shortcomings of study texts, so that the textbooks can be improved before they are issued for use by large numbers of students. Some authors of science textbooks have calculated the familiarity index of terms to establish how well students know the terms before they are introduced in the textbook. Later on, they used the knowledge to decide upon the new terms and group them into those that are not known and should be introduced and those that can be easily acquired by students.

In the 1990s we decided to expand the use of the method of self-assessment to gain general estimates of the quality of texts in textbooks. We presumed that students had developed a clear and stable opinion as to the quality of any textbook after using it for a year. Readability of texts in ten textbooks of physics, chemistry, biology and geography was assessed by students. Readability of the same textbooks was assessed by scholars who used different methods for the purpose. We found that students can be reliable informants in the assessment of a number of text parameters such as clarity of expression, level of interest, level of expressing significant notions of life, quality of illustrations. Student expertise compared to teacher expertise and results of readability studies proved to be fully reliable. The good textbooks and the poor textbooks could be clearly distinguished. In fact, we had developed a convenient method of monitoring the study information environment in the preparatory period of the school reforms in Estonia when the introduction of new curricula in the 1990s led to the process of replacing most of the old textbooks by new ones.

Findings from experiments

In the 1993/1994 school year, just before the new textbooks were to be released, we could assess the new textbooks for seven fundamental subjects for Forms 7-10. That means that we have collected enough background knowledge for successful comparisons. Another series of studies of the same kind was conducted five years later in the 1998/1999 school year embracing nine subjects for Forms 7-12.

The experiment was in both series conducted parallelly in schools with instruction given either in Estonian or Russian. This allows to draw interesting conclusions about the impact of text quality in textbooks on information behaviour of students as the information environment in schools of the above-mentioned types is significantly different. Another factor at work is the need of schools with Russian as the language of instruction for new textbooks as the old practice of using textbooks published in Russia has been abandoned because of the discrepancy in the curricula. It is not an easy task to write new textbooks for Russian schools in Estonia, as the number of books needed is limited, and new textbooks of many subjects are needed. Therefore, the current situation is to be carefully studied to establish priorities.

The comparison of the results of the two series of study clearly points to the fact that the new textbooks are better illustrated and contain more material of practical value. Yet the textbooks have not become more interesting. This is, most probably, related to the general concept of the subject. Our previous studies on alternative textbooks by different authors aimed at meeting the requirements of the same programme (three textbooks of physics and three textbooks of chemistry) show similar results as to the level of abstractness and complexity of texts and the frequency of occurrence of the vocabulary.

It is worth noting in this context that the mean value of interest offered by the textbooks was uniformly lower than the values for clarity of expression, practicality and illustrativeness. This can be partially due to the lack of interest of the authors, celebrated scientists in their respective subjects, in motivating the reader to study the textbook. The authors first and foremost pay attention to the clear and logical presentation of the subject; they seldom think of the user of the textbook, a student, for whom the text in the textbook may be the first academic text he/she ever reads.

A look at the graph in Figure 1, reflecting the perception of a text in a physics textbook by boys and girls, will suffice to understand that the textbook has been written for boys by a man (See Figure 1).

Figure 1. Differences between the indexes of comprehensibility of the physics textbooks for boys and girls.

The male orientation dominating in science textbooks has been mentioned by many scholars from various countries. Among the textbooks, physics textbooks are of special interest, as textbooks of physics regularly present different levels of interest and knowledge of the subject by boys and by girls. Physics textbooks are thought to be clearly male oriented. The teacher of physics is expected to pay much more attention to explaining the notions to the girls than to the boys. The authors of physics textbooks should take this finding into consideration in writing textbooks by including more varied practical examples and additional explanations. Our studies have shown that the interest of boys and girls in studying physics differs significantly, the difference remaining stable all through the school-years. The results of the study are given in Figure 2, they are reflecting the opinion of students of Form 8. (See Figure 2).

Figure 2. Distribution of answers to the question "Your interest to this subject"

( physics , 223 girls, 209 boys ).

These differences have deep-rooted causes; they are related to the traditions in educating children, the roles of men and women in the family and in the labour market. These differences of interests and aspirations are practically not reflected in the physics textbooks used in schools of Estonia. Girls suffer more. Their first encounter with the subject of physics can be very demanding. The estimates given by girls on the clarity of expression and interest of the texts in the textbook for Form 8, the first textbook of physics, allow to conclude that their active independent search for new additional information on the topics of physics presented in the textbook will be minimal. Our studies have constantly shown that girls prefer to acquire their knowledge of physics exclusively from the explanations of the teacher.

The studies on chemistry textbooks have revealed how differently a text is understood by strong and poor students. The results of our study presented in Figure 3 suggest that a teacher of chemistry will have to work hard to develop an individual approach to different groups of students.

Figure 3. Differences in a text understanding for "strong" (a) and "poor" (b) students (chemistry).

As the study has revealed big differences in the comprehension levels, the question arises for whom the textbook has been written. Who actually is the student who is denoted by the abstract term of the “average student”? This question is difficult to answer. A possible solution of the problem would be the use of several alternative textbooks of graded difficulty levels to cover the topics in the programme in the more polarized subjects. It is clear, however, that a new textbook is to be written to present the material required by the programme in a way that would be easily understood and eagerly studied by most of the students.

Textbooks and other sources of information

Today, we cannot say that all the new textbooks meet the above-mentioned requirements. If the texts in the textbook are difficult to understand, the teacher’s explanation of the topic will be important. The role of independent cognitive study will be minimal, which is a dangerous tendency under the conditions of the era of information. Some students may rely exclusively on the teacher’s explanation, shunning independent work upon an academic text. Figure 4 presents the relationship between text quality and the role of the teacher’s explanation in nine fundamental subjects of Form 9.

Figure 4. The relationship between text quality and the role of the teacher's explanation in nine subjects (n=1619).

A careful look at the graph shows a mirror symmetry pointing to an inverse relationship: the more difficult the text is, the more explanation given by the teacher is required. An important conclusion offers itself – a poor textbook not only dampens the interest in the topic and independent search for additional knowledge but it also restricts the creative endeavours of the teacher. The teacher will have to conduct long monologues trying to interpret the texts in the textbook into a language his/her students can understand. There will be no room for the teacher to do any research, discuss unsolved questions and debate points of theory.

The same kind of dependence was found in teaching languages. It was discovered that there were significant differences in the use of reference books and other sources of information by Estonian and Russian students in their studies of English. With the out-of –the-classroom information sources being identical, Estonian and Russian students used different textbooks of English. Teachers of Russian schools felt that their Estonian colleagues were in a better position to teach English, as the textbooks they used were considered to be of better quality. A questionnaire conducted among the students fully confirmed the feeling. In Russian schools teachers of English spent twice as much time on explaining things as did their colleagues in Estonian schools.