Evaluating Adult Literacy Work

7808

Evaluating adult literacy work

Alan Charnley and Professor H. A. Jones

National Institute of Adult Education and University of Leicester

These two papers include extracts from A. H. Charnley and H. A. Jones, The Concept of Success in Adult Literacy by permission of the publisher, Huntington Press, Cambridge.”

Part I - Criteria of Evaluation

Professor Jones

A. Evaluating literacy work raises a number of immediate difficulties:

(i) Such evaluation is customarily carried out in schools and is there capable of pretty exact, normative measurement. Reading ages have some meaning when directly comparable with chronological age because they measure progress made, not only against average progress, but also against further progress that may be expected towards a further norm. But a ‘reading age of 8’ means nothing when applied to a man of 38: not only does it offer an invalid comparison because a man of 38 is in no way comparable with a child of 8: but it also can say nothing about desirable further progress. (Would we expect him to have attained a reading-age of 16 by the time he is 76?).

Yet many teachers of adult sub-literates carry ‘reading-ages’ in their heads as subjective rule-of-thumb measurements of their students, simply because they know of no other.

(ii) There is more general utility in attempts at evaluation of standards of progress based on the concept of ‘competencies’ (Kohl, Holmes and Stringer)[1]. They do not claim to be exact but to assess broad stages of advancing competence in various identified skills and then to evaluate the degree of confidence with which each skill is tackled at that level. See Appendix I attached (Kohl pp 126-27)

For the teacher wanting to monitor a student’s progress or to diagnose his emerging requirements, such a scheme is clearly useful, though in practice it has been found complicated and slow.

(iii) Much of the difficulty lies in the term ‘literacy’ itself. It appears to describe a state to which anyone may aspire and which, once attained, will remain static - an accomplishment of the same order as learning to swim or to ride a bicycle. Politicians’ statements at the start of the literacy campaign appeared to be based on this assumption: illiteracy was something to be ‘eradicated’, like bindweed from the garden or nits from the hair; and there could be a time-scale for doing so. The Adult Literacy Resource Agency was set up for one year in the first instance and subsequently extended for a further two.

If that view were correct, meaningful questions could be framed about the percentage of the population who are illiterate, or sub-literate, and about the rate of progress towards eradication. (One can imagine the Education Office decked with a huge thermometer, as for charitable appeals, showing today’s total rescued from illiteracy). Moreover, once the threshold of literacy had been defined, every teacher could measure his student’s progress towards it with confidence.

B. Alas, that happy state is not. Kohl’s scheme suggests as much; and a moment’s thought confirms it. Who amongst us is absolutely literate? May not the most literate be defeated by an official form, a bureaucratic regulation, a legal formulation, or an article in the Journal of Progressive Semiotics? Then too, are we not in a Heracleitean situation of eternal flux, in that the standards of competence required in a changing world are constantly rising? Is there indeed any escape from a solipsist view that ‘literacy’ can be defined only in the terms of each separate individual?

One line of escape is to assert that there are certain basic skills without which further progress in learning is not possible: literacy can then be equated with the possession of these and anything beyond is ‘post-literacy’. But in practice two consequences have seemed to flow from this quite popular line of thought. The first is that the only available system of analysing skills progressively, and thus distinguishing what is basic, is that used for children’s reading; so we are back to reading-ages of 7, or 9, as the basic threshold. Experience shows that adult problems are not those of children. Adults have much more difficulty with the reading and writing of the little words, the small cogs and pinions of syntax like ‘which’ or ‘although’ or ‘he-ought-to-have-been-told’ or ‘he-was-having-to-be-helped’ than with apparently harder words like ‘telephone’ or 'public bar’ or ‘redundancy’.

A second consequence of thinking in these basic terms has been a heavy concentration of phonics in the teaching. Many tutors, especially volunteers working with a single student (and including, I am afraid, many trained teachers), have felt that phonic work gave a sound, unequivocal base of skill which could be imparted in an objective, ‘academic’ way (that is, with the authority of a formulated school discipline behind it) and which it was then up to the student to apply as he chose. Again, however, experience has shown that adults make but slow progress by these methods and find themselves usually quite unable to take the crucial step from learning-with-teacher to independent use in the real world.

A different line of escape from the dilemma has been adopted in an ingenious and influential pamphlet from the Advisory Panel of ALRA. It is called An Approach to Functional Literacy[2], borrowing the term from UNESCO but giving it a different connotation. The pamphlet is a manual for tutors, but behind it lies a vision of the student functioning, with the aid of his reading and writing, in a world whose reality is determined by his own needs and purposes. Thus the relevant skills are applied to tasks, and it is the student who must select the task. A suggested list of tasks begins:

applying to join a club/association

backing a horse

choosing and booking a holiday

claiming FIS and other benefits

filling in pools coupons

looking up a street in A-Z

It continues with 27 further suggestions, inviting tutor and student to go on and add more of their own. The component skills of reading and writing are then set out as a repertoire from which selections can be made for the task in hand; and for all the tasks there is an overt outcome in action by which the student can evaluate his learning for himself.

C. We are now close to the world of behavioural objectives, so beloved of curriculum evaluators a few years ago but now somewhat under question. The persuasive work of Mager (1962) and Gagne (1966,1967)[3], deriving partly from Bloom’s taxonomic approach and partly from programmed learning, had a great influence on early training schemes for adult-education tutors: if teachers of craft skills could be trained to specify step by step the objectives their students were expected to reach, clear evaluation of teaching and learning could be made. But, as Elsdon (1975)[4] has pointed out, the areas of adult education in which this could be done are limited and may not be among the most important. Nevertheless, the ‘functional’ approach to literacy teaching outlined above makes a valid and practical use of the concept of defined instructional objectives as a basis of evaluation.

D. In the evaluation of technical courses, the use of precise instructional objectives has been of much value, not only towards assessing the learning of individual students and the efficacy of course arrangements, but also towards the design and construction of the course itself. Each component skill can be judged against desired performance in actual work conditions or the equivalent, and there are thus external bench-marks for the evaluation of the course. Thus Vilensky and Fraser (1977)[5] list 54 skills to be learnt by students on a hairdressers’ course; they compare the timing of each as performed by the trainees with the timing considered desirable by their teachers; and so, using Stake’s congruency[6] and Provus’ discrepancy[7] concepts, they feel able to offer an assessment of the course as a whole.

Many studies of this kind have been carried out[8] both formally and informally, and courses have no doubt been modified in consequence. The studies have followed the line of Gagné who would reduce all elements in a course, whether of content or of method, to the specification of objectives:

Possibly the most fundamental reason of all for the central importance of defining educational objectives is that such definition makes possible the basic distinction between content and method. It is the defining of objectives that brings an essential clarity into the area of curriculum design ... the kind of clarification that results by defining content as ‘descriptions of the expected capabilities of students’ ... Once objectives have been defined, there is no step in curriculum design that can legitimately be entitled ‘selecting content’.[9]

Yet already in 1971 a student of mine, J. H. Chatfield, was examining from this point of view the City and Guilds course 293, Workshop Technology, and assessing each item of the syllabus according to the uses made of it by technicians in engineering factories associated with the motor industry. The actual use of many items of the syllabus was limited; the potential use was not much greater; and the argument therefore for slimming down the syllabus was strong. But the technicians interviewed asserted almost unanimously that all items of the syllabus should be retained because together they constituted an important and coherent body of background knowledge. In other words, in their severely practical view, specific behavioural objectives could relate to only a part of a total desirable course.

It is at this point that the work of Elliot W. Eisner[10] becomes instructive. Behavioural objectives are used, he said, in ‘a predictive model of curriculum development’. But what about those forms of education in which the outcome cannot be predicted as behaviour? How do you specify the behavioural objectives of studying say, Paradise Lost, or the Crusades, or woodcarving? In answer, Eisner (1969) advances the idea of ‘expressive objectives’ which are unpredictable but which may be expected to result from what he calls ‘educational encounters’. An expressive objective is ‘evocative rather than prescriptive’:

The expressive objective is intended to serve as a theme around which skills and understanding learned earlier can be brought to bear, but through which those skills and understandings can be expanded, elaborated, and made idiosyncratic. With an expressive objective what is desired is not homogeneity of response among students but diversity. [11]

E. How then do we approach the evaluation of adult literacy tuition? Two major pieces of research in the last couple of years have had to attempt answers to this question. The NIAE/DES research project was described at last year’s SCUTREA conference and the conference papers include an account of the problems and constraints as then perceived. The report of that research now awaits publication and it is possible to refer briefly to some of its findings. The second is a study carried out at the University of Leicester and is described by Dr. Charnley in Part II of this paper.

For reasons set out in last year’s paper and needing no further rehearsal here, both researches had to eschew a quantitative approach and work out an appropriate qualitative methodology. But one of the important factors was the issue of confidentiality, and this bears directly on this question of modes of evaluation.

In one sense, adult literacy tuition is predictive, in that the teacher expects the student eventually to be ‘able to read’: but to read what, with what purpose, when, with what depth of understanding, and with what result? There is no linear progress here, such as could be assessed by advancing series of objective tests. Each piece of reading or writing is a domain of its own, defined by the student’s purposes; and those purposes derive from his status as an autonomous adult, exercising will and judgement within the context of his own life and experience. Only a child can learn to read; an adult learns to read something and to some end.

Alan Champion (1975)[12] offers ‘intentions’ instead of the more usual ‘need/wants’ as a description of what we should be looking and responding to in our relations with students. This is sharply applicable to the literacy student as the two researches have, somewhat surprisingly, disclosed. Both studies have concluded that progress in literacy is closely bound up with the student’s self-image, and that the growth of confidence - not simply confidence in one’s ability to learn, though that is part of it, but rather confidence in the assertion of one’s intentions and in the contemplation of the self making that assertion - is preliminary to any progress in reading or writing skills. This is why the confidentiality of the transactions with students had always to be respected for as long as he believed it to be important, and why the research methods adopted, whereby the researcher became accepted as part of the transactions, yielded such rich material about the perceptions of students and tutors, some of which Dr. Charnley summarises below.

It was clear that successful participation in literacy tuition indeed very often the simple decision to carry on with the tuition at all - depended less on choice of reading schemes or methods of teaching than on the relationship that the student perceived. Many ascribed their failure at school to a feeling that they had languished unnoticed at the back of the class: as long as they gave no trouble they were left alone. What spurred them now was the discovery of an adult who would take notice of them and their intentions. Where tutor and student had common interests, or where a student’s special knowledge or interest was used by the tutor as part of their shared work, momentum picked up and horizons widened.

This is very close to what Eisner calls ‘educational encounters’ and if we want a theory of evaluation for adult literacy tuition I would suggest that it lies along those lines. The literacy student needs to find ‘a theme around which skills and understanding learned earlier can be brought to bear but ... expanded, elaborated, and made idiosyncratic’, or expressive objectives. The AURA group’s manual shows what use can be made of specific instructional objectives in designing literacy work for adults but only as a proximate means. It is the student’s intention, revealed in his choice of tasks, that gives the reality, the drive, and the perception of the ends. The over-riding objectives therefore lie in Eisner’s expressive domain: there is no predicting where the encounter might lead if it has the right quality and is not trammelled by instrumental objectives imported by the tutor.

It is worth adding that the researches also revealed an immense and salutary amount of learning by the volunteer tutors - about the lives of some of their fellow-men, about the education system and its outcome, and about themselves and their thresholds of tolerance and understanding. They too found in these close encounters of an educational kind the achievement of powerful expressive objectives.

Part II: Concepts of Success

Alan Charnley

Part I of this paper has already shown how the NIAE research scheme was influenced by the expectation of receiving highly sensitive adult literacy students and consequently how steps were taken to safeguard confidentiality, namely that students would be contacted only by permission of their tutor and tutors only by agreement with organisers. Because of this facet of the research no statistical procedures for random sampling were possible.

The Leicester University research described in this part of the paper accepted these national constraints as the background consensus of opinion.

Early in 1975 a questionnaire was sent to tutors and structured questionnaires were read out to a pilot group of students. The number of tutor returns was disappointing, (this was the national experience) and the students clearly objected to answering structured questions. They had a horror of questions which reminded them again of school experiences of failure and which they considered did not ask them to relate their individual personal experience.

It was clear that there was a risk of large number of drop-outs. A more imaginative approach was required creating a relaxed atmosphere in which the students and tutors were able to reveal their perceptions and perspectives. Therefore, tape recordings of structured conversations with tutors and students were gathered and the evidence was derived from these recordings.

A questionnaire was issued to tutors in 1975, suggesting various criteria of success. The tutors were asked to place these in order of importance and to add further criteria. Of course, some tutors recorded their own perceptions, others their perceptions of their students’ priorities and others considered the criteria on both bases. From this evidence, a rank order of groups of criteria emerged which could be regarded as a first operational hypothesis:

FIG A

(a) Affective personal achievements)

(b) Cognitive achievements)

(d) Socio-economic achievements)

(e) Affective social achievements)

But the perceived criteria that emerged from the tape-recorded evidence of two interviews separated by a period of about 9 - 12 months with 49 tutors and 35 students were, in order: