MUSE Evaluation Plan (WP5)
V4
September 2003
Angela Little and Pat Pridmore
List of Contents
1.0Introduction
2.0The evaluation style
3.0The evaluators
4.0The evaluation plan
5.0Methods of collection of evidence
5.1Teacher-centred training materials
5.2School case study: BASELINE
5.3Teachers’ engagement with the new model of In Service Training
5.4Teachers’ ability to design and implement cross-curricular teaching plans
5.5Teachers’ attitudes to the new model of multigrade pedagogy
5.6Students’ attitudes to the new pedagogy
5.7Implementation and Impact of the training programme
The Muse Project
Evaluation plan to assess the impact of the proposed training programme
V4
Angela Little and Pat Pridmore
1.0 Introduction
The overall aim of the MUSE project is to develop and evaluate ‘an innovative training programme for the teachers in the multigrade schools’ (project doc p.25) through open and distance learning (ODL).
The training programme has three foci:
Methodological approaches to multigrade teaching and practice
The use of ICT in the classroom
The design of cross curricula applications and projects (based mainly on the use of ICT)
According to the project document (page 42) the purpose of the evaluation plan/scheme is to assess the impact of the training programme.
At the team workshop held in Rhodes in June 2003 it was decided that the training programme would be modular and that teachers would select those modules that suited their needs.
To avoid confusion we need to distinguish between the two levels of ‘target’ groups for the training. The first are the teachers who will engage with training packages with the above 3 foci. The second are the students in the classroom who will engage with the applications and projects mediated by the teacher. For the purpose of evaluation and WP5 our main attention will be the teacher.
In each country teachers will work very closely with partner institutes – the University of the Aegean (UOA), Greece; the University of Cadiz (UC) Spain and the Chydenius Institute (CI), Finland.
2.0 The evaluation style
The chosen style for the evaluation is ‘participatory’ and ‘user-centred’. This implies a qualitative and continuous approach, supplemented with quantitative information where appropriate. A participatory style is inclusive and in tune with the teacher-centred pedagogical approach of the project, the ‘teacher centred approach’. Educators from the partner institutions will work alongside the teachers to jointly develop and evaluate the training programme.
‘Teacher trainers’ (i.e. partner institute staff) and their ‘trainees’ (i.e. partner school staff) will be involved throughout the planning and evaluation cycle for the training materials. The role of the teacher trainers and their trainees is to work in partnership with the project team to design and adapt the methodology, generate and analyse data, share findings and link them to action. This can be a very empowering process for those involved since it shares power, helps develop teachers’ research skills and demonstrates that their views count. (See IDS 1998).
The evaluation style will use a judicious mix and sequence of approaches to assess the impact of the project on the quality of teaching and learning in the multigrade classroom.
3.0The evaluators
Three sets of partner institutions and people will play key roles in the evaluation process. These are
Schools and school teachers
The schools are:
- Salakos primary school, Rhodes
- Vintturin-Tastulan primary school, Finland
- Veikko Vionoja primary school, Finland
- C.P.R. Fields of Tarifa, Spain
The school teachers are:
In Salakos, Greece: Dimitris Zorzos, Theodora Georgovasili
In Vintturin-Tastulan, Finland: Leena Harju
In Veikko Vionoja, Finland: Reeta Puskala, Sari Van Schaik, Mauri Niemisto, Maila Koivumaki, Tuula Kellokoski
In C.P.R Fields of Tarifa, Spain: Manuel Quilez Serrano, Juan Baquero Perez
Teacher educators in the local support institutions: UOA, EA, CI, UCA
The partner responsible for the evaluation model: EID, London
4.0The evaluation plan
As mentioned above, the purpose of the evaluation plan is to assess the impact of the in-service training programme with 10 teachers. The evaluation is intended to be small scale and in depth. Although the small number means that the findings from the ‘pilot’ project cannot be generalised to the wider population of teachers, the evaluation style will illuminate the current situation and provide rich insights into the potential impact of the training programme. In the original project proposal (page 42) the evaluation approach had the following features.
It would be based on:
-the development of specific training objectives for the various modules of the training programme
-systematic data gathering throughout all cycles of the action research implementation phase
-a systematic reporting system
-a mixed methods approach combining both qualitative and quantitative data.
The impact of the in-service training programme would be evaluated by assessing:
-teacher’s engagement with the new model of in-service training
-teacher’s ability to design and implement cross-curricular teaching plans, projects and activities that are relevant to their specific school environment and use ICT
-teachers’ attitudes to the new model of multigrade pedagogy.
-students’ achievement using the new model of multigrade pedagogy in the classroom.
During the Rhodes workshop (June 2003) we evaluated progress to date and modified the above approach as follows.
(i)the timing of teacher’s engagement with training materials. It became clear during discussions that if the teacher-centred approach was to be adopted in a meaningful way then the relationship between WP 2 (Teachers needs analysis) and WP 3 (development of the training programme) required more attention. Five of the partner institutions had undertaken teacher needs analysis – but the implemented timetable did not permit for a meta-analysis across country contexts to be completed prior to the completion of the first draft of the training packages. The structure and content of the training packages is being derived from documentation available in the public domain and from the experience and ideas of staff of the partner institutions, based on their experience. The views of those who would implement the training packages had not yet been sought systematically. It was decided that partners would be invited to comment immediately on the draft structure and content of the training packages. A workplan was developed for this (Evaluation Plan V3) and subsequently implemented.
(ii)reporting system: Given the focus of the project on the use of ICT it was agreed that as much of the reporting system as possible should be done digitally. It was also revealed that very few, if any, of the partner institution staff and partner school staff had logged on to the MUSE website and its shared working space. EA had already taken action to encourage this activity. More action and encouragement was needed in the future. The implementation of the project depends on the effective use of digital technology not only by the teachers and students in the partner schools but also by the staff in the partner institutions.
ACTION: All partners, with continued support from EA
(iii)student achievement. The main objective of this project is training for teachers in 4 schools. The planned periods for implementation of the training in the classrooms are limited. The original project document envisages two periods of school centred work – phase A 10-13th month and phase B 17-20th month. Given these conditions it is now considered to be unrealistic to undertake systematic assessments of children’s learning. Inter alia this would have involved pre and post assessments of children’s learning, the development of assessment tools in relation to curriculum objectives (for materials not yet developed) and comparisons with teachers and multigrade schools not participating in the project. These activities are too ambitious for the current MUSE project. They could be considered as activities within an extension project that implemented a training programme on a larger scale.
Although the impact of teacher training on students’ achievement will not be assessed, children’s views and reaction to the ICT based curriculum applications will be sought.
5.0Methods of collection of evaluation evidence
5.1 Teacher-centred training material
The teacher-centred nature of the initial draft of the training materials will be assessed through digitally recorded responses to the project web-based draft material undertaken jointly by the teacher and the local partner institute).
A detailed workplan was developed to encourage teacher participation in the initial design of the training materials. This was presented in V3 of the evaluation plan, dated July 2003.
By September 2003 teachers in all partner schools had commented on the draft structure and content of training. So too had the two members of EID and members of each of the partner institutions. All comments had been processed by MO (EA) and substantial changes made to the training materials. The materials presented to the MUSE team in September had been substantially modified. (a collection of all comments and contributions to this process is available form M0/EA on request).
5.2 School case study: BASELINE
Baseline evidence is needed from each school before the training programme commences, or as near to its inception as possible. A baseline school case study should be undertaken jointly by the partner institutes and the school teachers. It is suggested that evidence is collected from the school and the classroom level as follows:
Checklist of information required for Baseline school case study
N.B. Some of the information listed below will be available already to colleagues in Spain and Finland. If so, colleagues do need to collect it again. However, it needs to be incorporated a school case study ‘portfolio’ of information to be sent to the EID evaluators.
The checklist seeks evidence at two levels – (i) the school, and (ii) the classroom within the school. The following lists are for guidance only. They should be amended and supplemented to reflect the specific conditions of multigrade teaching in the three countries.
1. School Level
1.0Background
1.1Name of school
1.2Name of community
1.3Community characteristics (population, economic activity, location of capital, location of nearest town, location of post primary schools)
1.4Financing the school. Who pays? (national government, local government, parents) Fees? Books? Uniforms? Meals?
2.0 Human Resources
2.1 No of teachers
2.2 Residence of teachers
2.2 No of pupils
2.3 Age/gender/grade of pupils
2.4 Kindergarten attached to school?
3.0 Physical Resources
3.1 Buildings
3.2 Space (inside/outside)
3.3 Equipment
3.4 ICT
teachers’ skills
computers per pupil
software
pupils’ skills
teachers’ experience of using ICT for cross-curricular projects
4.0 Curriculum, Grades and Subjects
4.1 The national curriculum
subjects
graded/ungraded
philosophy
curriculum reforms
recommended methods
weekly/annual study hours by subject and grade
recommended grade combinations in 1,2 and 3 teacher schools
recommended subjects to be taught as graded and in combined grades
are some subjects designed for a 2 or 3 year cycle
subject teaching hours in 1,2 and 3 teacher schools
4.2The weekly/annual timetable of the case study school
5.0 Teacher attitudes to multigrade teaching
What are the case study teachers’ perceptions of the opportunities presented by multigrade schools and classes?
What are the case study teachers’ perception of the challenges presented by multigrade schools and classes?
2. Classroom Level
Observe at least one lesson (without the use of ICT), for each participating teacher. Observe at least one lesson (with the use of ICT), for each participating teacher.
How/what to observe
1. Basic information. Teacher name, date/time subject(s), lesson(s), period/length, number of teaching groups, grade combinations(if appropriate)
- Classroom layout. For each class observed note the layout of the class, the furniture, the pupils, black/white boards, display areas, equipment, windows, doors etc (see example from Salakos school presented in Annex 1).
Observe and note down where pupils sit/work and note, in particular,
the type of group in which they sit
e.g. pupils sit individually, not in groups,
pupils sit in pairs,
pupils sit randomly in small group,
pupils sit in grade group,
pupils sit in cross grade ability group,
pupils sit in interest group
pupils sit in friendship group,
pupils sit in same sex group
pupils sit in mixed sex group
other
the type of group in which they work/learn
e.g. pupils sit in groups, work individually,
pupils sit in groups, work in pairs,
pupils sit in groups work as a group
other
- Teacher activity-student activity
There are many ways of observing lessons. Some evaluators sample teacher behaviour, some student behaviour. Some sample time; others events. Since the focus of the MUSE project is on teacher practices and teacher training we suggest that the observation of the classes should focus on the teacher and his/her behaviour. We suggest that the time of lesson start be noted. Thereafter the time should be noted at each moment when the teacher significantly changes his/her activity. If the class is divided into clearly separable teaching groups then a column should be created for each group (See Annex 2 for an example). Because pupil activities are linked with teacher activities, the pupil activity should also be noted.
The points of pedagogic significance that observers will wish to focus will vary from country to country, dependent on the philosophy, structure and content of the respective National Curricula. However, the following checklist may be useful:
Is the lesson teacher centred/directed?
Is the lesson pupil centred/directed?
Does the teacher manage the time well?
During how much of the time is the teacher talking out loud?
During how much time is the teacher talking to the whole class/a group/individual pupils?
Did the teacher prepare a lesson plan in advance of the lesson?
How much autonomy do pupils have?
Are the pupils enthusiastic and actively engaged in their learning?
How much time is spent by all pupils ‘on task’?
Do pupils behave respectfully to each other and the teacher?
Do pupils help each other?
How is pupil learning assessed?
Does the lesson start and finish on time?
What levels of talk are acceptable to the teacher?
What levels of noise are acceptable to the teacher?
- Baseline assessment of each teacher in relation to MUSE training objectives
The observer should assess the profile of teacher skills in relation to the learning objectives of the MUSE training programme. The skills are attached in Annex 3. An example of this assessment for Salakos school is presented in Section 7 of the Salakos Baseline case study.
5.3 Teachers’ engagement with the new model of in-service training
Evidence of the teachers’ engagement with the new model of teacher training (three main packages) will be collected through
- The school teacher ‘trainees’ are expected to monitor and self evaluate their progress with the module. These will be kept digitally. Teachers will be encouraged to keep a log of dates and times when they work on the materials and when they receive support from the trainer. Their views on the quality of the materials (user-friendliness, attractiveness, the activities given, the readings required, usefulness of the new pedagogic model being taught) and of the quality of the support given (dates and times of each interaction with trainer, nature of the interaction, friendliness, usefulness of advice given), factors that help or hinder engagement. To assist this process teachers will be requested to complete an evaluation format at the end of each module or sub-module (this will be dependent on the structure of the modules). Teachers/partner institutes are expected to report back to MO(EA) every 15 days after the start of the training. A draft reporting format is presented in 4. MO will modify prior to sending training module content.
- Trainers will also keep records of dates and times of each interaction with trainer, nature of the interaction, advice given, comments on Progress etc. These will be kept digitally. MO (EA) will integrate the format into the training materials website.
- Content analysis of interactions between trainers and trainees. This will be done by EID on the basis of 1 and 2, to the extent possible/feasible. It is acknowledged that much of the interaction between trainee/trainer will be undertaken in Greek, Spanish and Finnish. This analysis may need to be done through interviews in English between EID and partners on the occasion of the next team workshop.
5.4 Teachers’ ability to design and implement cross-curricular teaching plans
Evidence to assess teacher’s ability to design and implement cross-curricular teaching plans, projects and activities that are relevant to their specific school environment and use ICT will be collected through
- Analysis of the lesson plans, including the quality of the activities and projects designed, and use of ICT.
- Classroom observation of the organisation and management of the teaching (direct whole-class, direct one grade with holding activities for other grades, individual/independent/small group learning, time on task).
- Observation of other aspects of classroom life (classroom atmosphere, familiarity with the use of ICT, etc)
- Semi-structured interviews and focus group discussion with trainers and teachers on what went well, and less well, during implementation and how the lesson plan can be modified.
- Focus group discussions with students on their views of the new pedagogic model.
- The selection and demonstration by the teacher and the local partner institution of ‘best practice’. This will occur towards the end of the implementation period.
ACTION by EID in due course (dependent on progress within WP 4