The ELT INSETT PROJECT
British Council Azerbaijan
Page 1
Azerbaijan ELT INSETT Project Review
British Council ELT INSETT PROJECT Azerbaijan
A consultative review of progress and policy
Report prepared by:Dr C Tribble
Commissioned by: Brian Maguire, English Language Projects Manager
British Council Azerbaijan
Submitted:10 February 2006
Executive summary
A consultative review of the British Council ELT INSETT project has been concluded following a document review process carried out in January 2006 and three day visit to Baku in the period February 6-8.
The overall assessment of progress to date is positive, with a strong team of INSETT trainers now established, an appropriate and highly appreciated INSETT course developed and piloted, and two cohorts of English language teachers (with the potential for delivering local support to colleagues, having completed or in process of completing the course. (see the discussion on under Findings Section 4 on pages 6 - 9).
The primary concern of the project team is the extent to which it will be possible to consolidate the positive outcomes of the first part of the project. Such consolidation will, ideally, require cooperation with an existing official ELT INSETT structure which is able to deploy a network of existing trainers able to learn from the INSETT Project Team and to deliver the ELT INSETT programme through local or central institutional frameworks. However, it is far from clear that such a network exists or is operationally effective in the case that it does appear to have official existence. Under these circumstances, the report concludes that it will be necessary to investigate two possible approaches to ensuring that the ELT INSETT project closes with the maximum possible impact and sustainability. The first will involve identifying and assessing the capacity of official ELT INSETT institutions. If it proves impossible to find such a partner, the second approach will depend on the strengthening of the capacity of NGO sector organisations to reach as many practicing teachers as possible.
Main recommendations from this report, focus on the completion of the evaluation progress and achievements in Phase 2, and the preparation for a possible Phase 3. These are given below:
Recommendation 1 / PHASE 2 EVALUATION The ELT INSETT Project Manager should work closely with the ELT INSETT team to finalise and implement an evaluation plan for Phase 2 of the project. Results of this evaluation process should be available at the end of the funding period for Phase 2 and should inform the final design of Phase 3 (including the ELT INSETT training programme design)Recommendation 2 / POLICY The British Council and the Project Board should seek immediate clarification from the Ministry of Education on the issue of which organisations have the authority and responsibility to provide ELT INSETT in the regions.
Recommendation 3 / POLICY The British Council should seek to ensure that any initiatives within Phase 3 harmonise with processes funded through the World Bank Education Reform project
Recommendation 4 / CAPACITY The British Council, in cooperation with the Ministry of Education and the Azerbaijan English Teachers Association, should take immediate steps to assess the extent to which national and local state structures in Azerbaijan are actually delivering ELT INSETT courses to teachers in the regions, and who is delivering this training. This may involve parallel surveys one carried out by AzINSETT, the other conducted by AzETA. This triangulation of data would be valuable.
Recommendation 5 / PROJECT DESIGN If it is possible to clarify these issues, a project design for Phase 3 should be established along the lines set out in the main recommendations (Section 7)
Table of contents
PREPARATION
1.Comment on the Terms of Reference
2.Approach
3.Main activities during the consultancy
3.1Document review
3.2Stakeholder interviews
3.3Evaluation workshop
FINDINGS
4.The ELT INSETT PROJECT: Phases 1 and 2
4.1Progress to date
4.2A re-statement of the ELT INSETT Project structure
4.3Outcomes and outputs
4.4Stakeholder relations
4.5Comments on the present project structure
4.5.1Present status of the ELT INSETT training team
4.5.2ELT INSETT Provision delivered through state structures
4.5.3Delivering a cascade.
5.A Phase 3 for the ELT INSETT Project?
5.1Azerbaijan Teachers' Institute AzINSETT (060206a AzINSETT.doc)
5.1.1Current provision
5.1.2Capacity to support ELT INSETT
5.1.3Assessment of potential and risks
5.1.4Interim conclusion
5.2University of Languages (UoL)
5.2.1Current provision
5.2.2Capacity to support ELT INSETT
5.2.3Assessment of potential and risks
5.2.4Interim conclusion
5.3Azerbaijan English Teachers' Association
5.3.1Capacity to support an ELT INSETT mini-baseline
5.3.2Interim conclusion
5.4Ministry of Education
5.4.1Current policy on ELT INSETT
5.4.2Interim conclusion
5.5A possible way forward
6.Acknowledgements
7.Conclusions and recommendations (draft!)
APPENDICES
1.Azerbaijan ELT INSETT PROJECT - Documents.doc
2._INSETT Project Evaluation NOTES.doc
3.ELT INSETT Evaluation Schedule + outcomes.doc
4.List of meeting notes:
5.ELT INSETT Evaluation Workshop Notes.doc
PREPARATION
1.Comment on the Terms of Reference
The initial terms of reference for this consultancy focused on two major purposes: the overall evaluation of the ELT INSETT project, and the pressing need to ensure clarity around the nature and feasibility of a proposed Phase 3. Following discussions with Brian Maguire, the EL Projects Manager for Azerbaijan and evaluation commissioner, we agreed that the primary focus of the evaluation would be on the feasibility of a possible Phase 3, and that a formal evaluation of the outcomes of Phase 2 would be best undertaken by the present ELT INSETT project team with the support of the British Council. This would ensure that the evaluation would be stakeholder driven, maximise learning, and provide a professional development for this group. It would also make best use of the time I had in Baku to work with the project managers on a review of policy for ELT INSETT.
The finalised terms of reference specified that the desired outcomes of the review process would be:
- an evaluative comment on progress to date (Phases 1 and 2) based on stakeholder assessmentsand an earlier documentary review
- an assessment of the risks implicit in the current approach to Phase 3
- a set of recommendations for a way forward for the coming 18 months which will maximise the sustainability of the outcomes achieved to date through the ELT INSETT project.
2.Approach
Informed by these terms of reference, the following approach to the consultancy was agreed.
- The commission would be seen as a consultative review process.
- This process would have the following major components
-a review of project documents in advance of the visit to Azerbaijan
-a specification of stakeholders to be included in an interview process
-a specification of interview focus and hoped for outcomes for each stakeholder category
-an interview schedule
-a set of detailed notes for each meeting undertaken in the consultancy (these are appended to this report)
-a training workshop for INSETT team members focusing on principles of evaluation in ELT projects
-an outline evaluation plan for Phase 2 of the project
-a report summarising progress to date and making recommendations for an approach to the elaboration of the proposed Phase 3
In general terms such an approach to the evaluation harmonises with the Realist Evaluation framework developed by Pawson and Tilley[1] a framework which has become mainstream in contemporary programme evaluation designs in recent years. In such an approach the main concern in a retrospective evaluation is to answer the question "What has worked, for whom, in what circumstances"? In prospective policy development, the same question becomes: "What might work, for whom, in what circumstances?". This report attempts to address both questions.
3.Main activities during the consultancy
3.1Document review
The Project Manager provided an extensive set of background and policy documents (see Appendices). Following the review of these documents a set of questions and issues was identified. These are summarised in Appendix 2_INSETT Project Evaluation NOTES.doc, and formed the basis for the Stakeholder interviews.
3.2Stakeholder interviews
Outline interview protocols were agreed with the project manager in advance of the consultancy visit. The protocol structure for these interviews can be found in Appendix 3ELT INSETT Evaluation Schedule + outcomes.doc
A series of interviews was held with the following major stakeholders:
Monday 6 February
14.30 - 15.30 Aqiya Nakhchivanli rector AzINSETT and Rena Atayeva head of International department, venue AzINSETT
16:00 – 17:00Rafiqa Aliyeva Baku INSETT head of ELT department venue BakINSETT
Wednesday 8 February
10:00 - 11:00Elmar Qasimov, vice minister ME, venue ME
14:30 - 15.30 Majid Aliyev, head of teacher training and re-training department, venueUniversity of Languages
16:00 - 17:00School visit Jale Mammadova, successful participant from Phase 1, venue school # 134
17:00 - 18:00Ragsana Mammadova President of Teachers' Association AzETA), venue AzETA office.
Detailed notes for all these meetings are listed in the Appendices (4) and are attached to this report. They constitute a primary evidence basis for whatever conclusions and recommendations arise from this review process. A set of publication quality portrait photographs of all interviewees was also prepared. These are available to the British Council on CD to use in its public communication work
The school visit was included in the consultancy as an opportunity for me to have a first hand feel for what is happening in EL classrooms in Azerbaijan, and also to develop a small photographic documentation of classroom practice. This documentation has been completed and a set of web and print ready images has been provided on CD to the British Council, with the understanding that the school will also received a set. Examples from this documentation have been used to illustrate this report.
I will not comment in detail on the meeting at BakINSETT. This was mainly of value in confirming the positive outcomes of Phase 1. A note of this meeting can be found in the appended document -060206b BakINSETT.doc
3.3Evaluation workshop
The other main activity for this review process was an Evaluation Workshop for the ELT INSETT team. The overall purpose of this one day workshop was to provide participants with an update on contemporary approaches to evaluation in education projects, and to prepare a plan for the summative evaluation of Phase 2. This plan can be found in the Appendices (Phase 2 Draft Evaluation Plan.doc). Detailed meeting notes for the day can also be found in the Appendices (5ELT INSETT Evaluation Workshop Notes.doc).
FINDINGS
In the following sections I will summarise progress to date in the ELT INSETT Project and outline issues that have emerged in the course of the review.
4.The ELT INSETT PROJECT: Phases 1 and 2
4.1Progress to date
In partnership with the Ministry of Education, the British Council Azerbaijan has been supporting the development of In-Service Education and Teacher Training (INSETT) since 2003. Phase One of this project completed in January 2005 with the following major outputs:
- a baseline study on ELT in Azerbaijan
- the recruitment and training of a team of 8 potential INSETT trainers
- the development of a locally appropriate ELT INSETT curriculum and training programme for ELT teachers in state secondary schools across Azerbaijan
- the delivery of this new INSETT programme to 25 EL teachers from the Baku region
Phase 2 has been underway since January 2005 and has focused on the consolidation of the capacity of the group of INSETT trainers through the delivery of a second round of training to and the preparation for a possible Phase 3 in which it is envisaged that the INSETT team will be involved in the training of existing or potential English Language INSETT Trainers.
Major outputs to date in Phase 2 have included the identification and recruitment of a cohort of 25 teachers from the 11 regions of Azerbaijan, and the successful delivery of a training programme to English language teachers from regions outside Baku. As a result of the project, the INSETT team has gained recognition (both from their peers and, officially, from the Ministry) as a valuable INSETT training resource, and the school teachers who were trained in Phase 1 have started to deliver local professional updating workshops for teachers in Baku and other regions in Azerbaijan.
The activities in the project have included international training, in-country training courses, conference participation and other professional development opportunities for the INSETT trainers themselves, as well as for participating teachers.
The overall assessment of the project by key stakeholders is positive. The Ministry of Education has given the INSETT team the official status of Trainer of Trainers, the INSETT team members are confident, very well informed on the needs of their trainees, and have the training skills they need to deliver the programme they have developed. Teachers who have gone through Phase 1 training are now able to deliver their own training workshops to colleagues and peers. These are valuable achievements.
4.2A re-statement of the ELT INSETT Project structure
During discussions with the project manager, the project coordinator, and the INSETT team, it was possible to clarify the outcomes and outputs of the ELT INSETT project. These outcomes (what we hope to achieve through the project) and outputs (what we can promise to deliver) were not transparent in earlier project documentation, and this in itself has created the possibility of policy confusion. In the following sections we will consider these recast outcomes and outputs and make suggestions for a way of achieving the desired project outcomes through a third phase.
During the workshop with the INSETT team we clarified the project theory of ELT INSETT in the following terms:
If we improve INSETT provision for English Language Teachers we will:
1.help English language teachers to use new text books more effectively,
2. prepare their own materials,
3. implement these materials inclassroom teaching in Secondary Schools in Azerbaijan, and
4. enable these trained teachers to support colleagues who have not shared their training experience
We also agreed that such a project theory has to be set within the overall framework of education reform processes in Azerbaijanand the factors which constrain this reform (notably local financial and human resource capacity to deliver adequate ELT INSETT) if it is to produce sustainable outcomes.
4.3Outcomes and outputs
In such a context we identified the following major project outcomes and outputs for the first TWO phases of the project:
- OUTCOME 1
-EL INSETT Trainers have the capacity to strengthen existing secondary school teachers' ability to provide a better educational experience for school students in Secondary Schools in Azerbaijan
-EVIDENCE
- Observation of training (peer / managerial)
- Learner self reports
- Portfolios of work produced and teachers' projects by trainees
- outputs
-A group of 8 trained and confident trainers with a broad range of appropriate skills in EL teacher training / trainer training
-An EL Teacher Training Programme (content / methodology) appropriate to the needs of EL teachers in state schools in Azerbaijan
-EVIDENCE
- Its existence
- Quality review by peer / external specialist observation / reporting
- OUTCOME 2
Teachers who have been trained by the INSETT Team are able to 1) use new text books more effectively, 2) prepare their own materials, 3) implement these materials in classroom teaching in Secondary Schools in Azerbaijan, and 4) enable these trained teachers to support colleagues who have not shared their training experience
- outputs
-Teachers' Projects (work done by participants)
-A trained group of 50 teachers (Baku + Regions)
-An EL Teacher Training Programme (content / methodology)
-A management system to enable participation
-Agreement and financial contribution from national authorities
Both of these outcomes are professionally desirable and accord with the interests of major project stakeholders in particular the Ministry of Education. They have also, to a very large extent, been achieved.
However, as presently configured, such a project design cannot achieve national impact, as the number of teachers who can be trained by the INSETT team is necessarily very small. To overcome this limitation, a Phase 3 in the project is envisaged. During this phase, a national team of INSETT trainers will receive professional development from the INSETT Project Team, and will be prepared to deliver the INSETT programme which has been elaborated through the project. The extent to which such a Phase 3 is feasible will be the focus of the following major sections of this report.
4.4Stakeholder relations
During the evaluation workshop with the ELT INSETT team we clarified the major stakeholders in the project as being:
- PRIMARY (immediate beneficiaries)
-English language teachers in Baku and the 11 regions of Azerbaijan
-The BC INSETT team of English language trainers
- SECONDARY (those funding, authorising, cooperating or implementing the project
-MoE (Authorising + Funding)
- Heads of Regional Education Departments
- School Directors
-British Council Azerbaijan (Funding + Implementing))
-British Council Russia (Implementing)
-Consultants (Implementing)
-BC Russia INSETT Team (Implementing)
-WarwickUniversity (Implementing)
-University of Languages (Cooperating)
-AzETA (Cooperating)
- EXTERNAL (none of the above but interested can be friendly or hostile)
-American Embassy
-AzINSETT
-BakINSETT
-British Council (Uzbekistan / Ukraine / Georgia)
-Methodologists in Regional Departments
-OSI
-Parents
-QafqasUniversity
-University of Languages
-World Bank Education Reform Project managers
These stakeholders interact in a structure of relationships which is summarised in the organogram below. The organogram makes clear how the project has developed to date and a number of possible contradictions which will have to be reconciled before any Phase 3 can be embarked on safely.
Figure 1: ELT INSETT Project Structure
4.5Comments on the present project structure
4.5.1Present status of the ELT INSETT training team
At the moment, although the members of the ELT INSETT have a formal recognition as Trainer Trainers from the Ministry of Education, this does not translate into any institutional, salaried positions as in-service Teacher Trainers or in-service Trainer Trainers. For all practical purposes, the team members remain English language teachers in secondary schools or English language instructors in the University of Languages. The teacher or trainer training they undertake within the ELT INSETT Project occurs outside the official Azerbaijan structures for INSETT.