Guidance for inspectors on the use of school performance data
Using the RAISEonline report during Section 5 guidance
Published:July 2007
Reference no:070120
Contents
Background
Definitions – taken from the Common Inspection Framework
Using RAISE to make hypotheses
Standards
The standards reached by learners
Gathering evidence about standards
Using the RAISE report to gather evidence about standards
Evaluating standards on entry
Gathering evidence about standards on entry
Using the RAISE report to gather evidence about standards on entry
Progress
How well learners make progress, taking account of any significant variations between groups of learners
Gathering evidence about progress
Using the RAISE report to gather evidence about progress
Achievement
Evaluating how well learners achieve
Evaluating achievement and standards in the sixth form
Foundation Stage judgements
AoE to Nursery and Reception
Attainment at the end of the Foundation Stage/entry to Year 1
Judging progress in the Foundation Stage
Annex A. The Foundation Stage Profile (FSP)14
Which FSP figures are important?15
Public Service Agreement (PSA) target
Annex B. Judging progress in Key Stage 18
Appendix A. Guidance for interpreting schools’ use of interactive reports
Background
Section 5 inspection judgements focus on the outcomes for pupils. Therefore consider the pupils’ test and examination results as set out in the school’s RAISEonline full report (the RAISE report). The interpretation of the RAISE data should be used to suggest the avenues needed to develop an understanding of the school’s performance at a particular time. It can be used to develop hypotheses in the pre-inspection briefing (PIB); it can also be used during the inspection to ask questions about the school’s strengths and weaknesses and, crucially, how the school is responding to them.
Inspection judgements are based on a far wider evidence base than the RAISE data, and include the school’s more recent internal assessments and observation of learners in the classroom. This will help you judge pupils’ current achievement. Many schools will also have access to other data packages, to which you should have due regard.
This guidance seeks to help inspectors use RAISE data and wider evidence gathering to evaluate standards, progress and achievement.
Definitions – taken from the Common Inspection Framework
Standards: an evaluation of the standard of pupils’ work in relation to their learning goals.
Progress: an assessment of pupils’ progress relative to their prior attainment and potential, with any significant variation between groups of learners.
Achievement: an overall assessment of pupils’ success in achieving challenging targets, including qualifications and learning goals with trends over time and any significant variations between groups of learners.
Using RAISE to make hypotheses
When considering data and their relevance to standards, progress and achievement, it is important that you remember to:
use the interpretation of the RAISE report to inform pre-inspection hypotheses, along with information provided in the self-evaluation form (SEF)
consider carefully the wording used in the PIB so that it is clear that judgements about standards and progress will not be made on the RAISE data alone
agree an interpretation of the RAISE data as early as possible with the school
ask a school, where that school believes that the data no longer provide a fair view of the school’s current performance because of recent improvements, to provide evidence to support its view; inspectors should tailor their inspection strategy accordingly
use professional judgement in interpreting data; the only fixed indicators provided in RAISE are for significantly above or below expectations
only investigate the data further if there is a real need to do so.
Standards
The standards reached by learners
Judgements about standards are referenced to norms, such as the percentage of pupils reaching a certain National Curriculum level compared with the percentage nationally. They are independent of judgements on progress. Grades for standards reached by learners are consequently described differently from all other inspection grades. Note that standards for schools are judged against those found nationally, not against similar schools; thus, in judging standards in a special school, you must compare them with national standards.
Grade 1: standards which are exceptionally and consistently high
Grade 2: standards which are generally above average with none significantly below average
Grade 3: standards which are broadly average to below average
Grade 4: standards which are exceptionally low.
Gathering evidence about standards
A broad range of evidence should be considered when evaluating standards. This includes:
the school’s results as set out in the RAISE report
the SEF and what it says about the latest standards in the school
the school’s data, the work the pupils have done in the past and the knowledge, skills and understanding that they are currently able to demonstrate.
An inspector’s early interpretation is normally based on the school’s latest test or examination results. However, where there are no national test results – for example in the Foundation Stage, special schools and pupil referral units (PRUs) or where the school and/or inspectors believe current standards differ from the latest results – it is especially important to seek out all the available evidence.
Using the RAISE report to gather evidence about standards
Before the inspection the SEF and the RAISE report will give you information about the standards reached in the past. The graphs entitled ‘Attainment Summary’ tell you about test scores.
Check the contextual information to see if there have been any significant changes, for example the arrival of substantial numbers of migrants.
At Key Stages 1–3, look at the overall, and English, mathematics and science average point scores (APS).
At Key Stage 4, look at the capped and uncapped average total point scores, as well as those for English and mathematics.
In all key stages, explore whether the school’s values differ significantly from national scores. Note that where a sig+ or sig- is shown this indicates that standards may be statistically significantly higher or lower than average – but not necessarily that they are exceptionally high or low. Remember that ‘significant’ in the RAISE report does not necessarily mean important.
Look at the trend in point scores over the years and any differences that exist between the school and the national scores. This will help you interpret whether standards are improving, declining or fluctuating.
Explore any similarities and differences between subjects and key stages – but do not lose sight of the ‘big picture’. Consider if any particular subjects have a disproportionate effect on overall scores.
Analyse the percentage of pupils reaching the nationally expected levels in the core subjects as shown in the attainment summary (Key Stages 2 & 3 only) and their relationship to the national picture.
Take account of the proportion of pupils entered for GCSEs and short courses and whether there are gender differences.
Be aware that differences in the Key Stage 4 summary tables may be because the school runs a high proportion of vocational courses.
Consider the impact on scores in schools where pupils are entered early for examinations.
Investigate carefully the performance of different groups of learners such as those from particular minority ethnic groups, those who qualify for free school meals, those with learning difficulties or disabilities and those from particular bands of ability.
Use RAISE data to track the performance of looked-after children who appear as ‘in care’. It is possible for schools to identify individual looked-after children on the pupil scatter graphs, for example.
Evaluating standards on entry
In order to decide how well pupils make progress within a school it is important to assess their standards on entry.
Gathering evidence about standards on entry
Take account of pre-inspection and first-hand evidence. In particular, look at standards on entry information in the SEF. In primary schools with a Foundation Stage (see Reference booklet) the school’s analysis of any baseline assessment data will be important. Data about the Early Learning Goals should be available, but inspectors should not attempt to make any direct links between achievement in these and progression to Key Stage 1 levels. In all schools, your own evidence from observations, talking to the staff and the youngest pupils, and looking at their assessment records and work will help to establish the pupils’ starting points.
Using the RAISE report to gather evidence about standards on entry
The RAISE report will guide you in understanding standards on entry in schools with Key Stages 2−4 pupils. The pages entitled ‘Contextual Information Prior Attainment of Pupils’ in Section 1 provide a year-by-year group breakdown of standards on entry to each key stage. The bar charts show school figures set against national figures.
Look at the school’s average figures and compare them with the national figures.
Check the bar charts for any unusual distributions, for example a low school percentage at the highest level.
Look for trends in standards on entry and whether year groups differ, taking account of the possible fluctuation in standards with small cohorts.
Find out from the school if the standards of the latest intake to the key stage differ.
Look at the percentage coverage. If it is low, check whether the standards of the missing pupils differ from those in the graphs.
Progress
How well learners make progress, taking account of any significant variations between groups of learners
The task here is to decide how well pupils make progress and how this matches up to what you would expect given the pupils’ context and starting points. The judgement on progress is, in effect, the value added by the school. A starting point is the contextual value added (CVA) information in the RAISE report. Evidence of the school’s success in meeting challenging targets will be important in evaluating the learners’ progress.
Gathering evidence about progress
As with analysing standards, it is important to consider a range of pre-inspection and first-hand evidence; for instance:
the latest CVA measures
the SEF and what it says about the progress made by pupils
the local authority’s (LA’s)/school’s own data analysis and progress records
discussion with staff and pupils
scrutiny of their work in lessons and what they have done in the past
the knowledge, skills and understanding that they are currently able to demonstrate
the demands made upon pupils by the teaching.
Many schools use data such as Fischer Family Trust (FFT) to evaluate the progress pupils make. Bear in mind that FFT analyses apply a slightly different methodology and formula to CVA. For most schools, the indicators are similar but occasionally they differ. If necessary, these differences should be explored with the school along with any LA’s internal comparisons between their own schools.
Using the RAISE report to gather evidence about progress
The CVA score for a school is a measure of the progress that pupils make from the end of one key stage to the end of another based on their national test results. Each individual pupil’s CVA is worked out using point scores. The calculation takes account of contextual factors such as prior attainment, gender, mobility, learning difficulties or disabilities, ethnicity, whether English is the first language, post-code income deprivation index, care status and eligibility for free school meals. The school’s overall CVA score provides an indication of the value added to all pupils in terms of their progress. However, it does not take into account data that are not held centrally but that may affect pupils’ progress, such as support from their family with their homework or illness at assessment time. Such factors are important to consider when evaluating progress.
The CVA methodology was developed in partnership with the Department for Children, Schools and Families (DCSF) and leading academics. The DCSF has completed pilots of CVA measures with schools and the vast majority agreed that it is a fairer measure than any other used previously to indicate a school’s impact. However, a mathematical model can never perfectly capture a complex ‘real life’ concept such as school effectiveness, and that it is why it is vital to use first-hand evidence.
It is important to distinguish CVA from standards. You need to look at all the available CVA information. Do not form your first impression just from the overall CVA graph or make assumptions based on percentile ranking – especially in small schools. Hypotheses should be followed up during the inspection and considered in the light of school circumstances and all the available evidence.
Go to ‘The contextual value added: overall’ chart of the RAISE report. This shows the school’s overall CVA and that for core subjects relative to the national mean of 100 for primary schools and 1,000 for secondary schools. Look at the trends over the years and whether there are significant differences between school and national figures.
It is important to remember that a school’s CVA measure falls within a range of values indicated by the upper and lower confidence limits; if this range crosses the national expectation line, then it is an indication that a school is ‘doing as well as expected’ or that there is insufficient data to be confident.
Analyse whether there are similarities and differences between subjects and key stages. This will give you an understanding of the questions that need to be raised with the school about the strengths and weaknesses in pupils’ performance.
Look at the ‘CVA performance of groups within the school’ page. Skim the CVA chart and the rows of the group CVA tables at the bottom of the page to see if they suggest that some ethnic/gender/socio economic groups are performing significantly better (sig+) or worse (sig-) than others. Be mindful of the size of groups. A small group with very poor performance may NOT show as significant, but there will still be individuals who have done badly in their tests. Again, this will give an understanding of the questions that need to be raised with the school about the strengths and weaknesses in groups of pupils’ performance.
Check the scatter graphs which show each pupil’s CVA score. Check gender differences and whether substantial proportions of pupils make progress in the top 10% (indicating exceptionally high progress) or bottom 10%, nationally (indicating exceptionally low progress). Remember to take account of data not held centrally, for example about pupils’ illness at assessment time. This may explain outliers in the scatter graph and could impact on the overall CVA score for a small cohort. Consider the gap between the pupils that gain the most and those that gain the least. Remember that a consistently higher attaining school might find it difficult to demonstrate good progress. Check the maximum possible score and consider whether ceiling effects might have constrained CVA.
Compare conversion rates with national figures to check whether the school successfully converts levels in core subjects at one key stage to expected levels at the next. This will provide another view of progress in the different subjects.
Achievement
Evaluating how well learners achieve
Inspectors make a professional judgement when they come to a view about overall achievement. This is based on what they have found out about standards and progress during the course of the inspection and includes an assessment of pupils’ success in achieving challenging targets. The latter evaluation should be based on first-hand inspection evidence and information from the SEF about statutory targets. The prior attainment tables in the RAISE report may also provide a rough check on whether targets are sufficiently challenging.
The headline judgement on achievement and standards is generally the same as the judgement on progress. For example, it is perfectly possible for a special school to have exceptionally low standards (grade 4) and outstanding progress (grade 1), leading to grade 1 for achievement and standards.
When coming to an overall judgement about standards and achievement, inspectors must remember to check the grade descriptors in the guidance to inspectors. Grade 1 performance is unusual by definition; it is outstanding or exceptionally high. Grade 4 performance is also markedly different from that of most schools; it is exceptionally low.