A guide to 2014 key stage 2 to key stage 4value added measures
CONTENTS / PAGE NO.Summary – Interpreting School and Pupil Group Value Added Scores / 2
What is Value Added? / 3
Changes to Value Added Methodology in 2014 / 5
Calculating KS4 Attainment / 6
Calculating KS2 Prior Attainment / 8
Calculating Pupil Value Added Scores / 11
Calculating School Value Added Scores / 13
Interpreting School Value Added Scores / 14
Calculating Pupil Group Value Added Scores / 16
Interpreting Pupil Group Value Added Scores / 17
Technical Annex– contents / 18
1
When evaluating examination performance, it is important to take into consideration that when pupils first join a secondary school, they have varying levels of prior attainment.
The purpose of value added measuresis to look at how much progress pupils have made from the beginning to the end of their compulsory secondary education (between the end of key stage 2 (KS2) and the end of key stage 4 (KS4) ) and how this compares with pupils of similar prior attainment.
Analysis shows that there is historically a strong relationship between performance of pupils at key stage 2 and examination results at the end of key stage 4. A value added measure uses this relationship to “estimate” how well each pupil is expected to perform in their key stage 4 exams.
This is done by looking at the actual KS4 performance of all pupils and working out the average KS4 points achieved by pupils nationally of similar prior attainment at the end of KS2.This provides an estimated KS4 outcome for each grouping of KS2 prior attainment[1]. This KS4 “estimated outcome” can then be compared against what the pupil actually achieved in their KS4 exams, to see whether or not they exceeded it. The difference between a pupil’s actual KS4 performance and their estimated KS4 performance gives the pupil their value added score.
The value added score for all pupils in a school can then be aggregated to calculate a school’s value added score, which helps to identify schoolswhose pupils make more progress or less progress than average, given their prior attainment.The summary diagram onpage2shows how to interpret these scores.
Value added scores are calculated for the following KS2-4 value added measures:
KS2-4 ‘Best 8 plus English and mathematics bonus’ value added measure
The ‘Best 8 plus English and mathematics bonus’ value added measure estimates how pupils perform in their best 8 GCSEs (or equivalent qualifications) with pupils receiving an additional bonus for their performance in GCSE English and mathematics, as the point scores for these subjects are counted twice.
KS2-4 English Baccalaureate value added measures
Value added scores are also calculated for the five English Baccalaureate subject areas:
- KS2-4 English Baccalaureate – English subject area value added measure
- KS2-4 English Baccalaureate – mathematics subject area value added measure
- KS2-4 English Baccalaureate – science subject area value added measure
- KS2-4 English Baccalaureate – humanities subject area value added measure
- KS2-4 English Baccalaureate – languages subject area value added measure
A pupil has a separate estimated KS4 outcome calculated for each of the five English Baccalaureate subject areas. These estimates are then compared against a pupil’s best score in the qualifications that feed into each English Baccalaureate subject area. For example, if a pupil achieved a B in GCSE geography and a C in GCSE history then it is the GCSE geography result (the pupil’s best result in the English Baccalaureate humanities subject area) which is used to compare against the estimated KS4 outcome for the English Baccalaureate humanities subject area value added measure. A pupil then has a value added score calculated for each subject area by finding the difference between their actual KS4 attainment in the subject area and their estimated KS4 attainment in the subject area.
All pupils are included in the English and mathematics subject area value added measures. However, only the pupils that have taken the required qualifications at the end of KS4 are included in the science, language and humanities subject area value added measures.
Following the changes to the way qualifications are counted in the 2014 performance tables outlined on page 6, the distribution of the KS4 Average Point Scores (APS) changed due to the impact of Review of vocational education (The Wolf Report[2])and early entry reforms. As a result, the existing modelling approach no longer provided as good a fit to the data, which meant biases in value added scores would have been introduced for pupils with very high and very low prior attainment. Therefore, to remove these model biases, which may impact on school value added scores, a revised and simplified value added model has been developed to better represent the progress pupils make.
This revised model uses a simpler mean line methodology based upon analysing pupil performance at each prior attainment band separately. This has the benefit of both removing unwanted bias in the methodology and simplifying the model so it can be more readily understood. See section C in the technical annex for further details of the attainment estimates for each prior attainment band.
Using the mean line methodology the value added shrinkage factor adjustment is no longer applied. The shrinkage factor was previously applied to provide better value added estimates for schools with a small number of pupils. This was calculated using the within school and between school variances that were routinely estimated as part of the previous methodology. However, on reflection including the shrinkage factor did not make enough of an impact to warrant the complexity it brought to the methodology. To promote simplicity, this will no longer be applied.
Please see pages 11 and 13 for further details of how pupil and school value added scores are calculated under the mean line methodology.
Qualifications included in the measures
Best 8[3] value added is based on pupils’ performance capped to eight entries. The 2014 best 8 scores follow the new performance tables’ rules;
- a reduced number of eligible qualifications are included compared to previous years[4]
- the size of each qualification has been capped at a maximum of one GCSE per entry
- a limit of two non-GCSE qualifications that contribute towards the measureand;
- early entry rules[5] apply, in which only a pupil’s first entry in an EBacc subject is counted rather than their best entry.
The qualifications included are also subject to usual performance tables discounting rules. Further detail is available here:
The subject value added measures include only results in qualifications that qualify for the English Baccalaureate. Further information on which qualifications count towards each subject area can be found here:
Key Stage 4 point scores
A pupil’s performance in their best 8 qualifications (plus English and maths bonus) continues to be expressed as a point score with a maximum of 580 points available. Subject point score calculations also remain the same as before with values up to 58 points as shown in table 1 below.
Table 1 Qualification point scores
Qualification / Point scoreGCSE – grade A* / 58
GCSE – grade A / 52
GCSE – grade B / 46
GCSE – grade C / 40
GCSE – grade D / 34
GCSE – grade E / 28
GCSE – grade F / 22
GCSE – grade G / 16
Further information on point scores can be found here:
A pupil’s average performance in Key Stage 2 English, maths and science tests will continue to be used as a baseline to compare pupil progress. The average KS2 point scores are converted into a KS2 fine level. The KS2 fine level is a simple conversion of dividing a pupil’s average KS2 point score by 6 and then rounding to 1 decimal place.
Worked Example
Samantha’s marks in her Key Stage 2 tests (taken in 2009) were 77, 76 and 74 in English, mathematics and science respectively.
The diagram below sets out how these are converted into an average KS2 fine level to be used as the prior attainment input into value added measures.
For English, mathematics and science assessments where a level of 3 to 5 is awarded in the test, the fine grade is calculated by:
Using the English test mark above, 77 is a level 5 as it is between 67 and 100 which were the level 5 thresholds for 2009. The fine gradeis therefore calculated as follows:
The fine points are calculated by multiplying the fine grade by 6: 5.29*6=31.74
The same calculations are applied to the mathematics and science test outcomes and the average of the three subjects iscalculated to give an average point score of 31.74. This is then converted to a fine level by dividing by 6 and rounding to 1 decimal place to give a fine level of 5.3.
Adjustments are made for pupils with incomplete test results:
- If a pupil does not have a test result in a subject then their teacher assessment level is used (see table below).
- If a pupil has a result missing in one or more subject, then prior attainment is calculatedfrom the remaining subject(s).
- If a pupil does not have a test score or teacher assessment result in any subject then they are excluded from the measure.
Table 2 – teacher assessment adjustment
If test level =3-5 / Use pupil’s fine pointsscore
2 / If teacher assessment available, use; / Award:
W = 3
Level 1 = 9
Level 2 = 15
Any higher = use pupil’s fine pointsscore
A,D,F,L,P,Z = Exclude pupil
If no teacher assessment available / Exclude Pupil
B, N / If teacher assessment available, use; / Award:
W = 3
Level 1 = 9
Level 2 = 15
Any higher = 15 (capped)
A,D,F,L,P,Z = Exclude pupil
If no teacher assessment available / Exclude Pupil
A, M, Q, S, T, X / If teacher assessment available, use; / Award:
W = 3
Level 1 = 9
Level 2 = 15
Level 3 = 21
Level 4 = 27
Level 5 = 33
Any higher = 33 (capped)
A,D,F,L,P,Z = Exclude pupil
If no teacher assessment available / Exclude Pupil
Notes on grade codes
A – Absent
B – Working below the level of the test
D – Disapplied
F – KS2 pupil not at end of KS2 and taking this subject in future years
L – Left
N – Not awarded a test level
M – Missing
P – Results for subject found in previous year’s dataset
S – Pending maladministration
Q – Maladministration
T – Working at the level of the tests but not able to access them
W – Working towards level 1
X – Lost
Z –Ineligible
Individual pupil value added scores are calculated firstbefore a school value added score can be produced.
The first step is to calculate a KS4 “estimated outcome” for all pupils that are at the end of KS4 in 2014.
This is based on theactual KS4 outcomes of all pupils nationally with the same level of achievement at KS2 (prior attainment). A pupil’s prior attainment is defined as the average of their KS2 English, mathematics and science results, in fine levels.
A pupil’s value added score is the difference between their estimatedandactual KS4 outcome.
Worked Example
Samantha has anaverage KS2 point score of 31.74giving a fine level of 5.3.Her best 8 plus English and maths bonus score is508. The national average best 8 plus English and maths bonus score for pupils who share the same KS2 results as Samantha is 482.17.
Samantha’s value added score is the difference between her actual best 8 score and the estimated best 8 score, that is, 508 – 482.17 = 25.83.
Section B of the technical annex provides a more detailed description of how pupils’ estimated KS4 scores and their value added scores are calculated.
In additiona KS2-4 pupil level value addedready reckoner can be found in the RAISEonline library at:
This provides the option for users to input data and test different scenarios.
Pupils in special schools
The estimated KS4 attainment for pupils in special schools is based on acomparison with pupils of the same prior attainment in mainstream schools.
This means that their value added scores are calculated based on averages derived from mainstream schools only.
Similarly, confidence intervals in special schools and their pupil groups arecalculated using the values from the mainstream pupil population.
The school’s value added score is the mean average of its pupils’ value added scores.
Worked Example continued
Samantha is one of 142 pupils in her school’s KS4 cohort,
Pupil # / Pupil name / Value added score1 / Samantha / +25.83
2 / George / -9.21
… / … / …
142 / James / +31.51
Sum / +1,776.42
The school’s best 8 value added score is therefore 1776.42/142 = 12.51. For presentation purposes, this score is added to 1000 and rounded to one decimal place. As such, this school would have a best 8 value added score of 1012.5.
When evaluating a school’s performancewe must be careful to note that it is based on a given set of pupils' results for a particular test paper on a particular day.
A school could have been equally effective and yet the same set of pupils might have achieved slightly different KS4 results on the day. And the school would almost certainly have shown slightly different KS4 results with a different set of pupils.This element of uncertainty needs to be taken into account when interpreting a school’s value added score; this is done using confidence intervals.
Confidence intervals show the range of scores within which each school’s underlying performance can be confidently said to lie. A school’s confidence interval is always centred on the school’s value added score. For example, if a school’s value added score is 1,010 and the size of the school’s confidence interval is 5 points, then the confidence interval ranges between 1,005 and 1,015 (i.e. 5 points either side of the school’s value added score).
The size of the confidence interval is determined by the number of pupils in the school at the end of KS4.Smaller schools have wider confidence intervals because their value added score is based on a smaller number of pupils, so there is less evidence on which to judge the school’s effectiveness.
To judge a school’s effectiveness, both the school’s value added score and the associated confidence interval needs to be taken into account. If the whole range of the confidence interval is above 1,000 (i.e. the lower confidence limit is greater than 1,000), we can say the school score is above the national average and is statistically significant, and we can be confident the school is helping its pupils make better than average progress. An illustration of how to interpret school value added scores is given on page 2.
Similarly, when the entire range of the confidence interval is below 1,000(i.e. the upper confidence limit is less than 1,000), we can say the school score is the national average and is statistically significant.
Finally, if the confidence interval straddles the national average of 1,000, then we can say that the school is not significantly different from the national average, in other words, we cannot confidently say that the schools value added score is definitely above or definitely below the national average.
The tableand diagram overleaf show how a school’s value added score and confidence intervals should be interpreted to reach one of the three definitions above.School A is an example of a school that is significantly above national average; School B is not significantly different from national average; and School C is significantly below national average.
For more information on the calculation of confidence intervals, please seeSection E of the technical annex.
Comparison of English Baccalaureate subject area value added scores
Confidence intervals must also be taken into account when comparing two or more English Baccalaureate subject area value added scores within a school (e.g. when comparing a school’s English Baccalaureate English value added score with their English Baccalaureate humanities value added score).
The size ofthe confidence intervals for each of the five English Baccalaureatevalue added measures for a school will vary in size (as they are based on different models and different numbers of pupils)– each interval defines the range of values within which we are statistically confident that the school’s true value added scorefor the respective English Baccalaureate subject area lies.
The value added score for a pupil group (e.g. disadvantaged pupils) in a school is calculated as the average value added score of all pupils that belong to that group in the school. Similarly, the value added score for a group nationally is the average value added score of all pupils in mainstream schools that belong to that group nationally.
Worked Example continued
Samantha is one of 30 ‘disadvantaged’ pupils (defined, as pupils who are either eligible for Free School Meals or are children who are looked after) among the 142 pupils in her school’s KS4 cohort, who gain a range of ‘Best 8’ value added scores:
Disadvantaged pupil # / Disadvantaged pupil name / Value added score1 / Samantha / +25.83
2 / George / -9.21
30 / Alison / +12.16
Sum / 347.41
The disadvantage pupil group value added score for the school is therefore 347.41/30 = 11.580. For presentation purposes, this score is added to 1,000 and rounded to one decimal place. As such, this school would have a value added score of 1011.6for its disadvantaged pupils.
Pupil group confidence intervals
Information on how confidence intervals are calculated for pupil groups is contained in section F in the technical annex.
When comparing pupil group value added scores, it is important to note that these scores are based on a given set of pupils’ results (who belong to a pupil group) for a particular test paper on a particular day. Confidence intervals are also calculated to give a range of scores within which we are statistically confident that a school’s pupil group value added score will fall. There are two ways in which a pupil group value added score can be compared; to the national average for all pupils(1,000) or thenational pupil group average.
For an explanation of how to interpret confidence intervals, please refer back to page14(Interpreting School Value Added Scores) and an illustration of how to interpret pupil value added scores is also given in technical annex A.
A technical guide to
Key Stage 2 to 4 value added
Contents / Page No.A. Interpreting school value added scores for pupil groups – disadvantaged pupils / 19
B. Calculating Pupil Value Added Scores / 20
C.2014 unamended attainment estimates / 22
D.Calculating Value Added Scores for pupil groups / 23
E.Calculating Confidence Intervals / 25
F.Pupil group confidence intervals / 27
1