1

FISCHER FAMILY TRUST

DATA ANALYSIS PROJECT

Information Taken from a Range of Fischer Family Trust Guidance Materials

OCTOBER 2007

CONTENTS

p2 – Key Points

p3 – Background Information
PA, SA, SX models; Type A-D estimates; LMU ability bands

ANALYSES TO SUPPORT SCHOOL SELF-EVALUATION

p5 – Contextual VA Report

p6 – Value-Added 3 Year Summary Report

p7 – Value-Added Significant Areas – Grid
with key terms defined

p8 – Some Combinations of Arrows & Boxes

p9 – Value-Added Significant Areas – Text

p10 – Contextual Value-Added Indicators (Core Subjects)

p11 – Estimates Summary

FFT DATABASE: ESTIMATES & VALUE-ADDED REPORTS

p12 – Amyuni pdf convertor info.; KS4 Pupil Subject Estimates;

p13 – KS4 School Estimates; KS3&4 Value-Added Pupil Reports

p14 – School Estimates: Notes

p15 – FFT TECHNICAL INFORMATION

p16 – FFT FAQ

FISCHER FAMILY TRUST (FFT) DATA ANALYSIS PROJECT : KEY POINTS

  • Calculations includeonly maintained mainstream schools with 10 or more pupils withresults.
  • Where mobility is high there might be significant changes in terms of the cohort population and prior-attainment between KS.
  • School level variables are used on a continuous scale, not by band or benchmark group, avoiding issues for schools “on the edge” of groups.
  • FFT use Teacher Assessment as well as Test data to reduce the impact of a particularly ‘good’ or ‘poor’ performance on one or more tests; highlighting any unusually large differences between TA and Test. Where a pupil was absent from a test their TA level is used if available.
  • Test levels are calculated using “fine grades” derived from test marks. e.g. the mark range for L4 is 45 to 60 (= 16 marks); a pupil achievesL4 with a markof 48; their mark is 4 above the minimum required for level 4;this is one-quarter of the total mark range for the level;their ‘fine grade’ = 4.25
  • Ability bands(UML) are based on prior attainmentcalculated from levels: they do not use marks, fine grades, subject differences, age or gender; they simply group pupils by overall attainment.
  • Thresholds for the ability bands were set in 1997/98 and have not changed - because of improvement in results, there will now be more than one-third of pupils nationally in the upper bandand less than one-third in the lower band.
  • Reports are based only on pupils where prior-attainment data is available from the previous KS: overallattainment figures may differ from published whole-school data including all pupils.
  • Pupils disappliedfor all NC subjects or not entered for GCSE/GNVQ examinationsare not included: it is important to take into account the % of pupils not entered.
  • Since 2006 KS4 reports include all approved level 1 and level 2 qualifications. Where the # symbol is used, this indicates that only GCSE and GNVQ qualifications have been included.
  • VA as a percentile rank against all maintained secondary schools nationally, with 1 for the highest value-added scores and 100 for the lowest: a change of 10 in percentile rank represents around 330 schools.
  • A change in rank from 40 to 60 represents a relatively small change in value-added and should not be regarded as indicating an issue (see graph). The difference between schools at 1 and 25 is around 18% whereas the difference between 25 and 50 is around 5%.
  • Highlights show where performance is significantly different when compared to the progress of ‘similar pupils in similar schools’ - this helps reinforce the message to focus on significance, not ranks, as a small variation in scores between two schools, can lead to very large differences in percentile ranks.
  • Significance depends on the number of pupils and the size of the difference.(e.g. a difference of 10% for 10 pupils is as significant as a 1%difference for 100 pupils: both represent 1 child).
  • “No highlight” means the value-added score has kept pace with the average performance of all schools – it does not necessarily meanthat there has not been any improvement.
  • The ‘3 Yr Ov’ figures are based upon combining the individual pupil data over a 3 year period.
  • Estimates areaverage grades: if the estimate is an A, a goodproportion will achieve A* or B.
  • In 85% of cases overall, pupils achieve within 1 grade of the estimated grade.
    Core subject estimates are most accurate (over 90% achieve within 1 grade). Estimates are least accurate for subjects where individual aptitude can impact significantly (e.g. Art, music) where around 70% achieve within 1 grade.
  • Unlike FSM which, at pupil level, is a ‘Yes/No’ indicator the ACORN geodemographic data provides a measure on a scale from ‘advantaged’ to ‘disadvantaged’ (overall correlation between school FSM and school Acorn factors is fairly high).

BACKGROUND INFORMATION

The Fischer Family Trust (FFT) Data Analysis Project calculations includeonly maintained mainstream schools with 10 or more pupils withresults in the latest key stage.

The FFT website includes current versions of all standard reports, guidance material and updates.

FFT KEY STAGE 3 & 4 VALUE-ADDED MODELS(SCHOOL & PUPIL)

PA:If pupil progress was the same as that made by all pupils nationally (taking account of each pupil’s priorattainment [test and TAlevels],subject differences(difference between attainment in each core subject test and the overall (mean) test level), month of birth, andgender.

SE:If pupil progress was the same as that made by pupils in similar schools(using FSMentitlement, average prior attainment anddistribution of prior attainment(standard deviation) - also ageodemographic ranking using pupil postcode togive another indicator of socio-economicbackground).

SX: Ifpupil progress was the same as theprogress made by similar pupils in similarschools,(based on the PAmodel, adjusted for a wide range of pupiland school contextual factors*, including mobility(time in current school),socio-economic background(linked to each pupil’s postcode and then averaged for the school), ethnicity andspecial needs.

*School level variables are used on a continuous scale, not by band or benchmark group, avoiding issues for schools “on the edge” of groups.

FFT KEY STAGE 3 & 4 VALUE-ADDED REPORTS – FACTORS INCLUDED

FFT SCHOOL & PUPIL ESTIMATES

Using national pupil-level datasets, current models use four types of estimates (A, B, C, D):

Type A: pupil progress the same as all pupils nationally,(taking account of each pupil’s prior attainment[test marks and TA levels*], subject differences, month of birth, andgender).

Type B:pupil progress the same as pupils in similar schools, (based on individualpupils’ prior attainment and gender and the performance of ‘similar’ schools [FSM entitlement, average prior attainment, anddistribution of prior attainment]).

Type C: pupil progress consistent withthat necessary to achievenational targets.

Type D: pupil progress similar to pupils in the ‘top 25%’ of schools nationally last year (schools with the highest KSKS value added).

* FFT use Teacher Assessment as well as Test data to reduce the impact of a particularly ‘good’ or ‘poor’ performance on one or more tests; highlighting anyunusuallylarge differences between TA and Test.Test levels are calculated using fine grades derived from marks.

ABILITY BANDINGSare based on prior attainment at the end of theprevious KS(test TA levels):

L = Lower (broadly equivalent to age standardised score of 95 or lower)

M = Middle (broadly equivalent to age standardised score of 96 to 105)

U = Upper (broadly equivalent to age standardised score of 105 or above)

The bands were set using data for 1997/98 and have not changed - so they alwaysrepresent the same range of prior attainment. Because of improvement in results, there will now be more than one-third of pupils nationally in the upper bandand less than one-third in the lower band.

“ANALYSES TO SUPPORT SELF-EVALUATION”

NB: Reports are based only on pupils where prior-attainment data is available from the previous key stage: overallattainment figures may differ from published whole-school data including all pupils.
Reports issued in the autumn term useprovisionaldata for the most recent year;this issubsequently amended in the spring release.

CONTEXTUAL VA (KS234):
Overview of VA over last 5 years using two indicators for both KS3 and KS4; enables view of contextual VA for the same cohort moving through each keystage; shows VA as a percentile rank(against all maintained secondary schools nationally, with 1 for the highest value-added scores and 100 for the lowest) and highlights where performanceis significantly different when compared tothe progress of ‘similar pupils in similar schools’.

The data are presented so trends can be examined for the same cohort of pupils byworking across, showing overall KS2 to KS4 performance and, within this, thecontribution of KS3 and KS4.

Analyses for both PA and SX models are included.

Core Subjects L5+= % of pupils at level 5+in all core subjects; Overall Points Score= mean coresubjectlevel (using ‘fine grades’); Capped Points Score= GCSE/GNVQ points (best 8 subjects).

Also,actual %5A*-C attainment is shown (highlightedif below 30%),as is% of pupils not entered for GCSE/GNVQ (highlighted when unusually high or low for the school’s %FSM).

NB: Key Stage 4 value-added scores ONLY include pupils who were entered for 1 or more GCSE or equivalent examinations: it is important to take into account the % of pupils not entered;
Where mobility is high there might be significant changes in terms of the cohort populationand prior-attainment between KS;
A change in rank from 40 to 60 represents a relatively small change in value-added and should not be regarded asindicating an issue, whereas a trend from 20  40  60  80 over 4 years would be highlysignificant.

VALUE ADDED (3 YEAR SUMMARY):
Provides a 3 year summary, covering KS2-3, KS2-4 and KS3-4 using both PA and SX models.

The report offers a one-page overview of pupil progress in each KS against key measures.

Where value-added has changed in significance (e.g. significantly below  not significant) over the 3 year period, one or two arrows will be shown (double arrow = rapid rate of change).

NB: An arrow indicates improvement or decline relative to all maintained schools- ‘no arrow’ means the value-added score has kept pace with the average performance of all schools – it does not necessarily meanthat there has not been any improvement;

KS2-3 Indicators: NC Mean Level(mean level achieved in all 3 core subjects,using fine-grade calculated from test marks); Core Subjects Level 5+ or 6+ (% L5+ or 6+ in all 3 core subjects).

The ‘3 YearsCombine’ and ‘3 Year Trend’ figures are based upon combining the individual pupil data over a 3 yearperiod,not theaverage of the three individual percentile ranks.

KS23, KS24 & KS34 VALUE ADDED (SIGNIFICANT AREAS - GRID) [SX]:
The report is based upon value-added data over a 3 year period and highlights groups where value-addedis significantly high or low (coloured box), or has changed in significance(arrow).

Indicators: Threshold (%L5+, %L6+, 5 or more A*/A, A*-C, A*-G), Threshold EM (5 or more A*-C, A*-G including EN and MA), Points/Level (Mean Level/Average Grade, Total Points, Capped Points) and Core Subject (EN, MA and SC grades); all shown for a range of groups.

Groups: all pupils, boys and girls (divided into Upper, Middle and Lower prior-attainment bands); each individual’s entitlement to Free School Meals;each pupil’s SEN Stage;ten ethnicity groupings.

NB : A pupil group analysis is shown ONLY wherethere are 10 or more pupils in the groupand value-added is significantly higher or lower than expectedorvalue-added is improving or declining in relation to national norms;
A single arrow= a change of one level of significanceover 3y (e.g. significantly below  not significantly different); adouble arrow = change of two levels (e.g. significantly above  significantly below);
If fewer than 10pupils in agroup, only the number of pupils is shown(value-added scores are not highlighted) - if a group has 10 or more pupils but no highlights, this means no scores or trends are significant i.e. progress broadly consistent with that for similar pupils in similar schools nationally;
Differences between overall (e.g. NC level or Points Score) and threshold(e.g. L5+, 5A*C) measures could indicate that a substantial proportion of pupils may be achieving just above or just below the threshold or the school may be concentrating its efforts on borderline pupils to the detriment ofhigher or lower attaining pupils.

# indicates that only GCSE and GNVQ results are used in the analysis.

KEY TERMS: VALUE ADDED(SIGNIFICANT AREAS)

Pupils KS2/3 Match- total number of pupils matched over a 3 year period.

Threshold- performance measured as % achieving at or abovea given level (e.g. L5+, 5A*C)

5A*C(EM)- % of matchedpupils achieving 5+ A* - C grades, includingEnglish and Mathematics

%A*G(EM) - % of matchedpupils achieving 5+ A* - G grades, includingEnglish and Mathematics

Points: Avg#- Mean Grade across all GCSE results (GCSE & GNVQ only)

Points: Total- Uncapped GCSE & equivalent Points Score

Points: Cap- Capped GCSE or equivalent Points Score (best 8 subjects)

Core Subject# EN – GCSE/GNVQ Points Score for the subject

KS23, KS24 & KS34 VALUE ADDED (SIGNIFICANT AREAS - TEXT)[SX]:
Similar to ‘Significant Areas – Grid’ but shows the data in words instead of a visual approach, showing pupil groupings ONLY wherevalue-added is significantly higher or lower than expected over 3y or value-added is improving or declining significantly in relation to national norms (only where there are 10 or more pupils in each individual year).

Reports are organised into sections for each subject at KS3 or each indicator at KS4.

The figures included under each heading are either point scores (KS4), levels (KS3)or percentages (both KS). AtKS4 the ‘old’ model point score of A*=8, A=7, etc. is used).

A KS4 Est/Act Difference of 0.27, e.g., represents an actual point score that is 0.27 higher than might be expected under the SX model.

A KS3 Est/Act Difference of 0.74, e.g., represents a mean NC level that is 0.74 higher than might be expected under the SX model.

An Est/Act Difference of 8.4%, e.g., represents the percentage of pupils attaining a given threshold is 8.4% higher than might be expected under the SX model.

NB: If a grouping is not shown then the value-added score or trend is not significant (pupils makingprogress consistent with that of similar pupils in similar schools);
Differences between overall (e.g. NC level or Points Score) and threshold(e.g. L5+, 5A*C) could indicate a substantial proportion of pupils achieving just above or just below the threshold orthe school may be concentrating its efforts on borderline pupils to the detriment ofhigher or lower attaining pupils.

CONTEXTUAL VALUE-ADDED INDICATORS (KS23 CORE SUBJECTS) [SX]:
An overview of VA for each core subject at KS3 over the last 3 years. For each indicator, the school's value-added score is ranked against all maintained schools nationally (percentile rank); differences are highlighted where significant.

The analysis also shows the number of boys and girls split into three bands based upon KS2 attainment (Lower, Middle, Upper). Pupil groups with significant value-added scores(taking into account the number of pupils) are highlighted. Where a pupil was absent from a test their TA level is used if available.

NB:In the right-hand section of the analysis showing performance of boys girls byability band the number represents the number of pupils in this group – not the percentile rank.

KS23 & KS2/34 ESTIMATES SUMMARY:
Includes estimates for future years which take account of theprior attainment of pupils in each cohort, the school’s context andthe school’s value-added performance in the previous 3 years. Estimates shown include currentstatutory targets.

Estimates for each threshold areshown in order from lowest to highest (4 estimates for each year): “A” (PA model); “B” (SE model), “D” (‘top quartile’ - based on the value-added performance, last year, of schools at the 25th percentile); and afourth [boxed] estimate that takes into account the school's own value-added performance (SE) over the past 3 years (e.g. ifSE estimate is 65% and school’s SE value-added score over 3y is3% higher than this, then the boxed estimate would be 68%).

NB:Pupils not matched to prior attainment are not included - with highlevels of mobility, the cohort may have changed significantly since the PLASC data was collected (in which case, individual pupil estimates may need to be taken into account);
Since the‘TQ’ estimateisbased uponthe SE model, schools working in areas with a high degree of disadvantage may have a ‘TQ’ estimate lower than the ‘PA’ estimate;
Estimates are based on progress made by pupils in previous years - they do not incorporate any value-added improvement; for schools improving rapidly, the ‘boxed’ estimate based on 3y may be lower than one based on thelast year only.

In KS4, two sets of estimates are included: one based on pupils’ KS2 scores and one based on pupils’ KS3 scores. Where pupils have made below the expected level of progress in KS3, it is important to take the (higher) KS2 estimates into consideration (since pupil progress KS2-4 is the most important measure).

The full contextual SX model is not used to calculate estimates –factors such as pupil ethnicity, EAL and mobility are not taken into account (to avoid ‘labelling’ certain categories of pupils as under-achieving).

NB. This means that estimates for pupils facing advantageous factors (e.g. Indian heritage or low socio-economic disadvantage) may be low.

Estimates are not targets – they provide a starting point for discussion which leads to the setting of targets.

FFT DATABASE: ESTIMATES & VALUE-ADDED REPORTS

NOTE: before using the FFT split database for the first time on any computer, it is important first to install the “Amyuni pdf Converter” which is available in a zipped file alongside the database files on ROSE. Please refer to the document called READ ME FIRST within the zipped file for information about how to install the file.

FFT KEY STAGE 4 PUPIL SUBJECT ESTIMATE REPORTS

Under “Y11 Estimates”, the percentages are the probability of attaining the given threshold (where “CS” is the probability of attaining grade C in En, Ma and Sc). To the right are estimates of each pupil’s most likelyGCSE grades, based onKS 3 priorattainment.