The Accountability Game: From Rules and Regulations to Real Improvement
by Dr Tony Townsend
Chair, Department of Educational Leadership
College of Education
Florida Atlantic University
One factor common to most education systems in the past decade has been the search for a better way of ensuring high levels of achievement for all students. Education systems from the USA to Uganda, from Austria to Australia and from the United Kingdom to Kazakhstan have focused their attention on improving the quality of education in their countries. Much of the work that has happened internationally has focused on increasing the levels of accountability for teachers and school leaders, for schools and for school systems. Many different strategies have been used, most with only marginal success.
The data that is available to politicians and the public is now monumental. It is now possible, with the judicious use of the internet, to compare country with country, state with state within a particular country, districts or regions within a particular state, individual schools with other schools in a single district and finally individual classrooms with other classrooms in a single school. By extrapolating one from the other it is now possible to make a rough comparison of the performance of students in a single classroom, with those in a classroom on the other side of the world. To show how this works, let me take the example of the country in which I am working at the moment, the USA.
Recent international comparative studies have compared the performance of the USA with other countries for fifteen year old students. These have included the Third International Mathematics and Science Study - TIMMS - see - in 1995, where 41 countries participated and the US was placed 17th in science and 28th in mathematics.
There is also the more recent Program for International Student Achievement – PISA (2000), conducted by the Organisation for Economic Co-operation and Development (OECD) - see - that tested student literacy in reading, science and mathematics. Thirty one countries were involved and the US was placed 15th in reading, 14th in science and 19th in mathematics. Table 1 provides a comparison of some of the actual scores for reading, mathematics and science literacy for the data collected in 2000.
Reading / Mathematics / ScienceFinland / 546 / Japan / 557 / Korea / 552
Canada / 534 / Korea / 547 / Japan / 550
New Zealand / 529 / New Zealand / 537 / Finland / 538
Australia / 528 / Finland / 536 / United Kingdom / 532
Ireland / 527 / Australia / 533 / Canada / 529
Korea / 525 / Canada / 533 / New Zealand / 528
United Kingdom / 523 / Switzerland / 529 / Australia / 528
Japan / 522 / United Kingdom / 529 / Austria / 519
Sweden / 516 / Belgium / 520 / Ireland / 513
Austria / 507 / France / 517 / Sweden / 512
USA / 504 / USA / 493 / USA / 499
OECD Average / 500 / 500 / 500
Table 1: Mean Scores for Reading, Mathematics and Science Literacy
The scores enable a comparison to be made between the USA and the best performing countries in the study. The table shows that the USA is around the OECD average for all three areas. In more specific terms, as well as the countries listed in the table, the US is just behind countries such as Austria, Belgium, Iceland, Norway and France in reading, behind Ireland, Norway and the Czech Republic in mathematics and behind Norway and France in science.
The PISA study also provides a detailed analysis of reading performance in 2000, and categorized readers into five different levels, from level one being virtually illiterate to level five where people were able to read and manipulate language at a high level. Table 2 shows the percentages of students that are performing at the various levels of reading literacy.
Below 1 / Level 1 / Level 2 / Level 3 / Level 4 / Level 5Finland / 2 / 5 / 14 / 29 / 32 / 18
Korea / 1 / 5 / 19 / 39 / 31 / 6
Canada / 2 / 7 / 18 / 28 / 28 / 17
Japan / 3 / 7 / 18 / 33 / 29 / 10
Ireland / 3 / 8 / 18 / 30 / 27 / 14
New Zealand / 5 / 9 / 17 / 25 / 26 / 19
Australia / 3 / 9 / 19 / 26 / 25 / 18
USA / 6 / 12 / 21 / 27 / 21 / 12
OECD Average / 6 / 12 / 14 / 29 / 32 / 9
Table 2: Percentages of students at various levels of reading literacy
The table shows that the USA performed quite well in comparison to other countries (7th) for the percentage of students who read at the top levels, with 12% of students reading at level 5, above the OECD average of 9% (the top country was New Zealand with 19%). But it performed very poorly for the percentage of students at the bottom two levels with 18% of students being at these two levels, the same as the OECD average (with the best country being Korea with 6%).
Across the OECD, on average, girls outscored boys by around 32 points (or 6 1/2%), with Finland (51 points) and New Zealand (46 points) having the highest discrepancies and Korea the lowest (14 points). The USA was slightly lower than average with 29 points discrepancy.
These figures largely confirm what other studies, such as the Goals 2000 research in the USA have also found - see - namely that only around one in three American students have a level of proficiency in reading that will serve them well in future study and work and that boys are not doing as well as girls. This is confirmed by other data, such as that collected by the Florida Department of Education that shows that despite over 60% of high school graduates moving onto post-secondary education, only just over half of them will actually complete a post-secondary qualification.
The Goals 2000 research also allows us to take the national figures and break them down into levels that allow comparisons at the state level. States are being compared with other states, and ethnic groups with other ethnic groups, through this study.
In the last year or so, I have spent time in both Michigan and Florida, so I will use these two states to demonstrate how the comparisons might be made. Both states can compare themselves against the nation as a whole, or against other states that might perform best, or worst, in the country. In addition, the performance of the various ethnic groups of White, Hispanic and African-American children can be compared to each other and to the performance of similar groups in other states or the nation.
In general terms, Michigan is at, or at or slightly above average in most areas measured and Florida is below average in most areas. The following table provides some details about how Florida and Michigan perform, in comparison to the whole country, and in comparison to the best and the worst states in the country, in areas related to student achievement and success. Perhaps surprisingly, despite Florida being below average in all discipline areas, the percentage of students who graduate is higher than the national average.
The curriculum columns indicate the percentage of students at that level who have been assessed as proficient or above in the discipline area. The last two columns indicate the percentage of students who complete High School and the percentage of secondary teachers who report disruptions in class. It is interesting to note that the best performing state in terms of reading is Connecticut, for mathematics, it is Minnesota and for science, it is Maine. However, the worst ‘state’ for all areas of student learning is the District of Columbia.
Reading / Reading / Mathematics / Mathematics / Science / High school completion / % of teachers reporting disruptions in classGrade 4 / Grade 8 / Grade 4 / Grade 8 / Grade 8
1998 / 1998 / 1996 / 1996 / 1996 / 1998 / 1994
Nation / 29 / 33 / 21 / 24 / 29 / 83 / 46
Michigan / 28 / - / 29 / 28 / 32 / 90 / 46
Florida / 23 / 23 / 15 / 17 / 21 / 85 / 58
Best state / 46 / 42 / 34 / 40 / 41 / 95 / 33
Worst state / 10 / 12 / 6 / 6 / 5 / 75 / 65
These comparisons suggest that the composition of the student cohort, and whether the student is in an urban or rural area, might be critical factors to success. The goals 2000 data set provides some support for these suggestions as well. The following table provides an analysis of reading scores across the country, broken down for gender and ethnic background. Although it is clear there has been progress across the country from 1992 to 1998, there are still some groups severely disadvantaged, particularly males and people from African-American and Hispanic backgrounds.
Grade 4 reading 1992 / Grade 4 reading 1994 / Grade 4 reading 1998 / Grade 8 reading 1992 / Grade 8 reading 1994 / Grade 8 reading 1998USA / USA / USA / USA / USA / USA
Whole group / 29 / 30 / 31 / 29 / 30 / 33
Male / 22 / 22 / 28 / 22 / 21 / 27
Female / 28 / 29 / 33 / 33 / 35 / 40
White / 31 / 32 / 39 / 34 / 34 / 41
Black / 7 / 7 / 10 / 8 / 8 / 12
Hispanic / 13 / 11 / 13 / 13 / 13 / 15
It is clear from the data that the group that has benefited the most in reading from the Goals 2000 effort is white girls and one suspects that the groups least touched have been African-American and Hispanic boys.
Although the same level of analysis is not provided for mathematics as there was for reading, it is clear from the data that similar patterns emerge. There has been some national improvement in mathematics from 1992 to 2000, but one suspects that the patterns established above may also hold true for this discipline area as well (except it is white males that have improved most in Mathematics).
Grade 4 Math 1992 / Grade 4 Math 1994 / Grade 8 Math 1992 / Grade 8 Math 1994 / Grade 12 Math 1992 / Grade 12 Math 1994USA / USA / USA / USA / USA / USA
Whole group / 13 / 18 / 20 / 25 / 13 / 16
Male / 14 / 20 / 21 / 25 / 16 / 18
Female / 13 / 17 / 18 / 24 / 10 / 14
White / 17 / 23 / 24 / 32 / 16 / 19
Black / 2 / 3 / 6 / 3 / 2 / 3
Hispanic / 5 / 6 / 6 / 8 / 4 / 6
In summary, the Goals 2000 activity has had some impact on student achievement in the United States, but not as much as is necessary for the United States to become a world leader, if the TIMMS and PISA reports are any indication. All of the national data suggests that further work needs to be undertaken, particularly in areas where high poverty, inner city and non-white populations live and go to school.
Within individual states, such as Michigan or Florida, it is possible for one school district to be compared with others and with the state as a whole. So some school districts can identify themselves as performing substantially better, others can indicate that they are about the same as, and others are seen to performing less well than the state averages. In the first instance, state tests such as the MEAP (Michigan Educational Assessment Program) test in Michigan and the FCAT (Florida Comprehensive Achievement Test) provides comparative performance for various school districts in the discipline of reading. The first set indicates the level of reading achievement for various individual school districts in the Genessee County School District in Michigan and the second set indicate the level of reading performance in various school districts in Florida.
Percentage of students at satisfactory or higher
MEAP reading scores 2002, Michigan
Grade 4 / Grade 7Flushing / 75 / 65
Bendle / 63 / 31
Beecher / 30 / 21
Flint / 29 / 21
County / 54 / 47
Michigan / 57 / 61
Percentage of students at lowest (level 1-2) and highest (level 4-5) reading levels
FCAT scores 2003, Florida
Grade 4 Reading / Grade 8 Readinglevel 1-2 / level 4-5 / level 1-2 / level 4-5
Florida / 40 / 29 / 52 / 19
St John’s / 26 / 41 / 35 / 40
Santa Rosa / 21 / 44 / 32 / 42
Broward / 40 / 30 / 49 / 20
Palm Beach / 42 / 28 / 52 / 19
Miami-Dade / 49 / 24 / 64 / 13
Gadsen / 53 / 15 / 75 / 4
Madison / 56 / 15 / 60 / 11
It becomes clear that within a single state, or even a single county, there are a wide range of outcomes achieved by individual school districts. A similar comparison can be made in mathematics as well. The tables below indicate the level of mathematics success for various Michigan school districts within Genessee county and the second set for school districts in Florida.
Percentage of students at satisfactory or higher
MEAP mathematics scores, Michigan
Grade 4 / Grade 7Flushing / 86 / 77
Bendle / 63 / 43
Beecher / 26 / 17
Flint / 44 / 10
County / 61 / 48
Michigan / 65 / 54
Percentage of students at lowest (level 1-2) and highest (level 4-5) mathematics levels
FCAT scores 2003, Florida
Grade 4 Math / Grade 8 Mathlevel 1-2 / level 4-5 / level 1-2 / level 4-5
Florida / 45 / 20 / 44 / 24
St John’s / 33 / 31 / 33 / 33
Santa Rosa / 30 / 31 / 25 / 40
Broward / 40 / 23 / 40 / 27
Palm Beach / 48 / 20 / 43 / 17
Miami-Dade / 52 / 16 / 58 / 16
Gadsen / 65 / 5 / 61 / 10
Madison / 71 / 8 / 67 / 12
If we take the case of Genessee County in Michigan, by making some additional analysis we can start to see some of the issues involved. For instance, if we identify the most successful school districts on a map of the county in green (lighter circles) and the least successful districts in blue (darker), then the following picture emerges.
What we can see is the schools in the inner urban area are those that perform least well and those in the outer suburban ring perform best. This pattern is duplicated in many cities across the United States.
Of course the difficulty with this type of comparison is that it takes nothing except raw data into account. So the size of the district, the complexity of the issues facing the district, the level of funding allocated to each child’s education, the cultural and social mix of the students and the level of urban and rural distribution, are all washed out in such a comparison.
Yet this analysis can be taken even further, so that individual schools within a particular district can be compared with each other and with the state as a whole. The list of how well an individual school has gone on the various FCAT tests is available for all to see on the internet. Some are identified as having higher and others as lower levels of reading and mathematics achievement at elementary school level. This can be seen in the listing below, taken from Broward county, in Florida, a county that performs slightly better than the state as a whole.
Grade 4 reading / Grade 4 mathematicslevel 1-2 / level 4-5 / level 1-2 / level 4-5
Florida / 40 / 29 / 45 / 20
Broward (county) / 40 / 30 / 40 / 23
Embassy Creek / 8 / 65 / 12 / 57
Bayview / 19 / 56 / 14 / 49
Everglades / 20 / 58 / 17 / 53
Sunland Park / 87 / 5 / 82 / 3
Robert Markham / 79 / 4 / 77 / 4
Thurgood Marshall / 75 / 3 / 78 / 4
What we see is that some elementary schools are doing comparatively well and others are doing really poorly within the county in both reading and mathematics. However, we are also able to make an analysis on the basis of more specific information, such as race. The percentage of students from various races in the identified schools is contained below.
white / hispanic / black / otherEmbassy Creek / 73 / 15 / 5 / 7
Bayview / 87 / 9 / 1 / 3
Everglades / 54 / 36 / 3 / 7
Sunland Park / 0 / 1 / 99 / 1
Robert Markham / 3 / 28 / 68 / 1
Thurgood Marshall / 0 / 3 / 96 / 1
We can see here that the low levels of achievement are associated with high levels of ethnic minority children, and in particular, children from African-American background. The same type of analysis can be conducted at the grade 8 level.
Grade 8 reading / Grade 8 mathematicslevel 1-2 / level 4-5 / level 1-2 / level 4-5
Florida / 52 / 19 / 44 / 24
Broward (county) / 49 / 20 / 40 / 27
Westglades / 24 / 39 / 12 / 55
Sawgrass / 25 / 37 / 18 / 49
Tequestra Trace / 28 / 37 / 15 / 50
Pompano Beach / 67 / 10 / 58 / 17
Attucks / 63 / 9 / 60 / 11
Henry Perry / 68 / 8 / 63 / 8
white / hispanic / black / other
Westglades / 80 / 9 / 6 / 5
Sawgrass / 64 / 18 / 11 / 7
Tequestra Trace / 52 / 39 / 3 / 6
Pompano Beach / 31 / 16 / 51 / 2
Attucks / 27 / 19 / 52 / 2
Henry Perry / 6 / 21 / 69 / 4
From this overview, we might make an assumption that the ability to succeed in FCAT testing is very much racially biased. But is also needs to be said that the pattern of achievement based on racial background in Broward County is very similar to that of the nation as a whole.
An analysis of how students in a class in a particular school in Broward county are learning in comparison to students in a school in Finland, for instance, can be backward mapped. On average, schools in America achieve 8% less than schools in Finland. Schools in Florida achieve, on average, about 15% less than schools in the rest of the United States and schools in Broward county, on average, about 12% less than students in other states. There are some schools in Broward, where students achieve about half of what other students achieve. If we put all of this together, then students in the average school in Finland are substantially better at reading than most students in Broward county.
The analysis shows that there is one specific area where America might feel really concerned about its performance. It is the differential scores based on family background, shown by the PISA analysis and the racial disadvantage shown by the state and local data. From the PISA information, the difference in mean scores based on family background between the top 25 percent of students and the lowest 25% of students is 90 points. Some comparative figures for countries include Korea 33 points, Finland 53 points, and Canada, 66 points. This suggests that the students who come from the most well-off families in the USA perform, on average, around 20% better than students from the poorest backgrounds.
From the Goals 2000 data and the locally collected data we can see that the level of performance of African-American children in particular is something that must be addressed if the USA is to compete with the best countries in reading or mathematics. This can only happen (for reading) by ensuring that those currently reading poorly (18%) are able to read better so that the US approaches the top three countries (around 8%) at this level. If the goal is to ensure that all students have high levels of achievement, then decreasing this difference becomes a key issue.
However, it could be argued that the response to this situation has been less than encouraging. There have been rules and regulations adopted at all levels of the education system in the USA. In the US, the charter school, the public school outside public schools, is the preferred option of governments, both Federal and State. This development, together with the gradual introduction of voucher systems in some states, has been justified by the argument for increasing choice for parents. What doesn’t seem to be said in any of the arguments is that some parents have more choices than others.
However, the choice movement does not yet seem to have worked. A recent report from the American Federation of Teachers (AFT, 2002), that considered the impact of charter school introduction on student achievement, concluded:
- Charter schools contribute to the racial and ethnic isolation of students….
- Charter school teachers are less experienced and lower paid than teachers in other public schools….
- Charter schools generally obtain funding for the type of students they educate that is comparable to other public schools….
- Charter school students generally score no better (and often do worse) on student achievement tests than other comparable public school students.
- Charter schools have not been held to the “bargain” they made—trading freedom from rules for increased accountability…In general, the schools have taken the freedom but have not delivered on their promise to produce results.
- Charter schools were supposed to experiment with new curricula and classroom practices, but they have proven no more innovative than other public schools….
- School districts with growing enrollments feel little competitive pressure and sometimes view charter schools as a solution to over- crowding….
- The problems associated with charter schools identified in this report are exacerbated in the charter schools operated by for-profit companies….
- The justification for charter schools has moved from one that is based on education and innovation to one that is based on choice and competition.
It is interesting, given these statements, that one of the key components of the No Child Left Behind Act is the support it gives to charter schools (see There is no evidence provided that this will make a difference. If we take the analysis of the MEAP scores in Genessee county, Michigan, for instance, we might suggest that the performance of charter schools in that county closely reflects that of regular public schools, some seem to have improved over time, others have stayed the same and others have regressed. Certainly the government preference for charter schools has not been based on any achievement data.