Summary of State Proposed Growth Models Undergoing Peer Review in 2005-2006

In November 2005, The United States Department of Education requested state proposals for accountability models that incorporate measures of student growth. States were encouraged to submit proposals to the Department for using growth models to demonstrate accountability under the federal No Child Left Behind (NCLB) Act. States submitting proposals were required to show how their growth-based accountability models satisfy NCLB alignment elements and foundational elements explained in the November letter. Those elements included the following:

NCLB Alignment Elements

  1. The accountability model must ensure that all students are proficient by 2013-14 and set annual goals to ensure that the achievement gap is closing for all groups of students.
  2. The accountability model must not set expectations for annual achievement based upon student background and school characteristics.
  3. The accountability model must hold schools accountable for student achievement in reading/language arts and mathematics.

Foundational Elements

  1. The accountability model must ensure that all students in the tested grades are included in the assessment and accountability system. Schools and districts must be held accountable for the performance of student subgroups. The accountability model includes all schools and districts.
  2. The State’s assessment system, the basis for the accountability model, must receive approval through the NCLB peer review process for the 2005-06 school year. In addition, the full NCLB assessment system in each of grades 3-8 and in high school in reading/language arts and math must have been in place for two testing cycles.
  3. The accountability model and related State data system must track student progress.
  4. The accountability model must include student participation rates in the state assessment system and student achievement on an additional academic indicator.

The Department received 20 proposals in February 2006. Seven (Hawaii, Maryland, Nevada, New Hampshire, Ohio, Pennsylvania, and South Dakota) of the twenty proposals applied to evaluate growth in the 2006-2007 school year, so they were not evaluated in the first round of reviews. Of the 20 submissions, eight (Alaska, Arizona, Arkansas, Delaware, Florida, North Carolina, Oregon, and Tennessee) were sent to be evaluated by a peer review group. The five states that submitted but were not sent by USDE for peer review included Colorado, Indiana, Iowa, South Carolina, and Utah. During the review of the eight proposals, the peer review group identified additional elements on which they evaluated proposals. Based on their Crosswalks paper, the peer review group determined that states should

  1. incorporate available years of existing achievement data, instead of relying on only two years of data,
  2. align growth timeframe with school grade configuration and district enrollment,
  3. make growth projections for all students, not just those below proficient, and
  4. hold schools accountable for same subgroups as they did under status model.

They determined that states should not

  1. use wide confidence intervals,
  2. reset growth targets each year, and
  3. average scores between proficient and non-proficient students.

The goal of this paper is to identify distinguishing practical and psychometric features of the submitted proposals. Since states function within unique political contexts and have unique data structures for their assessments, no one growth model best meets all states’ needs. By comparing and contrasting growth model proposals on specific features, states looking for a growth model can use this information to find a model that best fits their particular circumstances. States with existing growth models might use this information to find similar models for which they can compare their model.

The tables in Appendix A summarize the eight proposals that were reviewed by the committee on practical and psychometric features.

Growth Model Features

Pilot Approved in 2006 – State growth model proposals approved by the USDE in 2006. These states will be able to include growth measures in accountability decisions for the 2005-2006 school year and are expected to present data that show how the model works compared with the current AYP model.

Resubmitting in 2007 – This field indicates if the state plans to resubmit their growth model proposal in 2007.

Name of Growth Measure – For states that have named their measure of growth, the name is listed.

All Students at Proficiency or On Track by 2013-2014 – This field indicates if states’ growth model proposals will have all students at proficiency or on track to be proficient by 2013-2014.

Scores on Vertical Scale – This field indicates whether the state has a vertical or developmental scale underlying its assessment.

Vertically Aligned Standards – This field indicates whether the state vertically aligned its performance standards.

First Year All Grades Tested – The years listed in this field indicate the first time that the state assessed students in the NCLB grades in reading and mathematics.

Includes Grade 3 Students – States that calculate growth for grade 3 students are noted in this field.

Includes Students Without Prior Year Score – States that measure growth for students who are missing the prior year test score are noted in this field. As examples, states may calculate growth for students without the prior year score by using scores from more than one year prior, the mean of the grade-level scores the prior year, or a pretest score to compute growth.

Includes SWD Taking Alternate Assessment – For states that have an alternate assessment for students with disabilities, this field indicates whether the proposed growth model can be applied to scores on that alternate assessment.

Includes ELL Taking Alternate Assessment – For states that have an alternate assessment for English Language Learners, this field indicates whether the proposed growth model can be applied to scores on that alternate assessment.

Grades for which Growth is Calculated – Some states calculate growth for grades 3-12, whereas other states only calculate growth for a subset of grades. This field indicates the grades in which states are calculating student growth.

Number of Years for Students to Reach Proficiency – States use different numbers of years for students to reach proficiency. This field indicates the number of years that states use in their growth-based accountability models.

Growth Tracked Only for Below Proficient Students – This field indicates if state growth models are only applied to students who are below proficient.

Use Confidence Interval – This field indicates if the state will use a confidence interval in any way in the growth-based accountability model. Some states propose using a confidence interval around the percent of students reaching proficiency or meeting growth targets and others propose using a confidence interval around students’ growth estimates.

Averaging of Calculations – This field identifies states that average calculations over more than one year or over students within subgroups.

Incorporates Available Years of Achievement Data – This field indicates whether states use all years of previous achievement data or if the state uses only two years of available data.

Growth Target Timeline Starts Over Each Year – This field indicates if states identify a starting year for student growth calculations and define time to reach proficiency by this starting year or whether states recalculate growth each year.

Growth Target Timeline Aligns with Grade Configuration – This field indicates if the number of years in which students are expected to reach proficiency matches with the grade configurations in the state. For example, if students below proficiency are expected to grow at a speed so that they will reach proficiency in three years, does the state set the three years in a way that maps the three years onto the grades that each school unit serves, such as the three middle school grades.

Accounts for Students Falling Off Track – This field identifies states that use growth to identify students who may be above proficiency but whose growth will likely result in their falling below proficiency at a later date.

Minimum N Same as for AYP Status Model – This field identifies states that apply a minimum sample size rule to the growth model. For example, many states designate a minimum N (or sample size), such that the state does not include a subgroup in AYP calculations or growth model calculations if that subgroup has fewer students than the minimum sample size.

Growth Applied after Status and Safe Harbor Provisions – This field indicates states that use growth as an additional method for meeting AYP after applying the status model and the safe harbor provisions.

Growth Reported at Individual Level – This field notes states that report growth on individual student reports or at the individual student level.

STATE
Variable / Alaska
AK / Arizona
AZ / Arkansas
AR / Delaware
DE
Pilot approved in 2006 / No / No / No / No
Resubmitting in 2007
Name of Growth Measure / N/A / Measure of Academic Progress (MAP) / N/A / N/A
All students at proficiency or on track by 2013-2014 / Yes / Yes / Yes / Yes, on target only
Scores on vertical scale / No / Yes / Yes, Grades 3-8
Vertically aligned standards / Yes / Yes / Yes / Yes
First year all grades tested / 2001-2002 / 2004-2005 / 2005-2006 / 2001-2002
Includes grade 3 students / No / No / No / Yes
Includes students without prior year score / No / No / No / No
Includes SWD taking alternate assessment / No / Yes by changing performance levels / For all but 1.7% SWD / Yes
Includes ELL taking alternate assessment / No information found / No information found / For all but 0.9% LEP students / Yes
Grades for which growth calculated / 4-9; 10 / 3-8 / 4-8 / 3-10
Number of years for students to reach proficiency / 4 grades 4-9; 3 grade 10 / Min (3 years or Grade 8) / 4 / N/A
Growth tracked only for below proficient students / Yes / Yes / Yes / No
Uses confidence interval / 99% / 99% around subgroup percents / Yes for overall AYP only, unknown level / 98%, one-tailed
Averaging of calculations / Yes, uniform / No / Yes, up to 3 years starting 2007-2008 / Yes
Incorporates available years of achievement data / No / No / No / No
Growth target timeline starts over each year / Yes / No, unless discontinuously enrolled / Yes / Yes
Growth target timeline aligns with grade configuration / No / Yes / No / No
Accounts for students falling off track / No / No / No / No
Minimum n same as for AYP status model / Yes, n=20 for subgroups, n=40 for SWD or LEP / No, report growth for n=10; need max (90% of students in status model or 40 students) for subgroups / Yes, n=40 for subgroups / Yes, n=40 for subgroups
Growth applied after status and safe harbor provisions / Yes / Yes / Yes / Yes
Growth reported at individual student level / Yes / Yes / Yes / No
STATE
Variable / Florida
FL / North Carolina
NC / Oregon
OR / Tennessee
TN
Pilot approved in 2006 / No / Yes / No / Yes
Resubmitting in 2007
Name of Growth Measure / N/A / ABCs Growth Model / N/A / N/A
All students at proficiency or on track by 2013-2014 / Yes / No / Yes / Yes
Scores on vertical scale / Yes / Yes / Yes / Yes
Vertically aligned standards / Yes / Yes / Yes / Yes
First year all grades tested / 2000-2001, grades 3-10 / Mid 1980s, new edition in 2005-2006 / 2004-2005 / 1992 (3-8), 2001 (high school)
Includes grade 3 students / No / Yes, using a pretest score / Yes, in terms of current year scores / Yes, in terms of status only
Includes students without prior year score / No / Yes, if pretest score on change scale / Yes, in terms of current year scores / Yes, in terms of status only
Includes SWD taking alternate assessment / No / Yes, in terms of status only / Yes, in terms of status changes only / Yes, in terms of status only
Includes ELL taking alternate assessment / No / Yes, if meet Full Academic Year and have previous scores on change scale / Yes, in terms of status changes only / Unclear
Grades for which growth calculated / 3-10 / 3-8 / 3-8, 10 / 3-8
Number of years defining on track / 3 years / 4 years / 4 years / 3 years
Growth tracked only for below proficient students / No / No / No / No
Uses confidence interval / No / No / Yes, around school average growth / Yes, 95% CI
Averaging of calculations / No / Yes, across students / Yes, across students / No
Incorporates available years of achievement data / Yes / No, includes previous or 2 previous scores / No, planned for future / Yes
Growth target timeline starts over each year / Yes / No, reset only when move to new LEA / Yes / No
Growth target timeline aligns with grade configuration / No / No / No / Yes
Accounts for students falling off track / No / Yes / Possibly / Yes
Minimum n same as for AYP status model / Yes, n=30 or 15% of tested population / Yes, n=40, yet growth calculated when n < 40 / No, n=21 scores over 2 years for growth / Yes, greatest of (n=45 or 1%)
Growth applied after status and safe harbor provisions / Yes / Yes / Yes / Yes
Growth reported at individual student level / Yes / Not to parents / Yes / Yes

Although there are a large number of possible ways to measure growth and design accountability systems, there are a limited number of methods that underlie those possibilities. The next section describes six model types and eight characteristics that differentiate the model types. A table is provided to summarize the information.

Model Descriptions

Improvement: The change between different groups of students is measured from one year to the next. For example, the percent of fourth graders meeting standard in 2005 maybe compared to the percent of fourth graders meeting standard in 2006. This is the only growth model described here that does not track individual student's growth. The current NCLB “safe harbor” provision is an example of Improvement.

Difference Gain Scores: This is a straightforward method of calculating growth. A student's score at a starting point is subtracted from the same student's score at an ending point. The difference or gain is the measure of an individual's growth. The difference scores can be aggregated to the school or district level to obtain a group growth measure. Growth relative to performance standards can be measured by determining the difference between a student's current score and the score that would meet standard in a set number of years (usually one to three). Dividing the difference by the number of years gives the annual gain needed. A student's actual gain can be compared to the target growth to see if the student is on track to meet standard.

Residual Gain Scores: In this model, students' current scores are adjusted by their prior scores using simple linear regression. Each student has a predicted score based on their prior score(s). The difference between predicted and actual scores is the residual gain score and it is an indication of the student's growth compared with others in the group. Residual gains near zero indicate average growth, positive scores indicate greater than average growth and negative sores indicate less than average growth. Residual gain scores can be averaged to obtain a group growth measure. Residual gain scores can be more reliable than difference gain scores, but they are not as easily integrated with performance standards in accountability systems such as NCLB because they focus on relative gain.

Linear Equating: Equating methods set the first two or four moments of the distributions of consecutive years equal. A student’s growth is defined as the student’s score in Year 2 minus the student’s predicted score for Year 2. A student’s predicted score for Year 2 is the score in the distribution at Year 2 that corresponds to the student’s Year 1 score. The linear equating method results in a function that can be applied year to year. If the student’s score is above the expected score, the student is considered to have grown. If the student’s score is below the expected (predicted) score, the student is considered to have regressed. Expected growth is defined as maintaining location in the distribution year to year.

Transition Matrix: This model tracks students’ growth at the performance standard level. A transition matrix is set up with the performance levels (e.g., Does not meet, Meets, Exceeds) for a given year as rows and the performance levels for a later year as columns. Each cell indicates the number or percent of students that moved from year 1 levels to year 2 levels. The diagonal cells indicate students that stayed at the same level, cells below the diagonal show the students that went down one or more levels and the cells above the diagonal show the students that moved to higher performance levels. Transition matrices an be combined to show the progress of students across all tested grades. Transition matrices are a clear presentation of a school's success (or lack thereof) in getting all students to meet standard.

Multi-level: This model simultaneously estimates student-level and group-level (e.g. school or district) growth. There is evidence that multi-level models can be more accurate than difference or residual gain score models. However, even though the statistics have been around for many years, only recently has the computing power, software and expertise been widely available. Therefore the results of this model appear to be more complex because the methods are still unfamiliar to many people.