DDM Implementation Brief on Parameter Setting

Suggestions, considerations and resources for determining if students have demonstrated high, moderate, or low growth.

Overview
Once a district has identified common assessments, and developed administration and scoring protocols, the next step is to define parameters. Parameters are the ranges of scores (see call out box to the right) on an assessment that indicate whether a student has demonstrated learning, growth, or achievement at expected rates, above expected rates, or below expected rates.Parameters provide a transparent definition ofwhat we expect from students.

Setting parameters may be a new practice for many educators, including even those with deep expertise in developing and administering assessments. However,all educators arefamiliar with grading student work, determining what types of work productsrepresent “A”level work, “B”level work, and so on. The knowledge and skills necessary to fairly grade student work and assessments are the same needed to set parameters. Both practices require being clear about expectations, considering all available information, and using professional judgment to make honest distinctions about student outcomes that meet, exceed, or fall short of expectations. Reminding educators of these similaritiescan improve their confidence in making important decisions about parameters.

Prior to Setting Parameters

Before setting parameters, districts are advised to have the following conditions in place:

Engage educators in identifying assessments that are well aligned to content and that provide valuable information to educators about their students
Develop clear protocols to administer and score assessments fairly and consistently across all students, classrooms, and schools.
Clearly communicatehow results from common assessments are used in the evaluation process. Specifically, that the Student Impact Rating is separate from the Summative Performance Rating and evaluators determine the rating by applying professional judgment to results from multiple measures over multiple years.

Suggested Next Steps

The recommendations in this brief can be helpful to districts as they proceed through the following stages of common assessments development:

Identifying an approach for setting parameters.
Ensuring that parameters are appropriate for all students.
Using parameters to support educator practice

Approaches to Determinating Parameters

There are two broad approaches to determining parameters: a normativeapproach involves setting parameters based on pre-determining the percentage of student scores that will fall into each category and acriterion approach defines parameters based on a fixed set of expecatations.Districts may use one approach for all common assessments, or use different approaches for different assessments.

Normative Approaches:

A normative approachinvolves setting a predetermined percentage of student scores that will fall into the high, moderate, and low growth categories. This approach supports comparisons across different assessments. For example, if a district determines that high growth is defined as the top third of student scores for all common assessments, all educators have a clear understanding of what that means; the art teacher and the physics teacher are using a common definition for high growth, even if they are using very different assessments.

Normative approaches can be based solely on the students who complete the assessment in a given district in a given year, or may use an expanded group of students by including students from other districts or who completed the assessment in prior years. In general, the more student scores added to the population, the more we understand about how students generally perform on the assessment. Therefore, when comparing their students’ results to the population, educators can be more confident that the parameters reflect meaningful differences in performance. For example, an educator would likely be more confident that a student is demonstrating high performance on an assessment if the student’s scores were in the top 20% of a national sample compared to the top 20% of the teacher’s class.

Normative approach based on current students: A normative approach that looks at only the performance of current students involves scoring an assessment and then rank ordering the results. Parameters are then applied to separate the scores into three groups: high growth, moderate growth, and low growth based on predetermined percentages. For example, a district might choose to use thirds. The lower third of scores would be designated as low growth, the middle third would represent moderate growth, and the higher third of scores would represent high growth. In this model, the parameters do not necessarily need to be in even thirds. Districts may instead choose to define low and high growth as the bottom and top 25% percent of scores respectively, leaving the middle 50%to represent moderate growth. Regardless of the specific percentages, the parameters are based on the predeterminedproportion and consistent across all educators using the same assessment. Educators may be familiar with the similar process of“grading on a curve.”

This approach has several advantages:

Data Requirements: This approach does not require pre-existing data about the assessment (i.e., past students’ scores), since parameters are based on pre-determined percentages for the high, moderate, and low categories. Educators might find this especially beneficial when trying out new assessments.
Comparability:This approach also supports comparisons across different assessments. For example, if a district sets the same normative parameters for all common assessments, evaluators and educators will be assured that all assessments have an equal level of difficulty because the same percentage of student scores will fall into the high, moderate, and low growth categories.

However, there are important drawbacks to this approach to consider.

Planning:Since the scores need to be rank ordered in order to determine which scores translate to high, moderate, and low growth, educators cannot know beforehand where the cut scores will fall.
Comparability: While one advantage of using a normative approach is that it assumes a consistent level of difficulty across all assessments, this is also potential a drawback. There may be real differences in the performance of students across different assessments that are hidden because the comparison group for students is limited to a single subject area.
Singletons:In some districts, there may be only one educator serving in a particular position. Since the percentage of student scores thatwill fall into the high, moderate, and low growth categories is predetermined, a normative approach that factors in only current students provides limited feedback to singletons. Concretely, a singleton who produces extraordinary growth in students would have the same percentage of student scores in the high category as every other singleton, making it difficult for an evaluator to draw conclusions about impact.
Yearly Comparisons:With a normative approach based only on current students, it can be challenging to see systematic high growth in agroup of students. For example, if three quarters of students made tremendous gains compared to students from previous years, an approach that breaks high, moderate, and low growth into even thirds would mask that growth because only the top 33% of scores would earn the high growth designation.That is, the same percentage of scores would be determined as high each year. By contrast, the same student results using a normative approach based on multiple years of student scores or student scores from multiple schools, districts, or states would allow an individual class’s high growth to shine through.

Normative approach based on a wider population: One way to address some of thedisadvantages of using a normative approach is to base norms on a larger group of students. Most educators are familiar with the Student Growth Percentiles (SGPs) calculated by ESEfor statewide assessments. SGPs are an example of a normative approach based on a group of students from multiple districts. Although thereis a set percentage of students that are determined to have demonstrated high growth, there is no set percentage for each district. As a result, it is possible for all students in a district to demonstrate high growth on the state assessment, while at the state level the percentages of student scores that fall into the three categories are fixed.

A larger reference population can be geographical or temporal. Districts using commercial assessments may be able to use a national norm group to provide a better reference point for defining high, moderate, or low growth. Districts can even use this approach with district-developed assessments by looking at student results across multiple years.

All normative approaches provide the advantage of being able to set consistent definitions of high, moderate, and low growth for all content areas, regardless of the assessments used and their various point scales and scoring processes. Considering a wider population of student scores provides the following additional advantages:

Planning:Parameters based on either prior years or a large population are more predictable and better allow educators to plan ahead.
Singletons: Using parameters based on a wider population is a great opportunity for singletons to take advantage of the power of common assessments. Looking at results from a wider population may allow an educator to identify areas of strength or weakness that they were not able to determine before. For example, foreign language teachers might consider one of the national assessments identified as a potential common assessment by the Massachusetts Foreign Language Association. These assessments allow educators to see how their students compare to a group of students from across the country.
Yearly Comparisons: Using parameters informed by results from prior years, educators are able to make comparisons across multiple years. For example, if more students in a teacher’s class demonstrated high growth this year than in previous years, he/she might think about what new instructional strategies could have led to this change and build on them in subsequent years.
Competition: By using a wider population of scores to determine parameters, the impact of any individual student’s score on the population is diminished.For example, if parameters are cut scores based on a national population, it is possible for all students in a given district to perform at the high growth level, whereas if parameters are based solely on the current students in a district, some percentage of students scores necessarily fall into the low growth category.

However, this type of normative approach is not without potential drawbacks:

Data Requirements:Collecting and analyzing data from a wider population, be it scores from multiple years or multiple locales, requires personnel time. Even for commercial assessments, data may not be structured in a way that easily informs the parameter-setting process.
Changes in Assessment: Appropriately, many districts and educators will want to make changes to their common assessments from year to year. If using a normative approach based on a wider population to set parameters, districts will have to ensure that educators are not discouraged from making necessary improvements to assessments. If modifications are made, careful consideration must be given to determine whether the changes are significant enough to warrant re-setting the parameters.

Criterion Approaches:

In contrast to normative approaches, criterion approaches involve educators using professional judgment to definehigh, moderate, and low growth. The advantage of this approach is that growth is considered in terms of the learning that a student demonstrates as opposed to how a student’s score on an assessment relates to other students’ scores. Educators may find a criterion approach more authentic than a normative approach because it involves engaging a group of professionals in the process of thinking through how they expect students to perform on the assessment. However, since the definition is based on professional judgment, it may be harder to articulate and interpret the expectations for students embedded in the parameters. For example, with a normative approach, a district could make the statement, “High growth means the top 25% of student scores.” Using a criterion approach, the statement is less cut and dry, “High growth reflects the range of scores that educators determine is representative of exceeding expected performance.” Comparisons across different assessments in different content areas ismore challenging with a criterion approach because the criteria for demonstrating high, moderate, and low growth depend on the specific assessment.

Since criterion approaches require the use of professional judgment, they require that educators put forward their best thinking. One district called the parameters they developed during the first year “hypothesized parameters”to make it explicit that expectations for what constitutes moderate growth may need to be refined in the future.

How should educators make determinations about what type of student work represents high, moderate, and low growth? Some have found it useful to ground decisions about parameters in discussions about the specific assessment items and their knowledge of student learning progressions. The video example on page 7 illustrates how a 5th grade teacher might set parameters for a pre- and post- test in mathematics using this strategy. The educator in the video uses his understanding of the Curriculum Frameworks, as well as his knowledge of past students to identify score ranges that represent three different groups of students: students entering 5th grade with math skills that are below grade level, at grade level, and above grade level. Using those three groups, the educator thinks about which specific items on the assessment each group would likely answer correctly on the pre-test and then considers which additional items each group would need to answer correctly on the post-test to meet his expectations for a year of learning.

Setting parameters can be a challenge for many educators because it requires a shift in their thinking. Most educators are adept at defining expected achievement in their classrooms. However, they mayhave less experience thinking about expected growth. While this is a challenge, the shift to thinking about growth is important work because it provides an opportunity to think about the learning of all students, regardless of where they started the year.

Criterion approaches have several advantages:

Alignment: Perhaps the greatest advantage of using a criterion approach is that learning is defined in relation to standards instead of the performance of other students. Using a criterion approach can help shift conversations away from scores and to student learning.
Data Requirements:While previous data can support educators in setting parameters using a criterion approach, it is not a requirement. Educators can use experience with similar tasks, items, or problems to inform the parameters they set.
Planning: Parameters should be determined prior to using the assessment. This allows for parameters to serve as a planning tool for educators. By having a clear understanding of what skills students are expected to demonstrate over a course or year, educators can backwards plan appropriate lessons and assessments to ensure they are on track for students to meet those expectations.
Singletons: One of the important advantages of using common assessments to inform educator practice is that they can relieve the isolation that some educators experience. By looking at the results of students from other classrooms, an educator can begin to understand his/her relative strengths and weaknesses. A singletonteacher using a criterion approach does not need to set parameters alone. For example, a ceramics teacher might invite the two other art teachers in the district to help develop parameters with him/her.In fact, the process may help support a broader and more cohesive approach to arts education and assessment.

Yearly Comparisons: Since parameters are not based on student scores, an educator can look across years to see how they have made improvements from year to year.
Competition: Since determinations of high, moderate, and low growth are not based on other scores, students who demonstrate high growth do not impact the ability of other students to also demonstrate high growth.

However, criterion approaches also have drawbacks that educators must consider:

Time: Compared to normative approaches, criterion approaches are time intensive. They require time for different educators to share their perspectives and groups to arrive at consensus-based decisions.
Experience: Since the process of determining parameters using a criterion approach involves educators’ professional judgment, it relies on educators with experience sharing what they know about past students’ learning progressions. Teams should plan to revisit parameters in the first few years to capitalize on increased educator experience and knowledge in the parameter refinement process.
Comparability: A criterion approach has the advantage of tying parameters closely to the standards and learning objectives for each content area. Unfortunately as a result, it can be difficult to arrive at a clear cross-content definition of moderate growth. This can make it difficult to know whether all groups of educators are using comparably rigorous definitions of moderate growth As a result, districts need to pay close attention to comparability across different measures.

Applying Student Results to Parameters

Setting parameters at the student level:

For most assessments, it makes sense to determine parameters at the student level, whether using a normative or criterion approach. In other words, parameters that will be applied to each student’s work to determined whether the student demonstrated high, moderate, or low learning, growth, or achievement. When looking across the assessment results, evaluators can ask, “Have more than half of the educator’s students demonstratedeither high or low growth?” If so, they will look to see if the results are consistent with a pattern of high or low growth in the educator’s studentson other measures and over multiple years. This information will ultimately inform the educator’s Student Impact Rating (see additional guidance on determining an educator’s Student Impact Rating).