Questions for Questar Assessment Team from PDK Presentation 11/12/15

Throughout the development and implementation of these new assessments, Questar will be working in support of the NYSED and New York State statutes, policies, and regulations, as coordinated with and at the direction of the State.

How can educators participate in Questar’s development of the test items, passage reviews and field testing?

As detailed in Component 1 of the original technical proposal, NYSED educators will be active participants in:

  • Item development (via item writer workshops)
  • Passage review
  • Item review
  • Range-finding
  • Performance Level Descriptor development
  • Standard setting
  • Review of scoring training materials
  • Local scoring of operational constructed-response test items

From start to finish, educators play critical roles in the lifecycles of the ELA and math assessments. Their involvement creates a concrete connection between what the students are taught in the classroom and what they are assessed on through this program.

Are there plans to link the assessments with teacher preparation programs (higher education)?

Linking assessments to teacher preparation programs was not part of the work Questar was initially contracted for, though the NYSED may have ideas regarding how to provide such connections.

What is the proposed length of the tests?

The length of tests at the time Questar originally proposed our solution was to be dependent on a variety of factors that would be finalized in collaboration with the NYSED (e.g., number of test forms and items, performance data, etc.). The NYSED will provide this information to educators and the public as it is finalized.

How is Questar determining the readability level of the tests?

Questar will determine the appropriate quantitative grade and readability levels by using the Degrees of Reading Power (DRP), Lexile, and Flesch-Kincaid measurements. Understanding that a quantitative approach is not the complete picture of a text, we will also determine the qualitative value using our text complexity worksheets, which we have individualized for informational and literary texts.

Will the field tests be embedded within the final assessments?

Yes. The test forms call for 7–8 field-test items to be embedded in each operational assessment form.

Will Questar be reducing the “voluntary” field tests?

At this time, the NYSED has yet to make any decisions on reducing stand-alone field testing, although the press release on July 9, 2015 stated, “Questar, Inc. will also provide computer-based testing (CBT) platforms that will help reduce the need for stand-alone field tests, and more importantly, help make our assessments even better instructional tools.” http://www.nysed.gov/news/2015/3-8-assessment-contract-awarded-questar-inc

How does Questar plan to reduce field test fatigue?

The NYSED has indicated that they are working with us, as their new test vendor, to shorten the test. Press release Commissioner Elia also said that, due to complaints she has heard from teachers, students, and parents across the state, there will be other changes, including shorter tests. “So we’re shortening the tests at every grade level,” said Elia. She said children who donot finish the tests in the allotted time might be allowed a few more minutes of time. “Are we really measuring whether or not they have comprehension or whether they can read quickly?” Elia asked.

What is the anticipated time for results in NYS (e.g., 48 hours in Indiana)?

The NYSED is responsible for turning around our final reports.Our expectation is that the timing would be similar to timelines from last year.

What will be the scoring procedures? Will the rubric remain the same?

For the first year, we are using scoring rubrics designed by the previous vendor. We donot anticipate major departures from the scoring rubrics in future years.

It is important to point out that scoring for the operational tests will be performed locally, with machine-scoring (multiple-choice items) completed by scanning organizations engaged by New York RICs, and with hand-scoring (constructed-response items) completed by New York educators using Questar’s online scoring platform, ScorePoint.

Questar will supply the New York scanning centers with the guidance needed to produce appropriate test-form documents for scanning. Thorough training and informational materials (scoring guides, use of ScorePoint, etc.) will be provided to educators, to ensure consistency in hand-scoring. In addition, Questar has a scoring hotline in place, providing access to scoring professionals who can answer test-related questions during local scoring.

For the stand-alone field tests, Questar will perform all scanning of machine-scoreable answer documents and hand-scoring of all constructed-response items. ScorePoint will also be employed for our hand-scoring process.

What is the timeline for Questar assessments to be in place in NYS?

The operational test will contain test items developed by Questar in the 2016-2017 school year.

What is Questar’s response/solution to the increased “opting out” in NYS?

We understand the significance of this issue and will follow all mandates of the State and implement revisions to the test aimed at addressing this issue (Questar’s assessment work is under the direction of the NYSED and as mandated by our contract).

What will the wait time be for questions to be released to educators?

Our understanding is that the timing for questions to be released is a policy matter that will be established by NYSED, and Questar commits to aligning our practices with that policy.

How will Questar’s assessments address test biases (cultural, racial, ethnic, SES)?

Our process includes bias reviews on assessment items before they are used and data analysis after field testing. As part of this process, our proposal committed to an educator review during which 72 New York educators will convene during a 3–5 day meeting to review passages and refine prospective field-test items; committee recommendations will be carefully documented by Questar for consideration during reconciliation; educators will also review the items for bias and sensitivity issues.

Questar has a bedrock commitment to building assessments that are designed to be usable by all students, to the greatest extent possible, including having items that are free of bias and/or construct-irrelevant content. They must also be accompanied by clear and precise testing directions. An assessment should:

  • Measure what it intends to measure and reflect the intended content standard
  • Respect the diversity of the assessment population
  • Have a clear format for text
  • Have clear pictures and graphics, including only essential illustrations
  • Have concise and readable text
  • Be amenable to accommodations
  • Include English language learners (ELLs) and students with disabilities in item tryouts and field/pilot testing
  • Minimize skills required beyond those being measured
  • Be accessible to all students (age, gender, ethnicity, disability, and socio-economic level)
  • Avoid content that might unfairly advantage or disadvantage any student subgroup

What are Questar’s accommodations for ELL’s in the testing format? Does the tool bar facilitate these needs?

The toolbar is represented by easily-understood icons, whichare helpful to ELLs. In addition, the translations of mathematics items made available for students taking the paper-based test can be made available for students online, over the course of the contract, though the full range may require an incremental implementation. Spanish will be available for the spring 2016 administration, and the other four languages chosen by the NYSED will follow in the spring of 2017.

Questar will not be supplying workstation hardware, so some translated mathematics items (particularly those including CR responses) will likely have to be delivered initially via paper-pencil tests, as keyboards that support non-English alphabets may not be widely available in NYSED schools. This aspect of the program will need to be addressed and fully defined in the requirements gathering process.

Questar states that the assessments will be written and built by educators throughout NYS – but who are these educators? Are there English Language educators on the development teams?

The New York educators Questar works with will be selected through the following process (which was also detailed in our proposal). Questar will recruit educators and subject matter experts for the item writer workshop via a nomination process. Schools will begin nominating educators six weeks before the workshop, with a one-week deadline to submit their nominations. Schools may nominate as many educators as they would like, but the final item writing team will include individuals who meet the item writer qualifications. Individual educators may nominate themselves if they provide sufficient, verifiable evidence to show that they meet the qualification requirements.

After receiving all nominations, Questar’s Assessment Design and Psychometrics team will select the item writers based on criteria such as background and experience with the content, experience writing test content, demographics, and needs of the NYS 3–8 ELA/Mathematics Assessments. Priority will be given to experienced item writers, as well as individuals with the appropriate teaching experience and knowledge of the standards.

At a minimum, all item writers will have:

  • Bachelor’s degree
  • At least 24 college credits in their respective content areas
  • Necessary content area knowledge to produce quality items for the student population
  • Experience in writing and editing items
  • Experience with the Common Core State Standards

The nominating process typically produces participants with a broad range of subject-matter expertise, as well as demographic diversity, and it is largely driven by nominations originating in New York schools.