M E M O R A N D U M

TO:Natalie Spaniolo–Acting Director,

Purchasing Operations

FROM:Doug Collier, Bureau of Assessment and Accountability

Office of Assessment Business Operations

DATE:May 4, 2011

SUBJECT: Award Summary for Request for Proposal (RFP)313R0200385 – Item Development for K-12 Assessments. This procurement was done using the Purchasing Alliance Program (PAL) between Michigan Department of Education and the Department of Technology Management and Budget.

Background Information/General:

This Contract will allow for the development, quality assurance, revision, and stakeholder review and recommendation regarding disposition of assessment items and contexts (e.g. passages, charts, and graphics developed as context for answering test questions). This RFP is for all subjects and tests included in the State of Michigan’s K-12 statewide assessments. The assessment programs include the following:

  • Michigan Educational Assessment Program (MEAP)
  • Michigan Merit Examination (MME) custom developed components
  • MI-Access (Michigan’s alternate assessments based on alternate achievement standards for students with significant cognitive disabilities), including:
  • MI-Access Functional Independence (FI, for students with mild cognitive impairment)
  • MI-Access Supported Independence (SI, for students with moderate cognitive impairment)
  • MI-Access Participation (P for students with severe cognitive impairment)
  • MEAP-Access (Michigan alternate assessments based on modified achievement standards for students with disabilities in elementary and intermediate schools)
  • English Language Proficiency Assessment (ELPA)
  • Secondary Credit Assessments (SCA)
  • Interim Benchmark Assessments (IBA) / Formative Assessments activities

The duration of this contract is for a period of five years beginning from the date of award. The Contract may be renewed in writing by mutual agreement of the parties not less than 30 days before its expiration. The Contract may be renewed for up to two additional one year periods.

Joint Evaluation Committee (JEC):

The JEC for this RFP consisted of the following individuals:

Doug Collier, Contracts and Finance Manager,(Voting)
Office of Assessment Business Operations / Glenn Gorton, DTMB(Voting)
Department of Technology Management & Budget
Pat King, BAA (Voting)
Office of Psychometrics Accountability, Research and Evaluation / Andy Middlestead(Voting)
Department of Education
Office of Standards & Assessment
Phil Chase (Non-Voting Advisor)
Department of Education
Office of Assessment Business Operations / David Judd (Non-Voting Advisor)
Department of Education
Office of Psychometrics Accountability, Research and Evaluation
Marilyn Roberts (Non-Voting Advisor)
Department of Education
Office of Assessment Business Operations / Vince Dean (Non-Voting Advisor)
Department of Education
Office of Standards & Assessment

Bidders:

The RFP was posted on the Bid4Michigan web-site on 12/21/2010. The following 7 Bidders submitted proposals by the published due date of 02/16/2011:

Bidder / Address / City, State / Zip / SDV
McCann Associates Holdings LLC / 6805 Rt. 202 / New Hope, PA / 18938 / No
Pearson / 2510 N. Dodge Street / Iowa City, Iowa / 52245 / No
Measured Progress / P.O. Box 1217 / Dover, NH / 03821 / No
Measurement Inc. / 423 Morris St. / Durham, NC / 27701 / No
Questar Assessment, Inc. / 4 HardscrabbleHeights / Brewster, NY / 10509 / No
Data Recognition Corp. / 13490 Bass Lake Rd. / Maple Grove, MN / 55311 / No
Educational Testing Services / 10999 Interstate Highway 10 West, Suite 400 / San Antonio, TX / 78230 / No

Selection Criteria and Evaluation:

The following chart represents the scoring of the particular factors:

Weight
1. / Statement of Work (Article 1) / 35
2. / Bidder Information (4.011) / 10
3. / Prior Experience (4.012) / 25
4. / Staffing (1.031 & 4.013) / 30
TOTAL / 100

Oral Presentation

Bidders who submit proposals may be required to make oral presentations of their proposals to the State. These presentations provide an opportunity for the Bidders to clarify the proposals through mutual understanding. Purchasing Operations will schedule these presentations, if required.

Site Visit

The State may conduct a site visit to tour and inspect the Bidder’s facilities. Purchasing Operations will schedule these visits if required.

Price Evaluation

(a) Only those proposals receiving a score of 80 points or more in the technical proposal evaluation will have their pricing considered for an award.

(b) Evaluation of price proposals includes consideration for a Qualified Disabled Veteran Preference. 1984 PA 431, as amended, establishes a preference of up to 10% for businesses owned by qualified disabled veterans meeting the minimum point threshold for passing.

(c)The State reserves the right to consider economic impact on the State when evaluating proposal pricing. This includes, but is not limited to: job creation, job retention, tax revenue implications, and other economic considerations.

Evaluation Results:

Pearson

The JEC determined that Pearson proposalbased on a score of 81, couldmeet the requirements of the RFP. This determination was accomplished by evaluating their responses to the selection criteria noted in section 3.022 of the RFP.

  1. Statement of Work (Article 1)Score: 27 / 35

The JEC determined Pearson provided a qualified Statement of Work to service this Contract, per the requirements of the RFP, with the following notations /exceptions.

  • JEC noted that Pearson did not state they agree to author item/context development and data review sections of the technical reports as required section 1.042
  • Contractor’s response; “We have included and discussed the appropriate reports throughout our response in Article I. At the Michigan K–12 assessments evolve, we acknowledge that additional reports ordocumentation may be needed. Should Michigan deem additional reports necessary, we willdiscuss them in our proposed weekly teleconference meetings. Such requests will bedocumented through a scope change process” note that the article 1 did not provide a response to the required task.
  • The JEC did not find where documentation of standard setting activities roles were provided or noted. As requested in section 1.042.
  • Pg. 1-44, 1-33 Training for context and Item writer authors, the RFP requires all materials developed to become property of MDE and available electronically; Pearson response - Existing training materials are proprietary – customization for Michigan will require a CCR It appears based on the Pearson response that training materials will be at an additional cost and not included in the cost proposal nor if the BAA would retain ownership in was not address.
  • The Pearson response to requirement 1.025 b.3.o does not meet the RFP requirement. “Evaluate writer performance: Contractor shall evaluate the performance of each context writer after each context writing session. The Contractor shall work with BAA to develop an evaluation tool outside of the IBS. This tool could be online based. The evaluations will be used to determine the reliability of each writer and impact the extent and volume of future writing assignments.”
  • Pg.1-24, 1-42, and 1-43 Targeted assignment of items/contexts; Pearson response includes analysis to create Item/Context development plan external to the IBS inventory system; potentially paying for resources and services not needed. “At the beginning of each development cycle, the content specialist will meet with Michigan and provide an inventory of the pool and recommendations for types and quantity of items to develop for each program, content area, grade level/span, standard, and benchmark/strand.” This is a function of the IBS and not required task of the contractor. The Contractor will work with the BAA staff to review the information provided by the IBS.
  • Pg.1-58 the contractor agreed to manage workflow of translations, but the RFP requested these translations will be submitted to the BAA in PDF format. Contractor stated “We will enter the translations into the respective fields for each item in the IBS.” Additionally the contractor did not state they will provide them in PDF format as required.
  • Pg.1-39 The RFP context workflow requires graphics to be created after the context is approved by committee; however Pearson narrative changes the process flow with context graphics being created prior to committee review.
  • Pg. 1-64 Committee review process includes Pearson Psychometric staff. Pearson proposal states the following; “A Pearson psychometrician will provide an overview of the purpose of the meeting, committee member tasks, and data review objectives. A Pearson content staff person will then convene each small group, and one of our psychometricians will be available to address measurement-related questions or concerns.” The JEC noted that this activity is not part of the RFP requirement.
  • The committee review process defined by Pearson on page 1-67 is external to IBS – does not follow processes documented in RFP.
  • Requirement 1.025 b.3.g.iii does not allow for the use of Apple, 1-92 Pearson to provide Windows xp or Apple os-x, this does not comply with RFP requirement.
  • Pg.1-15 Pearson states multiple reviews in place during the Pearson review process; the JEC noted this infers existing process or practices are Pearson’s rather than the BAA IBS process flow.
  • The JEC noted that Pearson provided good information about different assessment types.
  • Pearson proposal discussed working on CCSS items for IBA already, including CBA items.
  • Pearson’s proposal didn’t address the 4 in person monthly management meetings as required in section 1.025 B.1.b
  • The Pearson proposal stated there will be 4 executive mgt meetings instead of the required 2 as noted in the RFP.
  • The JEC noted Pearson's Cvent system appears to be a good registration management system.
  • Pearson proposal includes draft Blue Print development, which is not in the scope of this RFP.
  • Pg. 1.35 Contractor agreed to enter attributes for items that may appear on Form 1; RFP requires entering attributes for items on the accommodated form is not always form 1.

2. Company Information (4.011)Score: 10 / 10

The JEC determined Pearson informationmet this requirement of the RFP with no exceptions.

3. Prior Experience (4.012)Score: 23 / 25

The JEC determined Pearson provided adequate Prior Experiencesto service this Contract, per the requirements of the RFP, and noted the following:

Grades 3-8 General Assessment / HS General Assessment / Modified/Alternate Assessment / ELL Assessment / End of Course or Interim Benchmark Assessment
Michigan 2005-2010 / Michigan 2005-2010 / Michigan 2005-2008 / Michigan 2006-2007
Louisiana 2005-2011, Grades 4 and 8 / Louisiana 2005-2011, modifies end of course assessment items. Some concerns please note experience with DRC experience.
Oklahoma 2007-2011
  • The JEC noted extensive experience with item/context development and committee reviews for general assessment programs in Michigan.
  • Developed SCA assessments, but less than three years experience.
  • Experience developing modified assessments, alternate assessments, and ELL assessments.
  • Included MME in the development years 1998-2002; it was noted by the JEC, that MME did not exist until Pilot in Spring 2006.
  • The JEC noted that Pearson included MME ELA Reading and Writing item development in prior experience; ELA has not been a part of the MME Michigan developed components with the exception of writing prompt on the Social Studies test.

4. Contractor Roles (1.031) / Staffing (4.013)Score: 21 / 30

The JEC determinedPearson's proposal provided adequate Contractor Roles / Staffing to service this Contract, per the requirements of the RFP, with the following noted concerns.

  • Pearson's proposal was unclear as to the actual amount of staffing provided in addition to the Content Leads in each area; includes N count and % for each content area – does the % represent total additional full-time employee (FTE) or for each additional N count See staffing chart pg 1-104 Example 4 advisors at 85% does this mean 4 staff x 85% each totaling 3.4 FTE or 4 staff totaling .85 FTE?
  • Resumes for additional staffing beyond the Content Lead in each content area were not provided therefore the JEC was unable to determine their qualifications.
  • Based on Joleen J resume the JEC had concerns as she is allocated to several other programs.
  • Pg 1-36 and 1-46 content areas omit Listening and Speaking.
  • JEC had concerns with Pearson’s statement on Pg. 1-29, Staff are listed “likely to fulfill roles” also, in Section two they requested that the Program Mgr. be the only person referred to as Key Personnel, see section 2.063 pg 2-14 and 2-50.

Based on the Joint evaluation committee’s review, the Pearson proposal did not offer the best value to the State, Best Value is determined by the Bidder meeting the minimum point threshold and offering the best combination of the factors stated in Section 3.022, and price. Pearson's five-year contract amount as originally bid was $32,502,921.00 and, requested best price of $32,156,549.00 was offered.

Total Score: 81 / 100

Data Recognition Corp. (DRC)

The JEC determined thatDRC proposal based on a score of91, couldmeet the requirements of the RFP. This determination was accomplished by evaluating their responses to the selection criteria noted in section 3.022 of the RFP.

  1. Statement of Work (Article 1 Score: 33 / 35

The JEC determined DRC provided a qualified Statement of Work to service this Contract, per the requirements of the RFP, with the following notations /exceptions.

  • Pg. 1-27, The JEC noted DRC's proposal devoted IBS IT expert that will train staff from DRC.
  • Pg. 1-11,12 DRC's proposal referenced varied item types that Michigan would like to develop using Computer adaptive testing etc.
  • Pg.1.60 – DRC's proposal referenced a Lansing location, which would be leased for office space for committee meeting.
  • Pg.1-69 DRC's proposal referenced they will invite up to 18 IWT members; this exceeds RFP requirements, but indicates facilities and equipment needed to accommodate additional item/context writers or committee members should the need arise would be provided.
  • Pg. 1-84, 1-87Contractor provided sample report contents, of what they could provide.
  • Pg.1-34 DRC's proposal referenced Artwork or graphics for Permissioned passages which would be entered into IBS – The JEC noted that this is inconsistent with process flowas noted in the ITB.
  • Pg.1-43DRC's proposal included Item Writer Training Manual – and noted that they are willing to customize for Michigan.
  • Pg.1-14 DRC discussed that they follow Federal guidelines including but not limited to Peer review participation.

2. Company Information (4.011)Score: 10 / 10

The JEC determined DRCinformation met this requirement of the RFP with no exceptions.

3. Prior Experience (4.012)Score: 23 / 25

The JEC determinedDRCprovided adequate Prior Experiences to service this Contract, per the requirements of the RFP,with the following notations /exceptions;

Grades 3-8 General Assessment / HS General Assessment / Modified/Alternate Assessment / ELL Assessment / End of Course or Interim Benchmark Assessment / Other
Alabama 2004-2015
Alaska 2004-2016, science beginning in 2005 (not social studies) / Alaska 2004-2016, science beginning in 2005 (did not note social studies)
Idaho 2006 - 2012 (not social studies) / Idaho 2006 - 2012 (not social studies)
Louisiana 1998-2011 BSC/CAC reviews; science and social studies item development 2003-2012 / Louisiana 1998-2011 BSC/CAC reviews / Louisiana 2001-2011 BSC/CAC reviews / Louisiana 2005-2011 BSC/CAC reviews
Pennsylvania 1992-2014 (did not note social studies) / Pennsylvania 1992-2014 (did not note social studies) / Pennsylvania 1992-2014 modified assessment items(did not note social studies) / Pennsylvania 2009-2015 / Pennsylvania 2009 - 2015, CAT grades 6-12 formative assessment
West Virgina 2009-2011, science alignment including item review meetings
The JEC did not see experience in Alternate assessment IW. / The JEC did not see experience in ELL IW.
  • DRC's proposal referenced extensive experience with item/context development and committee reviews for general assessment programs and HS end-of-course assessments
  • DRC's proposal referenced experience developing modified assessments; BSC/CAC reviews for alternate assessments and ELL assessments.
  • DRC's proposal referenced long term continuity and increased services/programs included in state assessment contracts. The JEC noted this as a positive indicator based on some of the references provided.
  • Pg. 4-10 and 4-29 The JEC noted experience with item development for Computer adaptive testing (CAT); computer based formative assessments (CBT); and Pg. 4-10 experience with interactive items (This is the direction the BAA is moving to in the future, however this was not required as part of this RFP but was viewed as a positive).

4. Contractor Roles (1.031) / Staffing (4.013)Score: 25 / 30

The JEC determinedDRCprovided adequate Contractor Roles / Staffing to service this Contract, per the requirements of the RFP, with the following notations/exception:

  • DRC's proposal provided a well laid out staffing chart, number of resources and total FTE in each content area.
  • DRC's provided resumes for all staff.
  • Pg. 1-22 and 1-77 Made reference to expert knowledge of Michigan content expectations; the JEC questioned what the basis is for this qualification.
  • DRC's proposal provided 3 meeting planners equivalent to 1 FTE – the JEC noted this provides for cross training and availability.
  • DRC's proposal provided 2 staff with background in computer adaptive testing and item types while this was not a requirement of the RFP it was noted as a value add.
  • DRC's proposal omitted Listening as content area in staffing chart; however it was noted in the resume for Anne Kirpes.
  • Content Lead/advisors in Writing/Speaking- some staff new to DRC and limited Elementary/MS item development experience was noted by the JEC.
  • DRC's proposal noted ELL Consultant and SWD Consultant available as needed; the JEC had concerns and questioned if this is this sufficient for ELPA and MI-Access item development? Additionally they are listed consultants.

Based on the Joint evaluation committee’s review, DRC’s proposal offered the best value to the state, Best value is determined by the Bidder meeting the minimum point threshold and offering the best combination of the factors stated in Section 3.022, and price. DRC's five-year contract amount as originally bid was $29,175,860.00 and, requested best price of $27,067,755.00 was offered.

Total Score: 91 / 100

Questar Assessment Inc.

The JEC determined thatQuestar’s proposal based on a score of81, couldmeet the requirements of the RFP. This determination was accomplished by evaluating their responses to the selection criteria noted in section 3.022 of the RFP.

  1. Statement of Work (Article 1) Score: 30 / 35

The JEC determined Questar’s provided a qualified Statement of Work to service this Contract, per the requirements of the RFP, with the following notations /exceptions.

  • The Questar proposal provided experience and knowledge with option descriptors, multiple rubrics for a single CR, item-specific custom rubrics, and gridded response questions per scope requirement Pg. 1.012 and Pg. 15-19
  • 1.021 – Pg. 31.7(2) addressed rigorous application/nomination process subject to BAA approval, although the JEC noted that limited information was provided.
  • 1.023 – p31.6 Questar proposal provided examples of meeting spaces that would be used, including suggestion of local high schools during summer months (Okemos). The JEC noted this as a positive use of school resources.
  • 1.023 – Pg. 37-39 Questar proposal noted Item Writers will be provided with an item checklist to ensure their items meet requirements for a good item prior to submission of the item, theJEC noted this as a positive.
  • 1.023 – Pg.39 references copyrighted images to accompany submitted items. This does not meet the RFP requirements. The IBS flow chart,notes to use internalBAA Composition team for graphic media.
  • 1.023 – Pg. 31 states “item writers who have participated in previous years will be trained remotely using the IBS”. This is not consistent with requirement 1.023.C that states training materials will not contain links to nor be interactive with the IBS.
  • Pg. 32, No. 10 – The JEC had concerns that Questar’s proposal to reduce the reading level of science or social studies items. This is not part of the BAA objectives for science and social studies items.
  • Pg. 42, #. 4 – RFP states that the contractor will not be charged with providing the translations of items, only review. RFP states in Section E, Number 4 – The RFP requires “The contractor will translate the reader script in all parts of the item and instructions that are read to the student in the target language.”
  • Pg. 42 #3 Questar’s proposal noted they “will work with an outside vendor, The Geo Group, a national leader in supplying translation and audio/video accommodation services with whom we have previously worked. The Geo Group will translate items for us with any notes of consideration. Following this, our Content Specialists will take their input and develop the reader scripts necessary to administer selected items in an audio version. The reader scripts (along with any necessary indication of how the item should be read) will be entered into the IBS by Questar staff.” This does not meet the requirements of the RFP, additionally Questar proposal did not note The Geo Group as a subcontractor. Nor does it align with the process flow as noted in the RFP on pg. 7 and 8.
  • All item writers will be required to sign a confidentiality agreement that stipulates that the writer shall not provide contexts/items developed specifically for the Michigan assessments to any other individual or entity for any other purpose JEC noted this as a positive.
  • The JEC discovered in the evaluation of Questar's pricing, (after they were determined to have passed the minimum technical score, the following line item 37a Item editing was added). The JEC requested the following from all bidders passing technical evaluation. “If applicable, for any line items that were added and not part of the pricing appendix as provided under this RFP, please provide a complete response as to when and how these costs would apply. If you have provided additional line items where we have noted that it has been “normalized out” no response is required.” Questar responded with the following; “Regarding line 37a, our understanding of the development tasks is that we will have to take the items developed by the Michigan educators and do some internal review and prep of the items before they move on to the next phase. We decided to separate these costs out as variable costs and will charge only for those items we need to review and “clean up” before they move on to the next phase. We decided on this approach to contain costs; the alternative would have been to incorporate the costs for this task into a fixed cost line, such as the Contract Management or Project Management lines.” JEC noted that the contractor will be required to review, cleanup and possibly edit items based on a flowchart provided on the IBS system in the RFP,which would occur at a minimum at three times for each item. It should be noted that cost noted below for Questar only allows for one edit, if three edits occurred as noted in the IBS flowchart the contract value would increase by an additional **$2,462,500.

2. Company Information (4.011)Score: 10 / 10