memo-dsib-adad-dec12item04
Page 1 of 2
California Department of EducationExecutive Office
SBE-002(REV.01/2011) / memo-dsib-adad-dec12item04
memorandum
Date: / December 4, 2012
TO: / MEMBERS, State Board of Education
FROM: / TOM TORLAKSON, State Superintendent of Public Instruction
SUBJECT: / Standardized Testing and Reporting Program: Independent Alignment Study of the California Modified Assessment.
Summary of Key Issues
An independent alignment study of the California Modified Assessment (CMA) was conducted in April 2012 by Data Recognition Corporation (DRC). The final report was received by the California Department of Education (CDE) in September 2012.
Overall, the CMA was found to meet the requirements for alignment in all subjects and grades. The categorical concurrence, depth of knowledge, and range of knowledge ratings for all three subject areas were within the acceptable range with the exception of a few standards that were found to be “weak” in one or more areas of alignment.
On December 4, 2012 the CDE sent the study for the U.S. Department of Education’s(ED) Peer Review and will address any necessary revisions upon request by the ED.
The DRC report’s recommendations can be found on pages 18, 25, and 33 of Attachment 1 to this memorandum. The complete study will be posted on the CDE STAR Technical Reports and Studies Web page at [Note: Invalid link removed.].
Background
The Elementary and Secondary Education Act (ESEA) reformed federal educational programs to support state efforts to establish challenging standards, develop aligned assessments, and build accountability systems for local educational agencies (LEAs) that are based on educational results. The California state legislature established the STAR Program in 1997, per California Education Code (EC) Section 60640. EC Section 60642.5 requires the State Superintendent of Public Instruction (SSPI), with the approval of the SBE, to develop tests that are aligned with the academically rigorous content standards adopted by the SBE to measure how well students in grades two through eleven in California public schools are learning the knowledge and skills identified in California’s content standards. The STAR Program includes the following tests: the CSTs, the CMA, the California Alternate Performance Assessment (CAPA), and the Standards-based Tests in Spanish. The CSTs, the CMA, and the CAPA results are used to monitor the adequate yearly progress (AYP) of LEAs toward meeting the accountability targets of the ESEA.
The CMA is an alternate assessment, based on modified achievement standards, for eligible students with disabilities who have an individualized education program (IEP) and meet the CMA eligibility criteria adopted by the SBE. The ESEA provides flexibility to states to develop alternate assessments based on modified achievement standards; to which the CDE has been developing the CMA to meet this need. Additional information on the CMA may be found on the CDE CMA Web page at [Note: Invalid link removed.].
In summary, California was required to conduct independent alignment and validation studies of the CMA. Previous alignment studies have been conducted of the STAR Program CSTs and the CAPAto meet peer review requirements and may be found on the CDE STAR Technical Reports and Studies Web page at [Note: Invalid link removed.].
Previous SBE Action
The SBE has previously taken no action related to theindependent alignment study of the CMA.
Fiscal Analysis
The 2011 Budget Act appropriated $600,000 ($200,000 in federal Title I funds and $400,000 in federal Title VI funds) for the work identified in the Request for Proposals, California Modified Assessment Studies, available on the CDE Web page at
All costs associated with the DRC CMA alignment study activities for the contract period November 1, 2011, through August 31, 2012, were included in the onetime federal Title I and VI funds, authorized in 2011 Budget Act.
Attachment(s)
Attachment 1: California Modified AssessmentAlignment Study Final Report, English Language Arts, Science, and Mathematics, April 10–13, 2012 (40 Pages)
10/8/2018 11:29 AM
memo-dsib-adad-dec12item04
Attachment 1
Page 1 of 40
CaliforniaModified Assessment
Alignment Study FinalReport
English LanguageArts,Science, andMathematics
April 10–13,2012
Prepared for: California Department of Education
Sacramento, California
By:Data Recognition Corporation
Contract Number: CN110174
10/8/2018 11:29 AM
memo-dsib-adad-dec12item04
Attachment 1
Page 1 of 40
Thefindingsinthisstudyarethoseoftheindependentreviewingteamanddonotrepresentthe opinion ofthe State ofCalifornia.
memo-dsib-adad-dec12item04
Attachment 1
Page 1 of 40
TableofContents
ExecutiveSummary...... 5
Introduction...... 6
StudyDesign...... 7
StudyMethodology...... 9
AlignmentCriteria...... 9
SourceofChallenge...... 11
AlignmentStudyProcess...... 12
AlignmentStudyParticipants...... 14
Data AnalysisResults—EnglishLanguageArts...... 16
Summaryof Results...... 16
Depth-of-KnowledgeConsensus...... 17
Conclusions and Recommendations...... 18
ESEA Requirements...... 20
ReliabilityamongReviewers...... 23
Data AnalysisResults—Science...... 24
Summaryof Results...... 24
Depth-of-KnowledgeConsensus...... 24
Conclusions and Recommendations...... 25
ESEA Requirements...... 27
ReliabilityamongReviewers...... 30
Data AnalysisResults—Mathematics...... 31
Summaryof Results...... 31
Depth-of-KnowledgeConsensus...... 33
Conclusions and Recommendations...... 33
ESEA Requirements...... 36
ReliabilityamongReviewers...... 39
References...... 40
AppendixA: Depth-of-KnowledgeLevels...... 41
EnglishLanguageArts Depth-of-KnowledgeLevels...... 42
ScienceDepth-of-KnowledgeLevels...... 45
Mathematics Depth-of-KnowledgeLevels...... 47
AppendixB: Depth-of-KnowledgeConsensusValues...... 49
EnglishLanguageArts Depth-of-KnowledgeConsensus...... 50
ScienceDepth-of-KnowledgeConsensus...... 80
Mathematics Depth-of-KnowledgeConsensus...... 92
memo-dsib-adad-dec12item04
Attachment 1
Page 1 of 40
AppendixC: SummaryTables...... 114
EnglishLanguageArtsSummaryTables...... 115
ScienceSummaryTables...... 133
Mathematics SummaryTables...... 139
AppendixD: Depth-of-KnowledgeLevels by Itemand Reviewers...... 153
EnglishLanguageArts by Itemsand Reviewers...... 154
Scienceby Itemsand Reviewers...... 172
Mathematics by Items and Reviewers...... 178
AppendixE: Standard and Depth-of-KnowledgeAlignments Assigned byReviewers...... 192
EnglishLanguageArtsStandard and Depth-of-KnowledgeAlignmentsAssigned
byReviewers...... 193
ScienceStandardand Depth-of-KnowledgeAlignments Assigned byReviewers ...... 247
Mathematics Standard and Depth-of-KnowledgeAlignments Assigned
byReviewers...... 265
AppendixF: Results ofIntraclass Correlation...... 307
EnglishLanguageArts...... 309
Science...... 311
Mathematics...... 312
AppendixG: Alignment StudyReviewers andBiographies of theNationalExperts...... 314
Listof Reviewers...... 315
Project Support...... 316
EnglishLanguageArts...... 321
Science...... 327
Mathematics...... 331
AppendixH: California Participant DemographicInformation...... 338
AppendixI: Alignment StudyTrainingPowerPoint...... 340
Appendix J: Alignment StudyAgenda...... 355
memo-dsib-adad-dec12item04
Attachment 1
Page 1 of 40
ExecutiveSummary
The CaliforniaModifiedAssessment(CMA)alignmentstudiesingrades3–11Englishlanguage arts,mathematicsgrades3–7,AlgebraI,andGeometry andgrades5,8,andhighschoolscience were held onApril10−13, 2012, in Sacramento,California. Thepurposeof eachalignmentstudy wastodeterminethedegreeofalignmentbetweenthecontentstandardsfor eachgradeandthe testitemsfoundonthecorrespondinggrade-levelCMA.Thealignmentstudy involvedeight grade-spangroupsofeight independentthird-partyreviewerswhoseprimaryrolewasfirstto judge the depth-of-knowledge levelofeachstandardandthen tojudge the depth-of-knowledge levelofeachtestitem,includingidentifyingtheprimaryandpossibly asecondary standardto which each itemwasaligned.Overall, the finalresultsindicatedthat thealignment relationships forthestudiesarestrongandclearly demonstratethattheCMAtestsarewellalignedtothe respectiveCalifornia standards.
Eightreviewersparticipatedinthealignmentstudiesoneachcommittee.Fourofthereviewers for eachstudy wereCaliforniaeducators whohadextensiveteachingexperience,including teaching studentswithdisabilitiesand/oradministering theCMAandexpertiseintheircontent areas.Theotherfourreviewersforeachalignmentstudywerenationalcontent experts. Each nationalcontentexpertalsohadexpertise intheir content area andexperience instandards development,curriculumand instruction development,testdevelopment,and alignmentstudies. Inaddition,one of the nationalcontentexpertsalsoservedasa groupleader.The listof the reviewersandabriefsummaryofeachnationalexpert’sprofessionalqualificationsisprovided in AppendixG.
Inadditiontothealignmentstudy reviewers,anationalalignmentstudy expert,Dr.Carsten Wilmesofthe WisconsinCenter forEducationResearch(WCER)Consortium,alsoparticipated inthe study.Dr. Wilmes isawell-knownalignmentexpert who hasbroadexperience in conductingalignmentstudiesusingtheWebbmodel.Overtheyearshehasworkedclosely with Dr.NormanWebbandDr.GaryCook,oftheWisconsinCenterforEducationResearch.The nationalalignmentstudy expert’srolewastooverseetheentirealignmentprocess,ensuring that procedureswerefollowedcorrectly.Thenationalalignmentstudyexpertalsoprovidedreviewers with alignment training.
memo-dsib-adad-dec12item04
Attachment 1
Page 1 of 40
Introduction
TheCaliforniaModifiedAssessment(CMA)isanassessmentofstudents’mastery ofCalifornia contentstandardsforEnglishlanguagearts,mathematics,andsciencedevelopedforstudents withanindividualizededucationprogram(IEP)whomeettheCMAeligibility criteriaapproved by theCaliforniaStateBoardofEducation.The testsaregiveningrades3–11Englishlanguage arts; grades 3–7 mathematics, AlgebraI, andGeometry;andgrades 5, 8, and high school science. They consistofmultiple-choicetestsinEnglishlanguagearts,mathematics,andscience.The CMA measures student achievement based on California’s content standards.
TheCMAalignmentstudiesarebasedontheworkofNorman Webb,Wisconsin Center for EducationResearch,University ofWisconsin–Madison,whostatesthatthealignmentofthe standardsforstudentlearningwithassessmentsformeasuring students’fulfillmentofthese expectationsisanessentialcomponentfor aneffectivestandards-basededucation system.This study modelsWebb’sprocedures,includingtheuseofthealignmentcriteriaofcategorical concurrence,depth-of-knowledge consistency,range-of-knowledge correspondence,andbalance of representation,as wellas Webb’s definition of alignment. Thedefinition is as follows:
Alignmentisdefinedasthedegreetowhichexpectationsandassessmentsare in agreementandserve inconjunctionwithoneanothertoguidethesystemtoward studentslearningwhatthey areexpectedtoknowanddo.Assuch,alignmentisa quality of the relationship between expectations and assessments and not a specificattributeofeitherofthese twosystemcomponents.Alignmentdescribes thematchbetweenexpectationsandassessmentthatcanbelegitimately improved by changing eitherstudentexpectationsorassessments.Seenasarelationship betweentwoormoresystemcomponents,alignmentcanbedeterminedby using the multiple criteria described in detail in a National Institute of Science Education(NISE)researchmonograph,CriteriaforAlignmentofExpectations and Assessments(Webb,2002).
Dr.CarstenWilmesprovidedtraining for allreviewerstounderstandWebb’salignmentmodel, depth-of-knowledge categories,andalignmentcriteria.He firsttrainedthereviewerstoidentify the depth-of-knowledge (DOK) level for the content standards and the test questions. The training includedreviewing thedefinitionsandkey wordsofthedepth-of-knowledgelevels,as definedby Webb(2006),andreviewingexamplesoftestquestionsalignedtodepth-of- knowledgelevels.Formoreinformationregarding theprocess,seethe sectiontitledAlignment StudyProcess.Dr. Wilmes’sprofessional qualifications areprovided in AppendixG.
memo-dsib-adad-dec12item04
Attachment 1
Page 1 of 40
StudyDesign
TheCaliforniaModifiedAssessmentalignmentstudieswere designedtoaddresstheElementary andSecondary EducationAct(ESEA)andtheUnitedStatesDepartmentofEducationStandards andAssessmentsPeerReviewGuidanceforaccountability.UsingDr.NormanWebb’scriteria of categoricalconcurrence,depth-of-knowledgeconsistency,range-of-knowledge correspondence,andbalance ofrepresentationalongwithqualitativeandquantitative results,the studywas based on the followingrequirements.
1. ThealignmentoftheCaliforniaModifiedAssessments(CMA)withthecontentstandards and how the cognitive load differsfrom theCaliforniaStandards Test (CST).
2. Thestate’sassessmentsysteminvolvesmultiplemeasures(measuresthatassesshigher- order thinkingskills andunderstandingof challengingcontent).
3. TheCMAmeasurestheknowledgeandskillsdescribedinitsacademiccontentstandards andnotknowledge,skills,orothercharacteristicsthatare notspecifiedintheacademic content standards orgrade-level expectations.
4. TheCMAitemsaretappingtheintendedcognitiveprocessesandtheitemsandtasksare at the appropriategradelevel.
5. The CMAandreportingstructuresareconsistentwiththesubdomainstructuresof its academic content standards.
Requirement 1:
ThealignmentoftheCaliforniaModifiedAssessment(CMA)withthecontentstandards and how the cognitive load differsfrom the CaliforniaStandards Test (CST).
Reviewersused the CMA contentstandardswhichwere identicalinstructureandwording tothe CSTstandards. However,someofthe CSTstandards werenotincludedinthe CMAblueprints. Categoricalconcurrence,orthenumberofitemsperreporting cluster,wasdeterminedwhenthe number of times reviewers assigned an item to a standard within a reporting cluster was averaged. Webb’s criteriaof sixitems per reportingcluster indicatedacceptable alignment.
The depth of knowledge for each standard was determined by individual reviewers and, following discussion,consensusratingswere reachedforalltheEnglishlanguagearts,science, andmathematicsstandards.TheseCMAconsensusvalueswere comparedtotheCSTconsensus values, and itwas determined whetherthe CMA values werebelow,at, or abovethe CST values.
Requirement 2:
Thestate’sassessmentsysteminvolvesmultiplemeasures(measuresthatassesshigh- order thinkingskills andunderstandingof challengingcontent).
Webb’sEnglishlanguagearts,science,andmathematicsdepth-of-knowledge definitionsand California-specific CMAsample items were provided anddiscussed inthe large-grouptraining ledbyDr.CarstenWilmes.Afterthelargegrouptraining,morecontent-specifictrainingofthe
memo-dsib-adad-dec12item04
Attachment 1
Page 1 of 40
definitionsandsampleswerepresented by eachgroupleader.(SeeAppendixA.)Thecontent- specific trainingincludedrichdiscussionsof thedepth-of-knowledgelevelsandthe nuancesof thecontentinrelationtothedepth-of-knowledge levels.After training,thereviewersreached consensusonthedepth-of-knowledge ofthestandardsfor Englishlanguage arts,science,and mathematics.Thereviewersthenindependently alignedtheitemsoftheassessmenttotheCMA standards andassigned aDOKratingto each item.
Requirement 3:
TheCMAmeasurestheknowledgeandskillsdescribedinitsacademiccontentstandards andnotknowledge,skills,orothercharacteristicsthatare notspecifiedintheacademic content standards orgrade-level expectations.
Reviewersassignedaprimaryand/orsecondary standardforallitemswiththeexceptionofthe mathematicreasoningstandards.Only contentstandardsfromthespecificgrades’blueprintwere provided to the reviewers.
Requirement 4:
TheCMAitemsaretappingtheintendedcognitiveprocessesandtheitemsandtasksare at the appropriategradelevel.
AsinRequirement2,reviewersfirstcametoconsensusastothedepth-of-knowledge levelof eachofthestandardsandthenindependentlyassignedonlyonedepth-of-knowledgelevelto eachoftheitems.Intraclasscorrelationwascalculatedtohelpdeterminethereliabilityofthe results and consistencyamongreviewers.
Also, Webb’s criterion of depth-of-knowledge consistency indicates that reviewers were assigning thedepthofknowledgetotheitemsthatwere theintendedcognitivedemandofthe standards.Reviewerswere abletoalignitemstothecontentstandardsfortheapplicablegrade withoutdifficulty.
Requirement 5:
The CMAandreportingstructuresareconsistentwiththesubdomainstructuresof its academic content standards.
Whenthereviewersindependently determinedwhichstandardalignedtoanitem,thejudgment wasrecordedasahit.Thetotalnumberofhitswasaveragedtodeterminehowmany itemswere assessedineachreporting cluster.Theaveragenumberofreviewers’hitswascomparedtothe state-approved blueprintfor each assessmentand its reporting clusters.
memo-dsib-adad-dec12item04
Attachment 1
Page 1 of 40
StudyMethodology
Alignment Criteria
TheCaliforniaModifiedAssessmentalignmentstudieswere basedontheworkofDr.Norman Webb,WisconsinCenterforEducationResearch,UniversityofWisconsin–Madison.Inhis work,Dr. Webbstatesthatthealignmentof thestandardsfor studentlearningwithtestsfor measuring students’fulfillmentoftheseexpectationsisanessentialcomponentforaneffective standards-basededucation system.TheCMAalignmentstudieswere designedtofollowWebb’s procedures,including theuseofdepth-of-knowledgelevels,andWebb’sdefinitionofalignment (Webb, 2002). Webb’s alignment model is based upon four criteriaas follows:
Categorical Concurrence—According to Webb (2002), an important aspect of alignment betweeneachreporting clusterandthe testiswhether bothaddressthe same contentcategories. The categoricalconcurrence criterionprovidesageneralindicationofalignmentifthereporting clusterandthetestincorporatethesamecontent.For thesealignmentstudies,thiscriterionwas judgedbyfirstallowingreviewerstomakeadeterminationastowhetherthetestasawhole includedquestionsmeasuringcontent from eachofthereportingclusters.The reviewersused their professionalopinions,aswellasWebb’ssuggestedcriteria,todetermine thatatleastsix questionsmeasuringcontentfromeachreporting clusterisagoodindicatorof categorical concurrencebetween thereportingclusterand thetest (Webb, 2002).
Using Webb’smethodology,thenumberofquestionsusedtodetermine categorical concurrence—sixfor thisstudy—is basedonestimating thenumberofquestionsthatcould produceareasonably reliablesubscaleforestimatingstudents’masteryofcontentonthat subscale.Ofcourse,manyfactorshavetobeconsideredindeterminingareasonablenumber, including thereliability ofthesubscale,themeanscore,andthecutoffscorefordetermining mastery.Using aproceduredevelopedbySubkoviak(1988)andassumingthatthecutoffscoreis themeanandthatthereliabilityofoneitemis0.1,itwasestimatedthatsixquestionswould producean agreement coefficientof atleast0.63.Thisindicatesthatabout63% of thegroup wouldbe consistentlyclassifiedaseither mastersor non-masters iftwoequivalenttest administrations were employed.The agreementcoefficientwould increaseif thecutoffscorewas increasedtoonestandarddeviationfromthemeanto0.77and,withacutoffscoreof1.5 standard deviations fromthe mean, to 0.88.
FortheCMAalignmentstudies,thecriterionwasjudgedby firstallowingreviewerstoalignthe itemstothestandardsthatmeasurethereportingclusters. Sixquestionswereassumedasa minimumforatestmeasuring contentknowledgerelatedtoareporting clusterandasabasisfor making some decisionsaboutstudents’knowledgeofthatreporting cluster.Ifthemeanforsix questionsisthreeandonestandarddeviationisonequestion,thenacutoffscoresetatfour wouldproduceanagreementcoefficientof0.77.Any fewerquestionswithameanofone-halfof thequestionswouldrequireacutoffthatwouldallowastudenttomissonly onequestion.This wouldbeaverystringentrequirementconsideringareasonablestandarderrorofmeasurement on the subscale. (SeeAppendixC.)
Depth-of-Knowledge Consistency—For the purpose of thisstudy, Webb’s definitionofdepth-of- knowledgeconsistencywasused.AccordingtoWebb(2002),depth-of-knowledgeconsistency
memo-dsib-adad-dec12item04
Attachment 1
Page 1 of 40
betweencontent standards and test itemsindicatesacceptable alignmentif whatiselicitedfrom studentsonthetestisatleastasdemandingcognitivelyaswhatstudentsareexpectedtoknow anddoasstatedinthecontentstandards.Therefore,forconsistencytoexistbetweenthetest itemsandthestandards,eachitemshouldbecodedthesamedepth-of-knowledge levelasthe standardoronelevelabovethedepth-of-knowledgelevelofthestandard.Accordingtothe Webbmodel,asameasure ofconsistency,atleast50%oftheitemscorresponding toareporting cluster shouldbe ator above the depth-of-knowledge levelof thestandard.For depth-of- knowledgeconsistency,thiscriterionwasjudgedbyfirstallowingreviewerstoassignadepth- of-knowledgelevel to each item. (SeeAppendixC.)
Thedepth-of-knowledgedefinitionsusedforthis alignmentstudy areasfollows:Level1(Recall and Reproduction), Level 2 (Skills and Concepts), and Level 3 (Strategic and Extended Thinking). Additional information concerningthelevels can befoundin AppendixA.
Range-of-Knowledge Correspondence—For the reportingclustersandthe testquestionstobe aligned,the breadthofknowledge requiredonbothmustbe comparable. The range-of- knowledgecriterionisusedtojudgewhetherthespanofknowledgeexpectedofstudents by a reporting clusteristhesameas,orcorrespondsto,thespanofknowledgethatstudentsneedin ordertocorrectly answerthetestquestionsassociatedwiththatreporting cluster.Foran acceptablerangeof knowledge,atleast50% ofthestandardsforareportingclustermust haveat leastonerelatedtestquestion.The range-of-knowledge correspondence criterionwasjudgedby firstallowingreviewers toalignthe itemstothestandards thatmeasurethe reportingclusterand then evaluatingthe range-of-knowledge correspondencebased on that data.(SeeAppendixC.)
BalanceofRepresentation—Thebalanceofrepresentationismet iftheemphasisofcontentand performance supplied by the questions (primary, secondary, or both) corresponds to the standards for thetest asa whole.Reviewersdeterminedwhetherthetestquestionswere distributed amongthe standards that wereassessed.(SeeAppendixC.)
Thebalance-of-representationcriterionisusedtoindicatethedegreetowhichonestandardis givenmoreemphasisonthetestthananother.Anindexisusedtojudgethedistributionofthe testquestions.Thisindexonly considersthestandardforareportingclusterthathasatleastone relatedassessmentitem.Theindexinthisstudy wascomputed by consideringthedifference betweentheproportion ofstandardsandtheproportionofhits(questionscorrespondingto eligible content) assignedtothe standards. An indexvalue of onesignifiesperfectbalance andis obtainedifthehitsareequally distributedamongthestandards.Indexvaluesthatapproachzero signify thatalargeproportionofthehitsareononly oneortwoofallofthecontentstandards. Dependingonthenumberofcontentstandardsandthenumberofhits,aunimodaldistribution hasanindexvalueoflessthan0.5.Abimodaldistributionhasanindexvalueofaround0.55or
0.6.Indexvaluesof0.7orhigherindicatethatquestionsare distributedamongallofthe standards, atleasttosomedegree.Indexvaluesbetween0.6and0.7indicate the balance-of- representationcriterionhasonly been“moderately”met.Thebalance-of-representationcriterion wasjudged by firstallowing reviewerstoaligntheitemstothestandardsthatmeasurethe reportingclusters.
The Webbmodelprovidesareliablesetofproceduresandcriteria forconductingalignment analysisstudies.The modelcombinesqualitative expertreviewers’ judgmentsandquantified codingandanalysisofstandardsandtestitems.Thisfinalalignmentstudyreportincludesaset
memo-dsib-adad-dec12item04
Attachment 1
Page 1 of 40
ofstatisticsforeachreporting clusterandgradeonthedegreeofalignmentbetweenthecontent embeddedinthestandardsforagivengradeandthe contentintheitemsonthecorresponding CMA.
TheWebbmodelhasbeenusedextensively inmanyalignmentstudiesthroughoutthecountry andhasbeenrecommendedforuseby theChief CouncilofStateSchoolOfficers(CCSSO).The alignmentcriteriaoftheWebbmodeladhere totheguidelinesspecifiedintheUnitedStates Departmentof Education’s Standards andTestsPeer Review documentsandare incompliance with the requirements specifiedbytheNo ChildLeft Behind(NCLB) legislation.
A summaryof Webb’s alignment criteriacan befound in Table1.
Table 1: Alignment Levels for theFourCriteria
AlignmentLevel / Depth-of-
Knowledge
Consistency / Categorical
Concurrence / Range-of-
Knowledge
Correspondence / Balanceof
Representation
Yes / 50% / mean is 6 or more / 50% / 0.70
Yes* / 40%–49% / mean is 5 to 5.9 / 40%–49% / 0.60–0.69
Weaker / less than 40% / mean is less than 5 / less than 40% / less than 0.60
*Indicates acceptable alignment; however, thealignment is not as strongas Yes.
The resultsforeachofthefourcriteriadiscussedinthissectionwere calculatedusingWebb’s methodology,reviewers’averaged ratings,andreviewers’comments.The resultsfor depth-of- knowledgeconsistency,categoricalconcurrence, range-of-knowledgecorrespondence,and balanceof representation arefound inAppendixC.
Source ofChallenge
Thepurposeofeachalignmentstudy wastodeterminethedegreeofalignmentamongthe contentstandardsfor eachgradeandthetestitemsfoundonthecorrespondinggrade-level CMA.Inaddition,theelectronicdatacapture toolprovidedopportunitiesforreviewerstooffer commentsand/orfeedbackonhowthetestquestionswere written.Reviewerswerealso encouraged tonote whetherthere wasasource-of-challengeissuewith a particulartest question orquestions.Asource-of-challenge issue mightincludeareviewer’sopinionthata particular questioncontainedmisleading informationorthataparticularquestionmightrequireprior knowledge.Allcommentsabouttheitemsand/or source-of-challengeissueswereprovidedto the California State Departmentof Education (CDE) forreviewandsubsequentaction,if required.
The source-of-challengecommentsarenotprovided inthisreport.Thefinalresultsof this alignmentstudyreflectonly theagreementbetweenthestandardsandthecorrespondingCMA. Inotherwords,thepurposeofthealignmentstudywasnottoprovideanopinionortoverify thegeneralqualityoftheCaliforniastandardsorthetest.Rather,thepurposeofthestudywas to determinethe degreeof alignment.
memo-dsib-adad-dec12item04
Attachment 1
Page 1 of 40
Alignment Study Process
Reviewers were asked to determine the degree of alignment between the standards (what studentsshouldknow andbeabletodo)foreachgradeandthetestquestionsfoundonthe corresponding CaliforniaModifiedAssessment.Inordertoaccomplishthistask,thealignment studyprocess involved four majorsteps:
Training
Assigningdepth-of-knowledgelevels to the standards foreachgrade or course
Taking each test
Determiningwhateachitemmeasuresandidentifyingthedepth-of-knowledgelevelfor each item
Ahigh-leveloverviewofthestepsintheprocessisprovidedonthenextpage.Thealignment study processalsoinvolvedtheelectroniccaptureofdata.Informationabouttheelectronicdata capturetooland its use in the process is providedbelow.
Useofthe Electronic Data CaptureTool
An electronicdatacapturetool was usedin thealignment studies. Thetool was designed
specifically tofacilitatethegatheringofindependentreviewers’judgments.Theapplication automatedtheprocess ofaligning thestandardsforagivencontentareaandthetestitemsfound on the correspondingCMA. Thetooland its reports madeitpossible togaugeinatimelymanner thealignmentbetween thestandardsandtheitemsontheCMAonthe basisofthecriteria.In addition,the toolalsoprovidedopportunitiesfor reviewerstoprovideadditionalinformation regarding items,including providing commentsrelatedtosourceofchallenge.Theitem-by- standard codings byreviewers werethen aggregated and analyzed.
Thenationalalignmentexpert,Dr.CarstenWilmes,providedtraining ontheoverallalignment processandthedepth-of-knowledge levelsandalsoservedastheleadfacilitator.Dr. Wilmeshas extensiveexperiencetrainingthird-party independentreviewcommitteemembersintheuseof electronicdatacapturesoftwareforalignmentstudies.Thetraining providedinformationon understandingnotonly thedepth-of-knowledge levelsbutalsoonhowtousetheelectronicdata capturetoolwhenassigningadepth-of-knowledgelevel to eachstandardand item.
memo-dsib-adad-dec12item04
Attachment 1
Page 1 of 40
Alignment Study Process
Step1: Receiving training
ReviewersreceivedtrainingonWebb’sdepth-of-knowledgelevels,thealignmentprocess,and
theuseoftheelectronicdatacapturetool.Thetrainingwasprovidedbythenationalalignment expert, Dr. Carsten Wilmes.
Step2: Dividing into content-area groups
Reviewers were divided into groups according to content area: English language arts,
mathematics,andscience.Reviewersreceivedadditionalhands-ontrainingontheuseofthe depth-of-knowledgelevels.
Step3: Determining thedepth-of-knowledge level ofeachCalifornia standard
Using the electronic data capture tool, reviewers individually determined the depth-of-
knowledgelevelofeachoftheCaliforniastandards.Agroupdiscussionfollowed.Reviewers reached consensus.
Step4: Taking the test
Reviewers took the CMA assessment, recording their answers in the test booklet.
Step5:Determiningwhateachitemmeasuredandthedepth-of-knowledgeofeachitem using the electronic data capturetool
Using the electronic data capture tool, reviewers independently determined the depth-of-
knowledge levelfor eachitemandthen identified matchestothe contentstandards.(Note: Reviewerswereallowedtoaligneachtestitemwithuptotwostandards,oneprimary andone secondary,andenter theinformationintotheelectronic datacapturetool.However,reviewers wereallowedtodetermineandenteronlyonedepth-of-knowledgelevelforagivenitemintothe electronicdatacapturetool.ThereviewersforMathematicsgrades 3–7were alsoaskedtoalign each item to amathematicalreasoningstandard.)
Throughoutthealignmentprocess,reviewersindependentlynotedanysourceofchallengefor each test item, providing written comments as necessary.
Step6: Answering debriefing questions
Using the electronic data capture tool, reviewers independently responded to debriefing
questions.
memo-dsib-adad-dec12item04
Attachment 1
Page 1 of 40
Alignment Study Participants
TheCMAalignmentstudy wascomposedofgradespangroupsforEnglishlanguagearts, science,andmathematics.Eachgroupwascomposedofeightreviewers.Fourofthereviewers for eachstudy wereCaliforniaeducators whohadextensiveteachingexperience,including teaching studentswithdisabilitiesand/oradministering theCMA,andexpertiseintheircontent areas.Theotherfourreviewersforeachalignmentstudywerenationalcontent experts. Each nationalcontentexpertalsohadexpertise intheir content area andexperience instandards development,curriculumand instruction development,testdevelopment,and alignmentstudies. Inaddition,one of the nationalcontentexpertsalsoservedasa group leaderfor eachgroup.The group leader’s task was to provide content-specific training on the depth-of-knowledge definitions and to facilitate theprocess described laterin this section.
California Experts
WhenselectingreviewersfortheCMAalignmentstudies,carewastakensothattheunique
diversity ofCaliforniastudentswouldberepresentedintheCaliforniaalignmentreviewers.An applicationletter was emailedtoCaliforniaDistrictCoordinatorstodisperse tothefield of teachersandadministrators. Prospective applicantssubmittedtheapplicationelectronically. Preliminaryselectionofparticipantswithatleastonealternatepergradespanwasmadeand submittedtoCDEfor approval.Verificationwascompletedtoensuretheapplicantseitherhad experienceteachingstudentswithdisabilitiesintheirclassroomsorhadadministeredtheCMA. It was determined that 100% of the participants had experience teaching students with disabilities and 56%of theparticipants hadadministered the CMA.
Thedemographicsthat wereconsidered in selectingparticipants areprovided in thetables below, and the actualcriteriaforselectingthegroups canbe found inAppendixH.
Table 2:CaliforniaReviewers’ Demographics—Region, Gender, andEthnicity
n= / RegionofCalifornia / Gender / Participants'EthnicityNorth / Central / South / Male / Female / Asian / Black / Hispanic / Two
or More Races / White
Reviewers / 32 / 8 / 6 / 18 / 7 / 25 / 2 / 2 / 4 / 4 / 20
Teachers / 24 / 7 / 3 / 14 / 6 / 18 / 1 / 1 / 2 / 4 / 16
Administrators / 8 / 1 / 3 / 4 / 1 / 7 / 1 / 1 / 2 / 0 / 4