ESEARCHRIEF

Publication #2003-014301ConnecticutAvenue,NW,Suite100,Washington,DC20008

Phone202-362-5580Fax 202-362-5533

TheUses(andMisuses)ofSocialIndicators:ImplicationsforPublicPolicy

By KristinAndersonMoore,Ph.D.,andBrettV.Brown,Ph.D.,withHarrietJ. Scarupa,M.S. February2003

verview Indicators–statisticalmarkers –areusedinmanyareasoflifetotrackpatternsand trendsovertime. Forexample,indicatorsareusedintheeconomicrealmtomonitortheupsand downsoftheeconomy,inthepublichealthfieldtotrackpatternsofdisease,andbythenation’sschoolsys- temstotrack studentachievement.Andwhilemostbaseballfansmightnotbe awareofit,they’reusing

indicatorswhentheycite“stats”onbattingaveragestodebatethemeritsofoneballplayeroveranother. Despitethewidespreaduseandacceptanceof indicatorsin so manyfields,indicatorstendtobeunderuti-

lizedinthebroadersocialpolicyarena.Yetsocialindicators,asindicatorsinthisarenaarecalled,canbe

veryvaluabletopolicymakers. Forexample, socialindicatorscanactasanearlywarningsystem,or “miner’scanary,”1 aboutaproblemsothatquickactioncanbetakentoaddressit.Considerthewayindi- catorsshowinganincreaseinbingedrinkingamongAmerican teenshavealertedthepublicaboutthis problemandspurredcoverageofitinthemedia.

Itmaybethatonereasonindicatorsarenotusedasfullyinthesocialpolicyarenaasinsomeothersisthe hazinessthatexists abouthowsocialindicatorsdifferfromothertypesofresearchinthesocialsciences. Inaddition,policymakersmayhavelittleornoknowledgeofthepurposesforwhichsocialindicatorsare bestsuited,andwhentheuseofindicatorsisinappropriate.ThisResearchBriefsetsouttohelpclarify theseissues. Itsuggestsfivepurposesthatsocialindicatorscanserve:description,monitoring,setting goals,increasingaccountability,and“reflectivepractice”(whichfunctionslikeaninternalevaluation).

Thebriefalsosoundssomecautionarynotesaboutthemisuseofsocialindicators. Forexample,itsug- geststhatitisinappropriatetousethesestatisticalmarkerstodeterminecauseandeffect.Thus,social indicatorscantellyouthattherateofbingedrinkingamongAmericanteens hasgoneupoverthepast decadebut,alone,theycan’ttellyouthataparticularfactororfactorscausedthisincrease.

ThisResearchBriefisdifferentfrommost ChildTrendsbriefs,whichcustomarilypre- sentdataandresearchfindingsrelatedto children,youth,andfamilies. Incontrast, thisbriefwasdevelopedtoraiseawareness andencouragefurtherdiscussionabouta research methodthatcanbehelpfultopolicy makersandothersconcernedwithimproving thewell-beingof childrenandtheirfamilies.

THERESEARCHCONTEXT

Togetabettergraspoftheappropriateuseofsocial indicatorsastoolsforpolicymakers,program

developers,andopinionshapers,itmay behelp- ful toreviewseveralofthebasictypesofquanti- tative socialscienceresearch. Hereweidentify fourmajorkindsof suchresearch:

■ Experimentalstudiesaretheonlytypeof researchthat trulycanaccountforcause and effect,and,therefore,areconsideredthe“gold standard”formakingconclusions aboutwhat causeswhat.Inexperimentalstudies,individ- ualsarerandomlyassignedeithertoatreat- mentorprogramgrouportoacontrolgroup, andthentheiroutcomesarecompared.An example wouldbeexperimentalstudieson

1The proverbialterm“miner’scanary”originatedinthepracticeofminerstakingacanarywiththeminto themineswheretheyworked.If therewasapoisonousgasleakin themines,thecanarywouldfalldead,alertingtheminerstothedangerandallowingthemtogetawaytosafety.

youthmentoringprogramsinwhichsome adolescentsarerandomlyassignedtohavea mentorandothersarenot.(Quasi-experimental comparisonstudiesaresimilartoexperimental studies exceptforonecrucialdistinction:the comparison groupsarenotassignedrandomly, socausalconclusionscannotbereachedwith certainty.)

■ Basicresearchstudiesaimtoincrease our basicunderstandingofaparticulartopic,such ashowdivorceseemstoaffectchildren. Longitudinal,multivariatestudieswouldfall under this category,thatis,studiesthatfollow individualsacrosstimeandexamine threeor morevariables. Inthecaseofchildrenof divorce,suchstudiescouldprovidesome insightsintohowwell–orhowpoorly–these childrenfareovertheyears andwhatfactors mightaffecttheiradjustment.

■ Implementationresearchstudiesserve essentiallyasmanagementtoolsthatallow peopletounderstandwhetherandhowwella servicewasdelivered. Forexample,astudyof thistypecouldhelpdetermine whether implementing anearlychildhoodeducation programactuallyresulted intheprovisionof measurablehigh-qualityservicestopreschool children.

■ Indicatorstudiesaretypically basedon cross-sectionaldata,thatis, informationthatis collectedatonepointintime,usuallythrough periodicallyadministeredsurveys.Good examplesmightbetheKIDSCOUNTreports2 spearheadedbytheAnnieE.CaseyFoundation andTheRightStartreports,3acollaborative effort betweentheKIDSCOUNT initiativeand ChildTrends.Drawingonmultiplesources,the reportsprovideover-timedataon keymeasures thatdescribethewell-beingofinfants,children, oryouth. Forexample,amongTheRightStart indicatorsaremeasuresofbeingborntoateen mother,oflowbirthweight,prematurely,and toamotherwhosmoked.

KEYUSESOFSOCIALINDICATORS

Socialindicatorsarewidelyusedbecausethey serveanumberof purposes:

■ Description:toinformcitizens and policy makersaboutthecircumstancesoftheir society,totracktrendsandpatterns, and toidentifyareasofconcernaswellas positiveoutcomes. Forexample,numerous reportsprovidedescriptivestatisticalinforma- tionaboutthecircumstancesofAmerica’s childrenandfamilies. Perhapsthebest knownare America’sChildren: KeyNational Indicators ofWell-Being,4the flagship documentoftheFederalInteragencyForumon ChildandFamilyStatistics,andtheannual TrendsintheWell-Being ofAmerica’s ChildrenandYouth,5whichisdisseminated bytheOfficeoftheAssistantSecretaryfor PlanningandEvaluation,U.S.Departmentof Health andHumanServices. Morerecently, online dataresourcessuch asthe ChildTrends DataBank6 havebecomeavailable. Such indicatorreportsandresourcesprovideahandy wayforthepublicorpolicymakerstogeta graspoftrendsthatappearpromising(e.g.,a decline inchild andyouthdeaths) andthose thatappear troubling(e.g.,anincreasein childhoodobesity).Indicator reportsalsooften provideinformationonsubgroup differences (e.g.,bygender,race/ethnicity, andpoverty status)withinthelargerpopulation, suchas therecentstatisticsshowingthatHispanics havehigherteenbirthrates thanwhites or blacks.Thus,usingsocialindicatorsforthe purposeofdescriptioncantellyoumuch morethanwhatAmericalookslike. Social indicators alsocanhelptodescribethe variabilitywithinthepopulationandthe differencesacrosssocialgroups.

■ Monitoring:totrackoutcomesthatmay ormaynotrequirepolicy intervention of somekind.Mostpeoplearefamiliar with usingindicators forthepurposeofmonitoring inthepublichealthfield.Forexample,theU.S. Centersfor DiseaseControlandPreventionhas setupavarietyofdiseasesurveillancesystems toidentifyemergingthreats tothehealthof thenation,aswellasthreatstospecific communitiesandtospecificpopulationswithin communities. Wehaveseenthisapproachin

2

3

4

5

6

operationinthewarnings issuedaboutthe presenceof theWestNilevirusincertainlocali- ties. Similarly,schoolsystems oftenuse indicators tomonitorhowwellchildren are doingonstandardizedtests,oftentargeting thoseschoolswithlowscoresforspecial interventions.Anexamplefromthesocial policyarenawouldbeusingsocialindicatorsto trackmeasuresofchildoutcomesthroughthe yearsofwelfarereforminordertomonitorthe well-beingofchildren,particularlylow-income children,duringatimeof majorsocialchange.

■ Settinggoals:toestablishquantifiable thresholdstobemetwithinaspecific timeperiod. Here, again,examplesfromthe publichealthandeducationfieldscomereadily tomind. TheHealthyPeople2010initiative,7 developedbytheU.S.Department ofHealth andHumanServices, identifies467specific, measurable goalsaimedatimprovingthe healthofallAmericansbytheyear2010.And, undertheprovisionsoftheyear-oldNoChild LeftBehindAct,8statescreatetheirown standardsforwhatachildshouldknowand learnforallgrades;andeachstate,school district,andschoolisexpected toshowyearly gainsinmeetingthosestandards.Usingindica- torsinsuchwaysallowsasocietytoexpressthe goalsitdeems importantandthevalues it cherishes(inourexamples,ahealthycitizenry andwell-educatedchildren). Thisroleofsocial indicatorsinsettinggoalsthatexpress values underscorestheimportanceofdevelopingsocial indicatorsthattrackoutcomesthatarepositive (e.g.,teenvolunteering) as wellas thosethat arenegative(e.g.,teenhomicide.)

■ Increasingaccountability:toachieve positiveorimproved outcomes.Business managers areheldaccountableforthe profitabilityoftheirenterprises,coachesfor the performanceoftheirteams,andpoliticiansfor meetingtheneedsoftheirconstituents. Increasingly,governmentandprivatefunders areusingsocialindicatorstoholdstates, communities, agencies, andindividual programsaccountableforimprovingoutcomes forchildrenandyouth. Thewordoutcomesis

noteworthy. Itsignals achangefromusing inputdata tousingoutcomedata tomeasure accountability. (Inputdatamightbethe numberofhoursateacherteachesorthe numberofhoursofpsychotherapyapatient receives. However, outcomedatawouldshow whethertheteacher’sstudents showed improvementsintheirgradesorwhetherthe patientwhosawa psychotherapistexperienced improvementsinhisorhermentalhealth.) Usingsocialindicatorstoincreaseaccountability issometimesconnected torewards or sanctions.Forexample, atthefederallevel, underwelfarereform,statesthatreduced birthsoutsideofmarriage themostwithout increasingabortionshavebeenrewardedwith substantialbonuspayments.Therisk,of course,isthatmany factorscandetermine trendsandonlysomeofthemmaybeunderthe controlofthepersonororganizationbeingheld accountable. Thus,cautionisnecessarywhen indicators areusedforthepurposeof accountability.

■ Reflectivepractice:toinformpracticesof communities andindividualprogramson anongoingbasis. Sometimescommunities developformallogic models–graphicplanning toolsdesignedtoshowhowparticularprogram activitiesarerelatedtoexpectedoutcomes. Theyusethesemodelstomonitorwhether theseprogramactivities areaccomplishing theirobjectives.Forexample,intheplanning stages forevaluatingSouthCarolina’sschool readinessinitiative,FirstStepstoSchool Readiness,ChildTrendsresearchersdevel- opedalogicmodel9tohelppeopleunderstand thelinksbetweenmaking aninvestment ina particularschoolreadinessprogramandwhat thatmightbeexpectedtoaccomplishovertime. Suchmodelsusesocialindicators tomonitor progressinallsuchlinksinthemodel.

Oneofthegreatadvantagesofsocialindicators (overothertypesofresearch),regardlessofthe particularpurposeforwhichtheyareused,isthat theycanbecomeavailablequicklysothattheycan beusedquicklytoinformandimprovepublicpoli- cy.Forexample,recentsocialindicators showing

7

8

9Thelogicmodelisincludedintheevalutationreport

adeclineintherateofteenbirthsoutsideof marriagemightbeseen as aharbingerofgreater familystability, moretwo-parentfamilies,and betteroutcomesforchildrenadecadelater. Poli- cymakersconcernedwith promoting and encour- agingmarriageneedtohavethistypeofinforma- tion,andthesooner,thebetter.

THEPROPERUSEOFSOCIALINDICATORS Asseenabove,socialindicatorscanbehelpful toolsforpolicymakers,practitioners,andthe public,but usingthem correctly requiresattentiontoanumberof issues:

■ Socialindicatorsneed tobemeasuredfor theappropriatepopulation.Forexample,if apolicyfocusesonservicesforlow-income children, thentheoutcomesshouldbe measuredfor low-incomechildren–not middle- classorallchildren.

■ Socialindicatorsneedtobemeasuredat theappropriategeographiclevel. For example,whilethe1996welfarereform resulted fromtheenactmentofafederallaw, welfarereformplaysoutatthestate andlocal level. Lookingjustattrendsonthenational level mayobscurehowapolicyisaffecting individualsintheirownstatesandhome communities.

■Socialindicators needtobewell- conceptualized. Thatis,socialindicators needtoaccuratelyreflecttheconcept thatthey areintended tocapture.Again,welfarereform providesausefulcontext. Ofcourse,welfare hasalwaysbeenaprogramaboutchildren;for manyyears,itwas calledAidtoFamilieswith DependentChildren.Properconceptualization suggestsrecognizingthatwelfarereformisnot justaboutoutcomesforadults,suchaswork; orforfamilies,suchaspoverty,butfor childrenaswell;indicatormeasuresneedto reflectthisconceptualization.

THE MISUSEOFSOCIALINDICATORS Socialindicatorscanbemisused–eitherinten- tionally(forpoliticaladvantage,forexample)or unintentionally(duetoinadequatetrainingand technicalassistanceforpractitioners andpolicy makerswhowouldusesocialindicators). Inap- propriateusesof socialindicatorsincludeemployingthem:

4

■ Toclaimcredit(ortocastblame)for societaltrends. Considertheissueofthe decline intheteenbirthrateoverthepast decade.Someadvocatesofabstinenceforteens claimcreditforthedeclinebased ontrendsin sexualexperience. Atthesametime,some advocatesofcontraceptivesforteensattribute thedeclinetoimprovementsincontraceptive use.Butdrawing eitherconclusionfrom indicatordataalonecanbemisleadingbecause trendsinindicatorscannotresolveissues of causality. Indeed,thecausesofsocialchange are generallyvery complexanditisdifficultto creditjustoneoreventwofactorsascausal.

■ Toclaimcreditforprogramsuccess. For example, supportersofsomeanti-drug programsmightcitethenumberofstudents whoparticipatedintheseprogramsandthe percentofstudents whohaduseddrugsas markers oftheprograms’success. Butsuch indicators cannotresolvewhetherthepartici- pantsactuallywerelesslikelytousedrugs becauseoftheprogramthantheywouldhave beeniftheyhadn’tbeenenrolledinthe program. Forthat,experimentalevidenceis needed.

■To evaluate the performance of individualsor programs, without consideringthelargercontext. For example,ifthenumbersshowthatteen drinking inacommunitygoesup,this shouldn’tnecessarilybetakentomeanthatit istimetothinkaboutfiringalltheschool healtheducators.Theremaybeotherforcesat work.Suchmisuseofindicatorsforpurposesof accountabilityisalwaysaconcern.

Generallyspeaking,socialindicatorsmakepoor toolsfor formal,scientificevaluationsofprograms, policies, andpersons.Traditionally, theroleof socialindicatorsinevaluationshasbeenrather limited,functioningas“miners’canaries”toiden- tifypoliciesorprogramsthatmaybeparticularly promising(orunpromising)anddeservingoffor- malevaluationusingmorerigoroustechniques.At thesametime,socialindicatorscanoftencomple- mentinformationderivedfromothertypesof research,togetherprovidingastrongercomposite pictureoftheeffectsofaprogram orpolicy.We haveseenthis,forinstance,insomeofthe

researchonhowwelfarereformprogramsappear tobeaffectingthewell-beingofchildren.Boththe indicatordataandexperimentalstudiessuggest thatforthemostpart,welfarereformhas neither harmed norhelpedchildrentotheextent anticipatedbyadvocates.

CONCLUSIONS

ThisResearchBriefmakes thecasethatsocial indicators,whenproperly used,canbevaluable toolsforpolicymakers,practitioners,themedia, andthegeneralpublic. Thesestatisticalmarkers canbeusedtodescribethecircumstancesofour society, tomonitorhowwellwearedoing,toset goalsthatreflectsocietalvalues,toincrease accountabilityforpoliciesandprograms,andto informpracticesinagivencommunityorpro- gram. Also,whetheratthestateorthenational level,socialindicatorsareespeciallywell-suitedto serveas“miners’canaries,”alertingsocietyabout trends,whetherpositiveornegative.

Moreover,particularlyinaneraofdevolution, whenresourcesandresponsibilitiesarebeing movedtothestates,socialindicatorsatthestate levelcanbeusefultostate-levelpolicymakers whenusedforthesepurposes.

Despiteallthis,whencomparedwiththebusiness worldandthepublichealthfield,indicators are usedfarless ofteninthebroadersocialpolicy arena,wheretheyarepotentiallyjustashelpful. Greateruseofsocialindicatorstoexpandour understandingofchildwell-beingisclearlyacase inpoint.

Anydiscussionoftheuseofsocialindicatorsalso needstoacknowledge theircurrentlimitations. Forexample,insomeareas,wemayhavegood measures,butwemaynothavegooddata. That seemstobethecase intracking thewell-beingof childreninthechildwelfarepopulation.The most basicinformationaboutthese children (e.g.,how manyhaveasthma,howmanyrepeatedagradein school)isoftennotavailable.Inotherareas, we mayhavedata,butwelackgoodmeasuresorhave measuresthatareinadequate.Consider“religiosi- ty,”thatis,religiousbeliefandpractice. Data fromnationalsurveyscantellusaboutteenagers’ churchattendance,butchurchattendancemaybe animperfectindicatorofreligiousbeliefand practice.Wealsolackadequatemeasuresonsuch

importantconceptsasparent-childcommunication andadolescentmental andemotionalhealth.

Anotherconcern, asmentionedearlier,istheneed todevelopandtrackmeasuresofpositivedevelop- ment–thekindofcharacteristicsthatcanhelp individualsdowellinlife, suchascloseand warm relationshipswithfamilyandfriends,civicinvolve- ment,ethicalbehavior,andaloveoflearning. Mostofthesocialindicatorsthatwetrackand analyze aboutchildrenandteensarenegative, suchasratesofinfantmortality, teenpregnancy, andsubstanceabuse.Whiletrackingnegativeout- comesisessential,positivemeasurescanenable policymakers,practitioners,themedia,andthe generalpublictofocusnotonlyonpreventing what’swrongbutalsoonsupporting andnurtur- ingwhat’sright.Developingsuchmeasures may well bethenextfrontierintheuseofsocial indicators.

Meanwhile,thosewhoareseekingtoexpand the rigorousandappropriateuseofsocialindicators oftenexplain theirconcernbyusingthesimple adage: “Whatgetsmeasured,getsdone.”

Someofthematerialinthisbriefisbased onapre- sentationmadebyKristinAnderson Moore,Child TrendsPresidentandSeniorScholar,toanAdHoc MeetingonMeasuresofChildWell-Beingthatwas heldattheBrookingsInstitutioninWashington,DC, September20,2002. Thebriefalsodrawsonideas presentedbyBrettV.BrownandKristin Anderson Moorein“Child andyouthwell-being:Thesocial indicators field,”achapterinHandbookofApplied DevelopmentalScience,publishedin2002 bySage Publications,andbyBrettV.BrownandThomas Corbettin“Socialindicatorsandpublicpolicyinthe ageofdevolution,”whichwillbeincludedinTrends inthewell-beingofchildrenandyouth,publishedby Child Welfare League of America Press(nowinpress).

AShortBibliography

Ben-Arieh,A.,Kaufman,N.,Andrews,A.,Goerge,R.,Lee,B.,Aber, J.L.(2001).Measuringandmonitoringchildren’swell-being. Dor- drecht:KluwerAcademicPublishers.

Brown,B.V.,Corbett,T.(in press).Socialindicatorsandpublicpolicy intheageofdevolution.InR.Weissberg,L.Weiss,O.Reyes,H. Wal- berg(Eds.),Trendsinthewell-being ofchildrenandyouth. Washing- ton,DC:ChildWelfareLeagueof AmericaPress.

Brown,B.V.,Moore,K.A.(2002).Child andyouthwell-being: The socialindicatorsfield.InR.Lerner (Ed.),Handbookofapplieddevelop- mentalscience.ThousandOaks,CA:SagePublications.

Hauser, R.,Brown,B.V.,Prosser,W.(Eds.).(1997).Indicatorsof children’swell-being.NewYork:RussellSageFoundation.

Hollister,R.,Hill,J.(1995).Problemsintheevaluationofcommunity- wideinitiatives.InJ.Connell,A.Kubisch,L.Schorr,C.Weiss(Eds.),New approachestoevaluatingcomprehensivecommunityinitiatives:Concepts, methods,andcontexts.Washington,DC:TheAspenInstitute.

Moore,K.A.(1997).Criteriaforindicatorsofchildwell-being.In R.Hauser, B.Brown,W.Prosser(Eds.),Indicatorsofchildren’swell-being.New York:RussellSageFoundation.

Murphey,D.(2001,June).Creatingcommunitycapacitytouseindicators. Paperpresentedattheconference KeyIndicatorsofChildand YouthWell- Being:CompletingthePicture,Bethesda,MD;available at

Stagner,M.,Zweig,J.(2001,June).Indicatorsofyouthwell-being:Taking thelongview.PaperpresentedattheconferenceKeyIndicatorsofChildand YouthWell-Being:CompletingthePicture,Bethesda,MD.

ChildTrendsgratefullyacknowledgestheAnnie E.Casey Foundationforsupportofitsworkonsocialindicators, especiallyastheyrelatetochildwell-being,aswellasfor itssupportofourcommunicationsactivities. Wealso thanktheJohnD.andCatherineT.MacArthurFoundation forongoingsupportofourResearchBriefseries. Addi- tionalsupportforChildTrends’communications effortsis providedbytheDavidandLucilePackardFoundation andtheWilliamandFloraHewlettFoundation.

Editor:HarrietJ.Scarupa

©2003ChildTrends

ChildTrends,foundedin1979, isanindependent,nonpartisanresearchcenterdedicatedtoimprovingthelivesof childrenandtheirfamiliesbyconductingresearchandprovidingscience-basedinformationtothepublicanddecision- makers.ForadditionalinformationonChildTrends,includingacompletesetofavailableResearchBriefs,visitour Websiteat Forthelatestinformationonmorethan70keyindicatorsofchildandyouth well-being,visittheChildTrends DataBankat