Exploratory Software forConfidence Intervals

UserNotes

GeoffCumming

SchoolofPsychologicalScience LaTrobeUniversityBundoora, Australia3086

ESCI (Pronounced "esky")

Commentsandsuggestionsarewelcomed.

Thesenotesareprovidedviafreedownloadfrom the ESCI website

WatchtheESCIsitefornewsandupdates. SeealsotheFAQsection.

UserNotes,version:23July2001

ESCIispublishedby LaTrobeUniversity, BundooraAustralia3086

2001

Acknowledgements

MichaelSmithsonraisednoncentraldistributionsas atopicof interest.

BruceThompsonpromptedtheCummingFinchpaper,andadvisedateverystage.

RodneyCarrshowed whatExcelcando.

FionaFidler,Sue FinchandNeilThomasoncollaboratedandadvised.

JoannaLeemanadvised,andsuggestedthenameESCI.

Geoff Robinsonprovidedtheroutinesfor calculatingnoncentralt.

DavidWalshbuiltthewebsite,developedgraphicsandadvised.

Contents

Overview2

TheCI primerarticle,CummingandFinch(2001)3

Downloading,installingandrunningESCI5

Thecomponentsof ESCI-delta7

  • NonCentral t
  • Power
  • Cljumping
  • Cloriginal
  • Cldelta
  • MAthinking

Overview

ESCI is a set of interactive simulations that runs under Microsoft Excel. ESCI-delta is the first set of ESCI simulations, and comprises 6 Excel workbooks.

With ESCI-delta you can:

  • Explore many Confidence Interval (CI) concepts.
  • Calculate CIs for Cohen’s standardized effect size 
  • Explore noncentral t distributions and their role in statistical power.
  • Use CIs for simple meta-analysis, using original units or d (standardized) effect sizes.
  • Explore all these concepts via vivid interactive graphical simulations.

NoteCohen'sstandardisedeffectsizeisamean,or meandifference,divided by astandarddeviation.Itisthusanumberof standarddeviations,likeaz score.In theCummingandFinch(2001) articlewe use thesymbolfor thepopulationparameter,andd for thesamplestatistic.(On theESCIwebsited is used for both,becausethesymbolcannotbereliablydisplayedon allbrowsers.)

The rationale for ESCI

Statisticsreformrequiresmuchwideruse of confidenceintervals(CIs) andeffectsizemeasures,inoriginalunitsandinstandardisedunits.WethereforeneedCIsfor standardisedeffectsizes.Unfortunatelyinmanycasesthesearenoteasilycalculated, and requireuse of noncentraldistributions.ESCIenablesyou tofindtheCI for Cohen'sfor your own data,whichrequiresuse of noncentral tdistributionsandaniterative algorithm.ESCIalsoallowsyou toexplorenoncentraltitself,anditsmostfamiliar application,whichisthecalculationof statisticalpower.

Statisticsreformalsorequireswideruse of meta-analysis.Thebestway toexplore(and teach)simplemeta-analysismaybeviaCIs, graphicallypresented.ESCIsupports the developmentof 'meta-analyticthinking'inoriginalmeasurementunitsandin standardisedunitsusingCohen's.

Futuredevelopments

FurtherESCIsimulationsareindevelopment.Theywillbeannouncedon theESCI

website.

TheCIprimerarticle,CummingandFinch(2001)

Thefollowingarticleexplainsconfidenceintervalsfor Cohen's. Itgivesanintroduction tononcentraltdistributionsanddiscusses relatedconceptsincludingpower andsimple meta-analysis.Itisillustratedwithpart-imagesfromESCI.ESCI-deltaisthesetof six Excelworkbooks thatarereferredtointhisarticle.

Cumming,G., &Finch,S. (2001). A primeron theunderstanding,use andcalculationof confidenceintervalsbasedon centralandnoncentraldistributions.Educational andPsychological Measurement,61(4, August), 530-572.

Abstract Reformof statisticalpracticeinthesocialandbehaviouralsciences requireswideruse of confidenceintervals(CIs), effectsizemeasuresandmeta- analysis.Theauthorsdiscuss four reasons for promotinguse of CIs:they(i)

arereadilyinterpretable;(ii)arelinkedtofamiliarstatisticalsignificancetests; (iii)canencouragemeta-analyticthinking;and(iv)giveinformationabout precision.Theauthorsdiscuss calculationof CIs for abasicstandardisedeffect sizemeasure,Cohen's(alsoknown as Cohen'sd), andcontrastthesewiththe familiarCIs for originalscoremeans.CIs forrequireuse of noncentralt distributions,whichtheauthorsapplyalsotostatisticalpower andsimplemeta- analysisof standardisedeffectsizes.TheyprovidetheESCIgraphicalsoftware, whichruns underMicrosoftExcel,toillustratethediscussion.Wider useofCIsfor andothereffectsizemeasuresshouldhelppromotehighly desirablereformof statisticalpracticeinthesocialsciences.

In thisarticlewe outlinethecasefor reformof statisticalpracticeby researchersin psychologyandotherdisciplines.Wegiveformulasandexplanationsfor calculatingCIs in originalmeasurementunits,andfor Cohen's. Someexamplesareincluded.Noncentralt distributionsaredescribed,andused indiscussions of statisticalpower andof simple meta-analysis.

Itmaybemostusefultoreadthepaperwhileworking withtheESCI-deltasoftware. Sincewritingthepaperthenew editionof theAPA manualhas appeared:

AmericanPsychologicalAssociation.(2001).Publication manualoftheAmerican PsychologicalAssociation(5thed.).Washington,DC:Author.

Thefollowingquotesfrom theManualindicatethateffectsizemeasuresandconfidence intervalsarelikelyto berequiredroutinelybyjournaleditors,andwillbeseenoftenby anyonereadingpublishedresearch. Thesearehighlydesirabledevelopments.

Forthereadertofully understandtheimportanceofyourfindings,itisalmost alwaysnecessary toincludesomeindexofeffectsizeorstrengthofrelationship inyourResultssection.Youcanestimatethemagnitudeofeffectorthestrength oftherelationshipwithanumber ofcommoneffectsize estimates...Thegeneralprinciple tobe followed...istoprovidethereadernotonlywith informationabout statistical significancebutalso with enoughinformationtoassessthe magnitude oftheobservedeffectorrelationship.(pp.25-26, emphasisadded)

Thereportingofconfidenceintervals...can beanextremelyeffectivewayof reporting results. Becauseconfidenceintervalscombineinformationon locationandprecisionandcan often be directlyusedtoinfersignificance levels,theyare,ingeneral,thebestreportingstrategy. Theuseofconfidence intervalsisthereforestronglyrecommended.(p.22,emphasisadded)

Theaimof ESCIistoprovidetoolsthatmayassistresearchers,teachersandstudentsto exploretherelevantconceptsandtoanalyseandpresenttheirdatainaccordancewiththese new requirements.

Downloading, installingandrunningESCI

Downloading

At theESCIsitethedownloads pagegivesdownloadof theseUser Notes, andaccesstothesoftwaredownloads webstore,whichoffers:

  • Freedownloadof theDEMO versionof ESCI-delta.
  • Download atlow costof thefullversionof ESCI-delta,withanINDIVIDUALlicence.
  • Download of thefullversionof ESCI-delta,withaLOCAL NETWORK licence, whichpermitsuse on anetworkof notmorethan50 computers.

For bulkdealsfor studentlicences,or anyotherlicenceenquiriespleaseemail

.

Thedownloadisof a1.2MB zipped.exefile.Savethistodisk,thendouble-clickitfor automaticunzippingintoafolder,whichcontainsthesixExcelworkbooks of ESCI-delta, totalsize3.5MB. Thelargestof theseisCldelta,at1.8MB. If thefolderincludesa README filepleasereadthatfirst.

DEMOversion

TheDEMO versionisnottime-limited.Itisthesameas thefullversionexceptthaton eachworksheetafew cells(parameters,or dataentrycells)aregreyedout.Thesecells maynotbechangedby theuser. In additionthesheetsareprotectedwithapassword and maynotbechanged.

UsingESCI

No installationisrequired.Torun ESCIyou needalicensedcopyof MicrosoftExcel97, or Excel2000,runningunderWindows 95 or later.(Unfortunately,as usualfor Excel,the transfertoMacintoshisusuallynotfullysuccessful.)

You must,ifaskedwhileopeningaworkbook, enablemacros. On eachIntrosheettherearetwo hints:

  1. Checkthatcommentsarevisible.Somecellsaremarkedwithsmallredtriangles.Restthemousethereandseeapopoutcomment.If thesearenotvisible,go toTool/Options/Viewandclick'Commentindicatoronly'. (Click'None'tohide comments.)Commentsgivethemainguidancefor usingtheworksheets.
  2. If your screenresolutionis1024x768, theusualExcelzoomfactorof 100% should giveasuitablesizeof image.Withascreenresolutionof 800x600 itmaybe necessarytochangethezoomfactor(on thetoolbar,or go toView/Zoom)of each sheettoabout85%. Otherzoomfactorsmaybeused, butavoidusing80% or less, or thereisarisk thatsomenumbersmaynotbedisplayedproperly:If thisproblem occursthenumberisreplacedby hashes (####).

Nothingishidden,so you canexamineformulasas you wish. In thefullversionthe worksheets areprotectedbutwithoutpassword (exceptthefirstIntrosheet,whichmaynot bechanged).You canthereforeunprotectasheet(Tools/Protection/Unprotectsheet)andexploreas you choose.Caution:Itisveryeasytomakeunwantedchangeson an unprotectedsheet.Beextremelycautiousaboutsaving.

Of courseitisessentialtokeepanarchivecopyof theoriginalworkbooks. Itisgood practicetostartwork eachtimewithafresh copyof theoriginalworkbook, andtosave onlywithanew name.

NonCentraItdistributionsandinteractivity

Probabilitiesunderthenoncentraltdistributionsarecalculatedby summinga100-term series.Curves areformedof 200 points,eachrequiringsuch acalculation.(In NonCentraltandPoweryou canseethe200x100 arrayof valuesby scrollingright.)In Cldeltathepointsof thetwo rednoncentraltcurvesarecalculatedby summinga50-term series,whichgivesacceptableaccuracyinthissituationandnotsuch alargefile.(Thetail probabilitiesarecalculatedwiththe100-termseries,for accuracy.)

The extensive computation needed for the noncentral t curves means that updating of the graphs is not instantaneous. Only on a very fast computer does dragging of a slider give smoothresponsivemovementof thecorrespondingcurve.

Therearecommentsaboutaccuracyinvariousplaces.In someextremesituationsthe calculationisnotsufficientlyaccurate,so avalueor acurveisnotdisplayed,or informationisgivenaboutthedegreeof precisionachieved.

Enteringdata

In CloriginalandCldeltayou typeinyour own data.You canpastedataintothedatacells,fromtheclipboard,afteryou havecopiedtotheclipboardfrom,for example,a differentExcelworkbook, or MicrosoftWord.Thereisaproblemhowever:Material pastedintoExcelbypasses anydatavalidationchecks,so for exampletextwould notbe rejectedwhen pastedintoanESCIdatacell.Evenworse, thevalidationsettingsarewiped by thepasteoperation.

Therefore,pastedataintoESCIwithcare,andcheckyou havejustonenumberineach cell.Don'tsavethefileafterpastingdatabecauseESCI'sdatavalidationtestshavebeen removed.

ESCI'sgraphs

As you movethemousepointeroverthegraphs (Excel'charts') inESCI,you mayseemanyyellowpopoutswithnumbersandlabels.At Tools/Options/Chartyou cantickor cleartwo boxes tospecifywhetheryou will'Shownames'and/or'Showvalues'. Itmay bebesttoclearthe'names' box sincemostnamesarenotinformative.However you may prefertosee'values' sinceinmanycasesthesegiveaprecisevaluefor aplottedpoint.

Printing

ESCIoffers no additionalprintingfacilities.You canuse thePrintScreenkeytocopya fullscreenimagetotheclipboard,for pastinginto,for example,MicrosoftPaintor Word. Alternativelyyou canprintfromExcel:Printjustthefirstpage,or setanareafor printing (File/PrintArea)andprintthis.

Thesimulationsthat makeupESCI-delta

SeetheCIprimerarticlefor thestatisticalcontextandformulasrelevantfor each simulation,andfor suggestedconclusionsthatmaybedrawn fromexplorationwiththe software.For severalof theworkbooks, followtherednumbersandnotethepopout comments.

Throughout ESCI-delta underlying populations are assumed Normal, and of homogeneous variance where that is relevant. All NHST (Null Hypothesis Significance Testing) is two- tailed.You canusuallysettheType1 errorrate , or theconfidencelevel,toanyvalueyou wish. InPowerandMAthinkingthefollowingsymbolsareused tomark conventionalstatisticalsignificancelevels(two-tailprobabilitiesthroughout):

ns / .10  p
? / .05  p.10
* / .01  p.05
** / .001  p.01
*** / p.001

NonCentralt

Explorenoncentraltdistributionsandcalculateaccurateprobabilities.

Thefamiliartdistributionhas thesingleparameterdf. Noncentraltdistributionshavein additionanoncentralityparameter. Hereyou canuse thecontrolstoseehow theshape of thenoncentraldistribution(red)changeswithdfandthenoncentralityparameter.Click boxes todisplayor hideacentraltdistribution(blue),for comparison,andvariousother features.You canalsodisplayaccuratenoncentralttailprobabilityvaluesfor your chosen df,, andtvalues.Noncentraltdistributionsareessentialfor statisticalpower andfor CIs for Cohen's.

Power

Explorestatisticalpower, whichinmostcasesrequiresnoncentraltcalculations.

Clicktodisplaythecentraltdistribution(blue)thatapplieswhen H0istrue,and/orthe noncentraltdistribution(red)thatisappropriatewhen Haistrue.Displaythetwo-tailrejectionregionfor thestatisticalsignificancetest,thenthetailarea(s)undertheredcurve thatgivestatisticalpower. Vary df,(Type1 errorrate),and(whichisdeterminedby theeffectsize, i.e.thedistancebetweenthenullandalternativehypothesisedmean values).You canalsodisplaythepercentagesof experimentsthat,ifHaistrue,would be expectedtogivevarioussignificancetestoutcomes:Thesepercentagesmaybesurprising.

Cljumping

Repeatedsampling,toillustratebasicconceptsofconfidence intervals(CIs).

Takeindependentsamplesfromanormalpopulation.Setthepopulationparametersand selectthesamplesizen. Clicktodisplayor hidethepopulationdistributionandvarious otherfeatures.Clicktotakeasinglesample,shown as adotplot.Means of successive samplesareshown cascadingdown thescreen.Runthesimulationandseethemeans 'dance', illustratingthe(surprisinglylarge) extentof samplingvariabilityfor your chosen populationandn. Setaconfidencelevelandshow aconfidenceintervalfor_ for each sample.Seta_0comparisonvalue(usually= _) for themeanandseewhichCIs capture thisvalue.One keyideaisthattheCIs vary,not_.

Cloriginal

CalculateanddisplayCIs foryour data, orthreesimpleexperimentaldesigns.

Thesethreesimpletoolsallowyou toenteryour own datafor threedesigns:Case1, Single Group; Case2, Two IndependentGroups; andCase3, PairedData.Thedataareshown as a dotplot.For Case1 you selecta_0valueas areferencevaluefor themean.TheCI, based on your dataset,for thepopulationmeanisshown. NotingwhethertheCI captures_0isequivalenttoconductingthecorrespondingttest.Thesetoolsuse original measurementunitsthroughoutandareprovidedas anintroductorysteptowards themore complexCldelta,inwhichstandardisedunits(Cohen'sd and) areshown alongside originalunits.

Cldelta

CalculateanddisplayCIs forstandardisedeffectsizes(Cohen's), orOne andTwo Groupdesigns. Requiresuse of NonCentraltdistributions.

Enteryour datafor Case1, SingleGroup, or Case2, Two IndependentGroups. The dotplot andCI areshown, inoriginalunits,as inCloriginal. Seta _0comparisonvalue, whichserves as thezerofor measurementof Cohen'sdand, thestandardisedeffectsize. ClicktodisplaytheCI for, butnotethatthisisnotyetaccurate.Tofindaccuratevalues,therednoncentraltdistributionsmustbepositionedso thattheirtailsthatoverlapd areof size(/2).Toachievethisyou canmovethesliders,or you canclickthebuttonstohave Exceldo thework.

MAthinking

ExploresimpleMeta-analysis, basedon graphicaldisplayof CIs, oreffectsizesin originalmeasurement units,and forstandardisedeffectsizes(Cohen's).

Themostbasicmeta-analysisconsistsof simplepoolingoverstudies,weightedby sample size.Thismaybecarriedoutinoriginalunits,or instandardisedunits.A simulationis providedfor eachof these,for theCase1, SingleGroup design.For thestandardisedunits, enterthed andnvaluesfor up to10 previousstudies.ClickthebuttonstohaveExcelfind anddisplaytheCI forfor eachstudy,andfor allthestudiescombined.You canalso enterd andnfor anadditionalstudy(perhapsour currentstudy) andclicktoshow theCI for thisstudy,andfor allstudiescombined.Clicktoshow thestatisticalsignificancetest results:Explorationof exampleswillprobablysuggest thateffectsizes(andsamplesizes) aremuchmoreinfluentialandinformativethansignificancelevels.

1