Statistical Training Manual - Help System

1.800.711.8363

TABLEOFCONTENTS

I. Introduction ………………………………………………………………………………………………2

II.Statistical Significance Testing ………………………………………………………………………2

A.What isstatistical significance?

B.How shouldstatistical significance be interpreted?

C.What factors influencestatistical significance?

D.Whattypes oftests are used to determine if a value is statistically significant?

E.What confidence levels(or intervals)areused?

F.How will statisticalsignificance be identified

III.Control Comparison Charts ………………………………………………………………………4

A.What are control and comparison charts?

B.Whatdocontrol chartsmeasure?

C.Whattwo types of errorscan be made witha controlchart?

D.How doyou determine ifthereisaspecial cause variation?

E.What are thedifferent types of control charts and when should they be used?

F.Whatdocomparisonchartsmeasure?

G.What data distributionsare assumed forthe different types of measures being evaluated?

H.What is thecomparative norm used forthe control charts?

I.How is special cause variation displayed using comparison charts?

IV. Definitionof Related Terms …………………………………………………………………………7

V.Appendix:Formulas Used for Statistical Significance Testing………………………………8

Comparion'sweb-basedsoftwareincludesacomprehensivestatisticalpackagewithinthesystem.Thispackageincludesstatisticalsignificancetestingat95%and75%confidenceintervalsforcontinuousvariables(e.g.,charges,costs,andlengthofstay)andrate-basedmeasures(e.g.,mortality,complications,andreadmissions).A99%confidenceintervalisusedforNationalHospitalQualityMeasuresasspecifiedbyTheJointCommission.Thepackagealsoincludestheabilitytoproduce presentation-ready control and comparison charts.

Thistrainingmanualisintheformoffrequentlyaskedquestionsandshouldcovermostofthequestionspertaining to the applicationandinterpretationofstatisticalsignificance testing. If youshouldhave additional questionsplease feelfreeto contact our client supportdepartment at 1-800-711-8363.

A.Whatisstatisticalsignificancetesting?

Statisticalsignificancetestingreliesonamathematicalformulafordeterminingwhetherthedifferencebetweenthemeanofonevaluewhencomparedtothemeanofanotherlikevariablecanbeattributedsimplytorandomor“commoncauses”associatedwithnormalvariationinthedataorwhetherthevariation(ordifference)isduetoassignableor“special”causesnotexplainedbynormalvariationinthedata.Forinstance,onemightwanttodetermineifthedifferenceinahospital’slengthofstayinaparticularDRGisstatisticallysignificantwhencomparedtoamarketnorm(orpeergroup)forthesameDRGinordertodirect and prioritize process improvement initiatives.

B.How should statistical significance be interpreted?

Whenavalue,suchas ahospital’smortalityrate,iscomparedtoa nationalnormandfound tobestatisticallysignificantataconfidencelevelof95%,onecanconcludethereisonlya5%probabilitythatthevariationisduetorandomchance,orsaiddifferently,onecanbe95%confidentthatthevariationisduetospecialcauses.Statisticalsignificanceindicatestherearediscoverablereasonsthatexplainwhythevaluesaredifferentandinsodoingappropriatelyidentifiesopportunitiesforimprovement(ifahighermortalityrate)orvalidatessuperiorperformance(ifa lower mortality rate).

C.Whatfactorsinfluencestatisticalsignificance?

Therearefour(4)importantfactorsthataffectthelikelihoodofstatisticalsignificancebeingidentified:

1.Theextentofthedifferencebetweenthemeanofthetwovaluesbeingassessed(i.e.,thegreaterthedifferencebetweenthemeanoftwovaluesthegreaterthechanceofstatistical significance existing).

2.Thevolumeassociatedwiththevaluebeingassessed(i.e.,themorevolumeassociatedwithavalue,thelessvariationisrequiredforstatisticalsignificancetoexist).

3.Thedegreetowhichvaluesinadistributiondeviatefromthemeanofthedistributionasmeasuredbythe“standarddeviation”(i.e.,asthestandarddeviationofthevaluebecomessmallerthereisanincreasedlikelihoodthatthevaluewillbestatisticallysignificant).

4.Theconfidencelevel(orinterval)usedtodeterminestatisticalsignificance(i.e.,thelowertheconfidencelevel,thegreatertheprobabilitythatthevaluewillbestatisticallysignificant).

Inshort,thereisanincreasedlikelihoodthatthedifferencebetweenthemeansoftwovalueswillbestatisticallysignificantwhen:(a)thedifferencebetweenthemeansbecomeslarger,(b)thedispersion(standarddeviation)ofthedistributionforthevaluebecomessmaller,(c)thevolume associated with thevalue increases, or (d) the confidencelevel is lowered.

D.What types of tests are used to determine if a value is statisticallysignificant?

Therearetwo(2)commonteststhatcanbeusedtodeterminestatisticalsignificance:thez-test(orz-value)andthet-test(ort-score).Thet-testisappropriatetousewhensmallsamplesizesareassociatedwiththevalue(note:asthesamplesizeincreasesthet-testapproximates the z-test in which case the z-test is used). Note:

Forrate-based measures the z-test (or z-value)is used.

For continuousvariables both tests areused depending on the sample size:

-Ifthe number of cases is≥ 25the “z-test”isused

-Ifthe number of cases is25the “t-test”isused.

E.What confidence levels (orintervals)areused?

Hospital Core ComplianceAnalysis

A99%confidencelevelisusedbasedonTheJointCommissionsurveyspecificationswhicharedesignedtoredflagamanageablenumberofhospitalsforperformanceimprovementactivities.Thehighertheconfidencelevelthelesslikelythevariationwillbeduetochancewhich reduces the probability thatahospitalwillbe red flagged.

All Other Analysis Modules

Botha95%and75%confidencelevelisusedtodeterminestatisticalsignificance.A95%confidencelevelisthemostcommonintervalusedinpublishedresearchandisconsideredthestandardforassessingstatisticalsignificance.A75%confidencelevelisusedtoidentifyadditionalimprovementopportunitiesbeyondthoseidentifiedata95%confidencelevel.Opportunitiesfoundata75%confidencelevelarelesslikelytobeduetospecial(orassignable)causes.Withlimitedtimeandresources,theuseoftwoconfidencelevelsallowsfortheprioritizationofimprovementobjectives(thosefoundata95%confidencelevelwouldbe a higher priority than those at 75%).

F.HowwillstatisticalsignificancebeidentifiedwithinComparion’ssoftware?

Comparion’ssoftware uses the following statistical significance designations:

↑** High at a 95% confidence level

↓** Low at a 95% confidence level

↑*High at a75% confidence level

↓*Low at a 75% confidence level Note:

Statisticalsignificanceisidentifiedashighorlowusingarrowsandcolorcoding,where“redup”arrowsindicate“high”and“bluedown”arrowsindicate“low”.Theconfidencelevelisidentified using a “double asterisk” (**)for “95%” anda“single asterisk” (*) for “75%.”

A.WhatareControlandComparisonCharts?

ControlchartsmeasuretheHCO’sobservedoutcomesovertimeinordertoanalyzethetypeofvariation“within”theHCO--commoncauseorspecialcause.Thecontrolchartenablestheorganizationtodetermineiftheprocessisstable,or“undercontrol”.Oncethestabilityoftheprocessisconfirmedthencomparisonchartscanbe usedtocomparethe HCO’sperformancetootherHCOs.Unlikecontrolcharts,comparisonchartsprovideastaticcomparisontoanexternalnormforapointintimeforthepurposeofevaluatingperformanceimprovementopportunities.

B.WhatdoControlChartsmeasure?

Controlchartsevaluatethestabilityofagivenprocessbydeterminingwhetherprocessisunderstatisticalcontrolornot.Statisticalcontrolisachievedwhenonlyrandomvariationexistsinthedata(referredtoascommoncausevariation).However,commoncausevariationdoesnotimplythattheprocessisfunctioningatadesirablelevelornot,itonlydescribesthenatureofthevariation,namelythatitisstableandpredictablewithingivenlimits-itisinstatisticalcontrol.Notethateveniftheprocessisincontrolonecanintroduceaprogramforimprovementinordertoachieveabetterresultinoutcomes(e.g.,theimplementationofevidenced-basedguidelinesorclinicalpathways).Specialcausevariationindicatesthattheprocessisoutofstatisticalcontrolandunstable.Itisnolongerpredictablewithinlimits.Thecontrolchartcandetectaspecialcausevariationbyidentifyingpointsoutsidetheupperandlower controllimits.

Acontrolchartisalinegraphwithfourlines:theuppercontrollimit(UCL),lowercontrollimit(LCL),acenterlinerepresentingtheoverallprocessaverageormean,andalinerepresentingtheprocessovertime.Controllimitsdescribethenaturalvariabilityofaprocessovertimeandare usually set at three standard deviations (or“three sigma”).

C.Whattwotypesoferrorscanbemadewith a controlchart?

Whenusinga controlcharttoidentifyspecialcausevariationtherearetwotypesoferrorsthatcan be made:

TypeI: concludes there is a specialcause whenitisnot present
TypeII: concludes there isno specialcause, when infact it is present

Usingthreesigmacontrollimitsoffersthebestbalanceforreducingtheprobabilityofmakingeither type of error. Note that “sigma” isused synonymously with“standard deviation”.

D.Howisspecialcausevariationdetermined?

There are three (3) tests that determine if special cause variation exists. They areas follows:

1.One data point above or below theUCLor LCL

2.Arun of eight(8) consecutive data points on one sideofthecenterline

3.Atrend of six(6)consecutive data points steadily increasing or decreasing

Note:

Specialcausevariationcanexisteveniftherearenopointsaboveorbelowthecontrollimits.
Whenoneormoremissingdatapointsexist,onlyTest1aboveshouldbeused.

E.Whatarethedifferenttypesofcontrolchartsandwhenshouldtheybeused?

Thereareseveraldifferenttypesofcontrolchartsthatcanbeuseddependingonthekindofdatathatisevaluated.Comparion’ssoftwarewillautomaticallygeneratetheappropriatechart.Refertothesectionbelowforasummaryofthetypeofcontrolchartsusedforeachtypeofmeasure.

Hospital Core Compliance Analysis Module

Type ofMeasure
Proportion orRate / Examples
mortality, complications,or / Type ofChart
p-chart
# of patients receiving smokingcessation advice
Continuous Variable / ALOS, charge, or time in minutesto PCI / M-chart
Continuous Variable(Lowsamplesize) / Usedwhen average number of casesis less than 10 / MR-chart
AllOtherAnalysis Modules
Type ofMeasure / Examples / Type ofChart
Proportion orRate / mortality, complications,or
# of patients receiving smokingcessation advice / p-chart

Continuous VariableALOS, charge, or time in minutesX-barS Chart

to PCI

Continuous VariableUsedwhen average number of casesXmR Chart(lowsample size) is less than 10

Notethatcontrolchartscanalsobeusedforskeweddata,becausetheyare“averagesofsamples”andtheytendtobenormallydistributedregardlessoftheshapeofthedistributionfrom which the sampleswere drawn.

UsingTheJointCommissioncriteria,intheHospitalCoreComplianceAnalysis,controlchartsarecreatedonlywhenthereareatleast12datapointsforagivenmeasure.The12pointscanincludemissingdatapoints.Thechartswilldepictacenterline(overallprocessaverage),

+3and-3sigmalines(UCLandLCL)andtheHCO’sobservedrate.Becausethesamplesizeformonthlydatapointsvaries,thecontrollimitswillbecalculatedmonthlyoftenresultinginuneven lines.

F.WhatdoComparisonChartsMeasure?

WhilecontrolchartsevaluatethestabilityofanHCO’sprocessbycomparingitsowndatatoitselfovertime,comparisonchartscompareanHCO’sdatatoanexternalnormderivedfromtheaverageperformanceofmultipleHCOs.ComparisonchartsrecognizethatanHCO’sprocessmaybeinstatisticalcontrolbutstillbeperformingpoorlywhencomparedtotheoutcomesofotherHCOs(i.e.,itispossibletohaveapoorprocessthatisstatisticallystable).ComparisonchartsalsoallowHCOstodocumentwheretheyareperformingbetter(orworse)thantheirpeers.Typically,acomparisonchartanalysisisgeneratedafterevaluatinganHCO’scontrolchartbecauseitisimportanttoknowthattheexistingprocessisstablebeforeembarking onadditional process improvement activities.

G.Whatdatadistributionsareassumedforthedifferenttypesofmeasuresbeingevaluated?

For rate-based measures,such asmortality rates,a binomial distribution is assumed;
For continuousmeasures,such ascharges, a normaldistribution is assumed.

H.Whatisthecomparativenorm usedforcomparisoncharts?

The comparative norms used arethe:

Predictedorexpectedrate(weightedmedianpredictedrate)ifthemeasureisrisk-adjusted;
Comparison group mean (weighted group mean) if the measure isnot risk-adjusted;
Weighted median rate if the measure iscontinuous and not risk-adjusted.

I.Howisspecialcausevariationdisplayedusingcomparisoncharts?

Specialcause variation isdisplayed in comparison charts by usinga:

P-value for tabular ornumerical display, or an
Expected range for graphical chart display

Theexpectedrange,alsoknownastheacceptanceinterval,isanintervalhavingupperandlowerlimitsthatrepresentsthesetofvaluesforwhichthenullhypothesisisaccepted.A

comparisonchartconsistsofactual(orobserved)rates,expectedrates,andexpectedrangesforagiventimeperiod.Theconfidencelimits(orlevels)describethedegreeofcertaintythatagivenpointisdifferentfrom theaverage.ForNHQMcore measures,a 99%confidencelevel isapplied. For all other measures a 95% level is applied.

Averageormeanisthemostcommonexpressionofthecenteringofadistribution.Itissignifiedby

xand iscalculated by totalingthe observed valuesanddividing by the numberof observations.

CommonCauseisasourceofvariationthatisalwayspresent;partoftherandomvariationinherentintheprocessitself.Itsorigincanusuallybetracedtoanelementofthesystemwhichonlymanagement cancorrect.

ControlChartisagraphicrepresentationofacharacteristicofaprocess,showingplottedvaluesofsome statisticgatheredforthatcharacteristic,andoneortwocontrollimits.Ithastwobasicuses:asajudgmenttodetermineifaprocessisincontrol,andasanaidinachievingandmaintainingstatisticalprocesscontrol.

ControlLimitisaline(orlines)onacontrolchartusedasabasisforjudgingthesignificanceofthevariationfromsubgrouptosubgroup.Variationbeyondacontrollimitisevidencethatspecialcausesareaffectingtheprocess.Controllimitsarecalculatedfromprocessdataandarenottobeconfusedwith engineering specifications.

SigmaistheGreekletterusedtodesignatetheestimatedstandarddeviation.In Comparion’ssoftware, three (3)sigma is used for all control charts (balances therisk of TypeI and II errors).

SpecialCauseisasourceofvariationthatisintermittent,unpredictable,orunstable;sometimesreferredto as an “assignable cause.” It is signaled by a point beyond the control limits.

StandardDeviationisastandardunitofmeasurefordescribingthedegreetowhichvaluesinadistributiondeviatefromthemean.Thestandarddeviationandvarianceareconsideredthetwomostimportantmeasuresofvariability(ordispersion)sincetheyarecommonlyusedincalculationsfordeterminingstatisticalinferences.Forinstance,inanormaldistribution68%ofthevaluesinadistributionalwaysfallwithinone(1)standarddeviationofthemean,95%withintwo(2)standarddeviations,and99%withinthree(3)standarddeviations.Thestandarddeviationiscalculatedbytakingthesquarerootofthe variance(referto Appendix for formulas).

StatisticalControlistheconditiondescribingaprocessfromwhichallspecialcauseshavebeenremoved,evidencedonacontrolchartbytheabsenceofpointsbeyondthecontrollimitsandbytheabsence of non-random patternsor trendswithin thecontrol limits.

Trendsarethepatternsinarunchartorcontrolchartthatfeaturethecontinuedriseorfallofaseriesof points. Atrend consistsof six (6) consecutive datapoints steadily increasing or decreasing

ThestatisticalpackageisdesignedtofollowcloselytheformulasandapproachusedbyTheJointCommissionfor the NationalHospitalQualityMeasures programin order to be consistent with industrystandards.

I.ProportionorRateMeasures(mortality,complications, and readmissions)

A.Statistical Significance

TodeterminestatisticalsignificanceweutilizeeithertheZ-statisticorthet-statisticandthencompare to valueson either theztableor thettable.

Symbols:

n= number of cases foraperiod

xo=mean of observed values fora periodso=standarddeviation of observed valuesxe=mean of risk adjustedvalue

1.Calculatingthez statistic

a.n>=25

z

Thisvalueisassumedtofollowanormaldistributionwhenthesamplesizeis

25.Thezvalueiscomparedtothevaluesontheztable.Ifitisoutsidethespecificconfidence interval itis statistically significant.

2.Calculatingthe t statistic

t

Thet-statisticisassumedtofollowat-distribution.Unlikeanormaldistribution,thet-distributiondependsonsamplesize.Thet-valueiscomparedtothevaluesonthettable.Ifitisoutsidethespecificconfidenceinterval it is statisticallysignificant.

B.ControlCharts

p-Chart – all proportion orrate measuresare analyzed using the p-chart.

ni= numberof denominator cases for the periodxi= numberofnumeratorcases fortheperiod pi= observedratefora month

CenterLineCalculation

pxi x1x2 ...xm

ni

n1n2 ...nm

wherem= number of datapoints

Upper and LowerControl Limits for eachperiod

p3

C.ComparisonCharts

Symbols:

n = numberof denominatorcases for theperiod

po= observed ratefortheperiod

pe=risk adjusted rate

1.CalculatetheZstatisticorZscore.Thisvaluefollowsanormaldistributionwhenthesamplesizeisnotverysmall.Anyvaluelessthan-2.576orgreaterthan2.576signalsstatistically significant difference betweenthetwo rates at 1% significance level.

Z

2.Calculate theUpper andLowerControl Limits of theConfidence Interval

ZZ2

Uo 

(po

1

2 )

2n

Z1

1

2

4n2



1

Po(1Po)

,whereZ

2.576

12

ZZ2

Lo 

(po

1

2 )

2n

Z1

1

2

4n2



1

Po(1Po)

,whereZ

2.576

12

Statisticalsignificancecanalsobedeterminedbycomparingtheexpectedratewiththeconfidence interval.

3.Calculate theupper limitand lower limit of the expected range

Ue= pe+(po-Lo)(If Ue>1 thenUe=1)Le= pe+(po-Uo)(If Le<0thenLe=0)

4.Sample Size Warning

(a)When the sample size (n)fora proportion measureislessthan 30 or

(b)WhenthenumberofHCOsinthecomparisongroupislessthan10,thenanappropriatewarningmessageisissuedtoindicatethatthedatamaynotbevalidduetoasmallsamplesizeandthatsuch datashould be interpreted and used with caution.

II.Continuous Variable Measures(charge,cost,LOS)

A.ControlCharts

MandMRCharts

=process meanor theexpected valueof thepopulation ofmedianobservedvalues
= process standard deviation or standard deviation of the population of medianobserved values
ni= numberofcases for the ithperiod
Mi= medianobserved value intheithperiod

M=the overallaverage of the period median observed values
Ri= range ofmeasurements intheithperiod
N= numberof periodsbeing evaluated
VAR(M((ni 1)/2)))=standarderrorofthemedianofnindependent,normallydistributedvariableswithunit standarddeviation
d2(ni)=theexpectedvalueoftherangeofnindependentnormallydistributedvariableswith unit standard deviation
d3(ni)=thestandarderroroftherangeofnindependentobservationsfromanormalpopulation with unit standard deviation

1.Calculate theCenter Line

a.M-Chart

Mn1M1...nNMN/n1...nN

b.MR-Chart

MedianChart:

Mn1M1...nNMN/n1...nN

Range Chart:

Ri d2(ni)

2.Calculate thecontrollimits

a.M-Chart

UCL/LCL=(M3

b.MR-Chart

MedianChart:UCL/LCL=(M3

VAR(M((ni1)/2)))

Range Chart:

LCL=maxd2(ni)3d3(ni),0

UCL=d2(ni)3d3(ni)

Note: months where the number of cases =1 are excluded from thecontrol chart

X-bar S Charts and XmRCharts

ni= numberofcases for the period

xi= mean of observed values for the period

si= standarddeviation of observed values forthe period

1.Calculate the Center Line

a.X-barchart

xni xi

ni

b.S-Chart – calculate theminimum variance linear unbiased estimate

hi

si

cc2

si c4

hi

,wherehi 

1c2

c4 isaconstantthatdependsonthesamplesize. Asthesamplesizeincreases,c4

approachesto 1.The exact formula forc4is:

(ni)

c 2

n 

( i1)

2.Calculate thecontrollimits

a.X-barChart

x3sc4ni

b.S-Chart

s(13

1c2)

Iftheaveragesamplesizeislessthan10,anXmRchartistobeusedinsteadofanX-barandS-chart,assumingtheobservedmeanvalueasasingleobservationfortheperiod.

NotethattheX-barSchartisapairedchart,thatis,theX-barchartwillrevealwhetherthereisaspecialcauseacrosstimeperiods,whiletheSchartwillrevealwhethertherearespecialcauseswithintimeperiods. FirstinterprettheSchartandifit is in control,then evaluate theX-barchart.

B.ComparisonCharts

Symbols:

n= numberofcases for aperiod

xo=median of observed valuesfora periodso=standarddeviation of observed valuesxe= median of riskadjusted value

1.Calculate thez statistic

a.n>=30

z

Thisvalueisassumedtofollowanormaldistributionwhenthesamplesizeisnotverysmall.Anyvaluelessthan-2.576orgreaterthan2.576signalsstatisticallysignificantdifference betweenthetworates at 1% significance level.

b.n30

t

Thet-statisticisassumedtofollowatdistribution.Unlikeanormaldistribution,thetdistributiondependsonsamplesize.Forexample,ifthesamplesizeis15,anyvaluelessthan-2.977orgreaterthan2.977signalsstatisticallysignificantdifferencebetween thetwo rates at 1% significance level.

2.Calculate an expected rangebased ontheteststatisticExpectedupper limit:Ue xe (xo Lo)

Expectedlower limit:Le xe (xo Uo)

Where;

Uo xo

Z 

1

and

Lo xo

Z 

1

ifn30

Or;

Uo xo t 

and

Lo xo t 

ifn30

12n

Iftheobservedvalueiswithintheexpectedrange,itisnotastatisticaloutlierat1%significancelevel.Iftheobservedvalueisoutsidetheexpectedrange,theobservedrateisastatistical outlier.

Note:

Whenthesamplesize(n)isverysmall,anappropriatewarningmessageisissuedtoindicatethatthedatamaynotbevalidduetoasmallsamplesizeandthatsuchdatashouldbeinterpretedandusedwithcaution.Awarningmessageisissued(a)whenthesamplesizeislessthan10;or(b)whenthenumberofHCOsinthecomparisongroupislessthan10fortherisk adjustedmeasureswhose riskadjusted data arenot available.