131 Appendix D - Critical Indicators of Project Performance

PublishedonOfficeoftheChiefInformationOfficer()

Home Policies131-ManagingInformationTechnologyProjects131AppendixD-CriticalIndicatorsofProjectPerformance

AppendixD:CriticalIndicatorsofProjectPerformanceEarlyWarningSigns

Introduction

ExperiencemanagingandoverseeingmanyInformationTechnology(IT)projectsinWashingtonStateandelsewherehastaughtusthatthereareseveralkeysuccessfactorsassociatedwiththeseITprojects.Experiencehasalsotaughtusthatthereisasetofwarningsignsassociatedwitheachofthesesuccessfactors–signsthatasuccessfactor,andhencetheoverallproject,maybeinjeopardy.Inthisdocument,wedescribethesuccessfactorsandtheirassociatedwarningsignsasaguidetomonitoringprojectperformance.

Thisdocumentisintendedforusebyexecutiveleadershipandprojectmanagementtohelpassessthelikelihoodofsuccessofaprojectwhileitisinprogress.

ExecutiveManagementSupport

TheExecutiveSponsormusthaveaglobalviewoftheproject,settheagenda,arrangethefunding,articulatetheproject’soverallobjectives,beanardentsupporter,beresponsive,andfinally,beaccountablefortheproject’ssuccess.Warningsignsinclude:

ThereisalackofstrongExecutiveSponsorship.Thisisevidencedbyanyorallofthefollowing:

Inabilitytocommittoandclearlyarticulatetheproject’sbusinessobjectives.

Nocommitmenttoregularattendanceatexecutivesteeringcommitteemeetings.Incompleteknowledgeoftheproject’scriticalissues.

Ineffectiveornoresolutionofissuesbroughtbeforetheexecutivesteeringcommittee.Notknowingthestatusoftheprojectatahighlevel.

Multipleexecutivesponsorswhoarenotallengagedtotheextentneeded.

Agencyexecutivesdonothaveexperiencemanagingthestaffneededforaprojectofthissizeandcomplexity.Thisisevidencedbyanyorallofthefollowing:

Agencymanagersnotunderstandinghowtheprojectmethodologyisimplemented.Lackofstaffawarenessofprojectimportancetotheorganization.

Inabilitytocommittoandclearlyarticulatetheproject’sbusinessobjectives.

Lackofaneffectiveprojectcommunicationsplan,oronethatisnottrackedandupdatedasneeded.

Theprojectdirectionisnotalignedwiththestrategicbusinessobjectives.Thisisevidencedby:

Anexcessivenumberofprocessgapsidentifiedduringdesign.

TheExecutiveSponsorisunabletodelivertheprojectasoriginallyproposed.Thisisevidencedbyanyorallofthefollowing:

Significantchangesinscope,schedule,orbudget.

Scope,schedule,orbudgetchangesarenotcommunicatedinaclearandtimelymannertoprojectstakeholders.Lackofcoveragefornormalbusinessoperations.

Thesteeringcommitteeisineffective.Thisisevidencedby:

Majorstakeholdersnotrepresentedonthesteeringcommittee.

Majorstakeholdersarenotgiventheopportunitytoprovidemeaningfulinputonissuesthataffectthem.

UserInvolvement

Primaryusersmustbeinvolvedinandsupporttheproject.TheymusthavegoodcommunicationskillsthatallowthemtoclearlyexplainbusinessprocessesindetailtotheITorganization.Primaryusersshouldalsobetrainedtofollowprojectmanagementprotocols.Finally,usersmustberealists-andawareofthelimitationsoftheprojects.Warningsignsinclude:

Stakeholdersdonothavetheinformationtheyneedtoprovidemeaningfulinputtotheproject.Thisisevidencedbyanyorallofthefollowing:

Communicationplannotbeingfollowed.

Usersdonothaveamethodtoprovidefeedbackonaregularbasis.

Notallrequiredinteractionsareincludedinthecommunicationplan;timingortargetaudienceisinsufficient.

Projecthasnotscheduledsufficienttimeforuserstounderstandandbecomecomfortableandproficientwiththenewsystem.Thisisevidencedbyanyofallofthefollowing:

Requirementsvalidationsessionsdonotreviewallprocesses.Trainingdoesnotcoverportionsofthesystemtheusersneed.

Usersdonotunderstandtheirroleinthetestingbecausetheyhavenotreceivedtrainingonpurposestructureofusecasesortestscenarios.

TestersdonothavetimetocompleteUATortoretestdefectsthathavebeenresolved.

Therearenoeffectiveforums(suchasFAQsorbrownbags)foruserstoaskquestionsofthe“experts”.Thereisnoeffectiveorganizationalchangemanagementplan.

Projectunderestimatedthetimeneededfortrainingpreparationandactualtraining.

ExperiencedProjectManager

Projectmanagersmustpossesstechnologyandbusinessknowledge,judgment,negotiation,goodcommunicationandorganization.Thefocusisonsofterskills,suchasdiplomacyandtimemanagement.Warningsignsinclude:

Theprojectmanagerdoesnotpossesstheappropriateskillstomanageaprojectofthissizeandcomplexity.Thisisevidencedbyanyorallofthefollowing:

Lackofformalprojectmanagementtraining.

Nopreviousexperiencewithprojectsofsimilarsizeandcomplexity.Doesnotunderstandtheproposedmethodology.

Communicationstyleisnotalignedwiththeagency’sculture.Doesnotinspireconfidenceinmanagementorstaff.

Cannotarticulatethebusinessobjectives.Inabilitytodelegateeffectively.

Focusedonstatusreportinginsteadofmanagingtheproject.Tooinvolvedintheprojectdetails.

Doesnotunderstandskillsetandskilllimitationsofprojectstaff.Doesnotunderstandneedfortransparency.

Doesnotknowwhatisimportanttocommunicate.

ClearBusinessObjectives

Theprojectobjectivesmustbeclearlydefinedandunderstoodthroughouttheorganization.Projectsmustbemeasuredagainsttheseobjectivesregularlytoprovideanopportunityforearlyrecognitionandcorrectionofproblems,justificationforresourcesandfunding,andpreventiveplanningonfutureprojects.Warningsignsinclude:

Theprojectdoesnothaveclearandagreeduponrequirements.Thisisevidencedbyanyorallofthefollowing:

Inadequateornofeasibilitystudy.

Businessstakeholderinvolvementwasinadequateortoolateintheprocess.Noevidenceofbusinesssponsorapprovalofrequirements.

Toomuchfocusontechnology.

Businessprocesseshavenotbeenidentifiedordocumented.Thisisevidencedbyanyorallofthefollowing:

Stakeholderparticipationisinadequateforidentificationofbusinessprocesses.

Projectoveranalyzesthebusinessrequirementsandprocesses;resultisinabilitytofinalizerequirements.Theagencyseestechnologyasthedriverfororganizationalorprocesschange.

Theprojectdirectionisnotalignedwiththestrategicbusinessobjectives.Thisisevidencedby:

Anexcessivenumberofprocessgapsidentifiedduringdesign.

Requirementsarenottraceablefromtheinitiationphasethroughtheexecutionphase,whichincludesuseracceptancetesting.

Projectformallyacceptsdeliverablebeforetestingiscompleted.

MinimizedScope

Scopemustberealisticallyabletobeaccomplishedwithintheidentifiedprojectdurationandmeasuredregularlytoeliminatescopecreep.Warningsignsinclude:

Theprojectdoesnotincludeadequatecontingencytomaintaintheoriginalscopeorschedule.Thisisevidencedbyanyorallofthefollowing:

Requiredstaffresourcesbeingremovedfromtheprojecttoaddressotheragencyneeds.Projectresourceplandoesnotcontainallowanceforbackfillorothernon-assignableduties.Aninordinateamountofcontractamendmentsareprocessed.

Earlydepletionofthechangecontrolbudget.

Projectisgivenfixedscope,schedule,andbudgetresultinginnoflexibility.

ResponsiveBusinessRequirementsProcess

Requirementsmanagementistheprocessofidentifying,documenting,communicating,trackingandmanagingprojectrequirements,aswellaschangestothoserequirements.AgileResponsiverequirementsprocessistheabilitytodorequirementsmanagementquicklyandwithoutmajorconflicts.Thisisanongoingprocessandmuststayinlockstepwiththedevelopmentprocess.Warningsignsinclude:

Theprojectisoperatingunderfaultyassumptionsthatcriticalaspectsofthebusinessenvironmentarestable.Thisisevidencedbyanyorallofthefollowing:

Changecontrolprocessesarenotbeingfollowed.Changecontrolboardisnotinplace.

Changecontrolparticipantsdonotunderstandtheirroles.Decisionsfrequentlyrevisited.

Thereisnolegacysystemfreezeplan.

Theprojectisfailingtoresolveissuesinatimelymanner.Thisisevidencedbyanyorallofthefollowing:

Projectteamdoesnotescalatequickly,spendstoomuchtimetryingtosolvetheproblem.Lackofattentiontoissueresolutiondatesthatareoverdue.

Executivesponsornotavailabletoresolveissuesinatimelymanner.Decisionsfrequentlyrevisited.

Standard Infrastructure

Establishastandardtechnologyinfrastructurethatincludesoperationalandorganizationalprotocols.Thisinfrastructuremustbecommonlyunderstoodandregularlyassessed.Warningsignsinclude:

Projectdidnotincorporateallapplicablestatestandardsintothesystemdesign.Thisisevidencedby:

Failuretoengageappropriatestateserviceprovidersearlyintheprocessforitemssuchas:

·Interfaces

·Security

·Userauthentication

·Encryption

·Network

·AutomatedClearingHouse(ACH)

·Creditcardprocessing

Theprojecttestingisineffective.Thisisevidencedbyanyorallofthefollowing:

Projectdoesnottestinanenvironmentthatapproximatestheproductionenvironment.Convertedproductiondataisnotusedtotest.

Projectteamreliesonmanufacturerspecificationsratherthanevaluatinghardwareperformanceoftheexistingenvironment.

Theprojectdidnotconsiderallcapacityrequirementsduringplanning.Thisisevidencedbyanyorallofthefollowing:

Projectdidnotrequestapplicationandsecuritydesignreviews.ProjectdidnotcoordinatewiththeIntegrationCompetencyCenter.

Serverhardwareandsoftwarefunctionalityandspecificationswerenotverifiedbeforeformalacceptance.

Projectmakesuseofunfamiliartechnologyanddoesnotincludeadequatecontingencyforresolutionofunusualandunforeseentechnologyissues.

FormalMethodology

Followingaformalmethodologyprovidesarealisticpictureoftheprojectandtheresourcecommitment.Certainstepsandproceduresarereproducibleandreusablemaximizingproject-wideconsistency.Warningsignsinclude:

Theprojectdoesnotusetheformalmethodologyproposedorplanned,orthereisnomethodology.Thisisevidencedbyanyorallofthefollowing:

Requiredstaffresourcesbeingremovedfromtheprojecttoaddressotheragencyneeds.Projectteammembersarenottrainedandsupportedinatimelymanneronthemethodology.Artifactsarenotreadilyavailabletoallmanagementandstaff.

Artifactsarenotupdatedinatimelymanner.

Projectstatusisnotcommunicatedcorrectlytoalllevelsoftheprojectteam.

Theprojectisnotorganizedinawaythatprovidesinterimvaluetotheagencyorthestate.Estimatesarenoteffectivelyvalidated.

Theprojectmanagerisnotabletomeasureandreportstatusaccurately.

Theagencydoesnotunderstandtheramificationsofselectingonemethodologyversusanother.Necessarymanagementplansarenotinplace.

Managementplansarenotrevisitedonarecurringbasis.

Themethodologychangeswithoutadequateforethought,planning,andjustification.Project’sapproachestoaddressQAfindingsareinadequate.

Processisinadequateforvalidatingprojectreadinesstoproceedpastcurrentphase.

ReliableEstimates

Berealisticintheestimatesforresources,time,budget,andscope.Warningsignsinclude:

Theestimatesforscheduleandbudgetareunreliable.Thisisevidencedbyanyorallofthefollowing:

Staffingandbudgetarenotadjustedwhenthemethodologychanges.

Theagencyhasahistoryofsignificantlyunderestimatingtheresourcesneededtocompleteaprojectwithinscope,schedule,andbudget.

ThevendorhasahistoryofmultipleengagementsinWashingtonStatewheretheysignificantlyunderestimatedtheresourcesneededtocompleteaprojectwithinscope,schedule,andbudget.

Aproposedchangetoprojectscheduleorbudgetdoesnotreflectadditionalcontingencythatshouldbeneededduetothelackofaccuracyofpastestimates.

Theprojectscope,schedule,orbudgethasnotchangedalthoughasignificantinterimmilestonewasmissed.

SkilledStaff

Properlyidentifytherequiredcompetencies,therequiredlevelofexperienceandexpertiseforeachidentifiedskill,thenumberofresourcesneededwithinthegivenskill,andwhenthesewillbeneeded.Ensuretheseresourcesareavailable

andassignedtotheproject.Softskillsareequallyimportantwhenidentifyingcompetencies.Warningsignsinclude:

Theprojectstaffdoesnotpossesstheskillsnecessaryfortheproject,ortheavailabilityofstaffisinadequate.Thisisevidencedbyanyorallofthefollowing:

Theprojectmanagerdoesnothavethenecessaryskillstomanageaprojectofthissizeandcomplexity.

Projectstaffarenoteffectivelysupervised.Thisisevidencedbyanyorallofthefollowing:

Reportingrelationshipsareunclear.

Staffreporttomultiplesupervisorsormanagers(matrixrelationships).

Projectmanagersandteamleadshavesupervisorroles;takestimeawayfromPMactivities.

Knowledgetransferisinadequatebetweenagencyandcontractor.Thisisevidencedbyanyorallofthefollowing:

Vendordoesnotintegratestatestaffintoalldevelopmentandconfigurationactivities.TheWBSdoesnotincludeadequatetimeforknowledgetransfer.

Knowledgetransfertasksarelistedattheendoftheprojectschedule.Thereisnomethodforevaluatingtheefficacyofknowledgetransfer.

Vendorexpectsstatestafftobeadequatelytrainedimmediatelyaftercontractexecution.Coreprojectteamhasnotbeenidentifiedbythetimetheacquisitiondocumentisreleased.

Vendorisunabletoprovideskillprofileortrainingrequirementstotheagencyinadvanceofwhentheskillsetisneeded.

Thestatestaffdoesnothaveadequatetimetoachievethebaseofknowledgenecessarytobeeffectiveontheproject.

ContractNegotiationandVendorManagement

TheStandishGroupdidnotidentifythisasasuccessfactor;however,basedonlessonslearnedfromWashingtonStateprojects,contractnegotiationandmanagementplaysamajorroleinprojectoutcomes. Warningsignsinclude:

Contractnegotiationormanagementofthevendorisinadequateorineffective.Thisisevidencedbyanyorallofthefollowing:

QAnotanactiveparticipateincontractnegotiations.

Agencynegotiatordoesnothavetheexperienceinnegotiatingcontractsofthisscopeorcomplexity.Agencydoesnotunderstandwhichpositionsitiswillingtonegotiateawayandwhy.

Agencydoesnotunderstandwhichtermsarerequired.

SOWdoesnothaveadequate,clearlydefined,measurableindicatorsofcompletion.SOWisnotusedasamanagementinstrumentbythePM.

Rolesandresponsibilitiesarenotadequatelydefined.

Agencyassumesthevendor’sresponsibilitiesinordertomeetamajordeliverable.TheSOWdoesnotincluderesponsibilitytocomplywithstatepoliciesandstandards.

Contractdoesnotreflectthebusinessgoalsoftheorganization;itdealsmorewithprotectingthestateagainsttherisksoflegalaction.

Implementation

TheStandishGroupdidnotidentifythisasasuccessfactor;however,basedonlessonslearnedfromWashingtonStateprojects,implementationplaysamajorroleinprojectoutcomes.Warningsignsinclude:

Projectdoesnothaveawelldefinedtransitiontoimplementtheproduct.Thisisevidencedbyanyorallofthefollowing:

Implementationplandoesnotincludenewormodifiedprocessesforagencyhelpdesktointerfacewithvendorsupport.

Agencyusersdonotknowhowtorequesthelponcesystemisimplemented.

HelpdeskisnotadequatelystaffedforthevolumeorcomplexityofcallsreceivedatGo-Live.

Agencyhasnotidentifiedmaintenanceandoperationsfundingforthecontinuedsupportofthesystem.

SourceURL: