GAO Best Practices for Cost Estimating
Appendix A
GAO Best Practices for Cost Estimating
Table B-1 shows the project team self-assessment and the ICR team assessment of conformance with the GAO best practices for cost estimating.
Table B1. GAO Best Practices for Cost EstimatingStep / Description / Associated Tasks / Project Team
Self-Assessment / DOE-PM Notation
1 / Define estimate’s purpose / 1. Determine estimate's purpose, required level of detail, and overall scope. / Met. Purpose is to inform the DOE CD-1 Cost Range, and support budget planning. Point estimate is intended to be at a preliminary stage of development, with an adequate range applied to provide confidence that the CD-2 TPC will be within or close to the proposed range.
2. Determine who will receive the estimate. / Met. Estimate is used to inform CD-1 request to the PME.
2 / Develop estimating plan / 1. Determine the cost estimating team and develop its master schedule. / Met. AssocPM for Schedule responsible for master schedule, appropriate L3’s and SME’s for input.
2. Determine who will do the cost estimate; outline the cost estimating approach. / Met. L3 managers are responsible for developing BOEs.
3. Develop the estimate timeline. / Met. The estimate timeline was a driver for scheduling CD-1.
3 / Define program characteristics / 1. Identify the program's purpose and its system and performance characteristics and all system configurations. / Met. CDR, FRS, TRS documents concepts, requirements, configurations, and specifications.
2. Any technology implications. / Met. New technologies are identified and thoroughly described in the CDR.
3. Program acquisition schedule and acquisition strategy. / Met. An Acquisition Strategy was prepared.
4. Its relationship to other existing systems, including predecessor or similar legacy systems. / Met. Performance enhancements with respect to existing Linac at FNAL.
5. Support (manpower, training, etc.) and security needs and risk items. / Met. Addressed in Assumptions Document (resources), PSVAR (security), and PPEP.
6. System quantities for development, test, and production. / Met. Defined by the KPP’s and detailed in the CDR.
7. Deployment and maintenance plans. / Appropriately met for this stage of the project. Preliminary estimates were developed to support the AoA and Acquisition Strategy. This will be further refined in the future Transition to Operations Plan.
4 / Determine estimating structure / 1. Define a work breakdown structure (WBS) and describe each element in a WBS dictionary (a major automated information system may have only a cost element structure). / Met. WBS and dictionary exists.
2. Choose the best estimating method for each WBS element; Identify potential cross-checks for likely cost and schedule drivers. / Met. BOE documents the estimating process, including information on the source (POs, scaling from analogous systems, vendor quotes). Cross checks against previous project experiences, assumptions, and risk analysis identified cost & schedule drivers.
3. Develop a cost estimating checklist / Met. Independent validation checklist performed for each BOE document.
5 / Identify ground rules and assumptions / 1. Clearly define what the estimate includes and excludes. / Met. Summarized in Assumptions Document, then defined in the WBS dictionary and each individual BOE.
2. Identify global and program-specific assumptions, such as the estimate's base year, including time-phasing and life cycle. / Met. Life cycle cost estimates are included in the Acquisition Strategy, and all other items are addressed in the Assumptions Document and BOEs.
3. Identify program schedule information by phase and program acquisition strategy. / Met. Addressed in the Acquisition Strategy and is informed by a detailed RLS developed by the project.
4. Identify any schedule or budget constraints, inflation assumptions, and travel costs. / Met. The CD-1 estimate was aligned with funding guidance from HEP and other assumptions and constraints were documented in the Assumptions Document.
5. Specify equipment the government is to furnish as well as the use of existing facilities or new modification or development. / Met. CDR describes how PIP-II fits into existing Fermilab Complex.
6. Identify prime contractor and major subcontractors. / Met. The Acquisition Strategy identifies FRA as the prime contractor and discusses major international partners.
7. Determine technology refresh cycles, technology assumptions, and new technology to be developed. / Met. CDR describes the conceptual technical design in detail.
8. Define commonality with legacy systems and assumed heritage savings. / Partially Met. Project documents explain how PIP-II linac replaces the existing linac, but savings are not described.
9. Describe effects of new ways of doing business. / Met. The strategy of using international partners to deliver significant portions of a US particle accelerator is unique and discussed in the Acquisition Strategy and PPEP. The collaboration with India is defined in detail.
6 / Obtain data / 1. Create a data collection plan with emphasis on collecting current and relevant technical, programmatic, cost, and risk data. / Met.FNAL andFRA havea good databaseof relevantcostdata, based onrecentprojects(Nova, Mu2e, LCLS-II).
2. Investigate possible data sources. / Met. Several data sources were used and documented in the BOEs.
3. Collect data and normalize them for cost accounting, inflation, learning, and quantity adjustments. / Met. Documented in the BOEs.
4. Analyze the data for cost drivers, trends, and outliers and compare results against rules of thumb and standard factors derived from historical data. / Met. Unit rates and historical data are predominately used to inform the conceptual cost estimate.
5. Interview data sources and document all pertinent information, including an assessment of data reliability and uncertainty. / Will be done by the ICR and IPR.
6. Store data for future estimates. / Met. Estimate information is kept in the PIP-II Document Database.
7 / Develop point estimate and compare it to an independent cost estimate / 1. Develop the cost model, estimating each WBS element, using the best methodology from the data collected, and including all estimating assumptions. / Met. Apoint estimate wasdeveloped usingthe estimatingassumptions and organized in BOEs.
2. Express costs in constant year dollars. / Met. All estimates use 2016 as the base year.
3. Time-phase the results by spreading costs in the years they are expected to occur, based on the program schedule. / Met. The RLS escalated costs based upon the time phasing of activities. Cost and schedule information in the PPEP reflect this.
4. Sum the WBS elements to develop the overall point estimate. / Met. Results to be presented at the ICR and summarized in the PPEP.
5. Validate the estimate by looking for errors like double counting and omitted costs. / Met. BOEs are aligned with a defined WBS and each was independently reviewed.
6. Compare estimate against the independent cost estimate and examine where and why there are differences. / N/A for this stage of the project. PIP-II CF plans to perform an ICE to support refinement of the cost and schedule estimates prior to CD-2. A comprehensive ICE will be done by DOE prior to CD-2.
7. Perform cross-checks on cost drivers to see if results are similar. / Met. Analogy methods and bases from similar work and projects.
8. Update the model as more data become available or as changes occur and compare results against previous estimates. / Met. Continue to do this through the lifetime of the project.
8 / Conduct sensitivity analysis / 1. Test the sensitivity of cost elements to changes in estimating input values and key assumptions. / Notmet. Asensitivityanalysis has not been completed.
2. Identify effects on the overall estimate of changing the program schedule or quantities. / Notmet. Asensitivityanalysis has not been completed.
3. Determine which assumptions are key cost drivers and which cost elements are affected most by changes. / Met. Keycostdrivers areidentified using a risk analysis.
9 / Conduct risk and uncertainty analysis / 1. Determine and discuss with technical experts the level of cost, schedule, and technical risk associated with each WBS element. / Met. A Risk Workshop was held in March 2017. Risk Management Board exists and meets, and risks are discussed at regular technical meetings.
2. Analyze each risk for its severity and probability. / Met. Each risk is assigned an impact and probability.
3. Develop minimum, most likely, and maximum ranges for each risk element. / Met.Eachrisk event hasacostandschedulerange.
4. Determine type of risk distributions and reason for their use. / Met. The risk register uses triangular probability distribution to increase credibility.
5. Ensure that risks are not correlated. / Met. Reviewed for correlations by the Risk Manager and RMB.
6. Use an acceptable statistical analysis method (e.g., Monte Carlo simulation) to develop a confidence interval around the point estimate. / Met. Developing Monte Carlosimulation to determine risk-based contingencyat a90% CL.
7. Identify the confidence level of the point estimate. / Met. 90%.
8. Identify the amount of contingency funding and add this to the point estimate to determine the risk-adjusted cost estimate. / Met. Results of the Monte Carlo analysis at a 90% CL are used to add cost impacts of risks and costs as a result of schedule impacts (standing army). Contingencyfor costestimatinguncertaintyis addedfor eachmajor costelement accordingto FRAcontingencyrules.
9. Recommend that the project or program office develop a risk management plan to track and mitigate risks. / Met. The project has a risk management plan that augments a lab-wide procedure.
10 / Document the estimate / 1. Document all steps used to develop the estimate so that a cost analyst unfamiliar with the program can recreate it quickly and produce the same result. / Met. The cost estimate was developed using a defined WBS and thoroughly documented BOEs and assumptions.
2. Document the purpose of the estimate, the team that prepared it, and who approved the estimate and on what date. / Met.The estimate purpose (for CD-1 Cost Range)andestimatingteam areidentified.
3. Describe the program, its schedule, and the technical baseline used to create the estimate / Met. Addressed in the CDR.
4. Present the program's time-phased LCC. / Met. The life cycle cost was documented in the Acquisition Strategy.
5. Discuss all ground rules and assumptions / Met. Documented in the Assumptions Document.
6. Include auditable and traceable data sources for each cost element and document for all data sources how the data were normalized. / Met.BOEs document sources and methods for creating the estimate data.
7. Describe in detail the estimating methodology and rationale used to derive each WBS element's cost (prefer more detail over less). / Met. BOE documents contain this information.
8. Describe the results of the risk, uncertainty, and sensitivity analyses and whether any contingency funds were identified. / Partially Met. Developing Monte Carlo model for risk, estimate uncertainty documented in BOE. No sensitivity analysis.
9. Document how the estimate compares to the funding profile. / Met. The estimate is being aligned to a funding profile using the RLS.
10. Track how this estimate compares to any previous estimates. / Partially met. Certain cost estimate variances from CD-0 are documented but not comprehensively for the entire project.
11 / Present estimate to management for approval / 1. Develop a briefing that presents the documented LCCE. / Information will be presented at the ICR, and a briefing is planned to the ESAAB. / Agree with project team assessment.
2. Include an explanation of the technical and programmatic baseline and any uncertainties. / Information will be presented at the ICR, and a briefing is planned to the ESAAB.
3. Compare the estimate to an independent cost estimate (ICE) and explain any differences. / N/A at this stage of the project. PIP-II CF plans to perform an ICE to support refinement of the cost and schedule estimates prior to CD-2. A comprehensive ICE will be done by DOE prior to CD-2.
4. Compare the estimate (LCCE) or independent cost estimate to the budget with enough detail to easily defend it by showing how it is accurate, complete, and high in quality. / Cannot be met at this time. Although operating costs for PIP-II have been estimated, the operating budget for those out-years is not available at this time.
5. Focus in a logical manner on the largest cost elements and cost drivers. / Cost drivers will be presented at the ICR and IPR.
6. Make the content clear and complete so that those who are unfamiliar with it can easily comprehend the competence that underlies the estimate results. / To be done at the ICR and IPR.
7. Make backup slides available for more probing questions. / To be done at the ICR and IPR.
8. Act on and document feedback from management. / Will be done by responding to review recommendations.
9. Request acceptance of the estimate. / Will be accepted with the approval of CD-1.
12 / Update the estimate to reflect actual costs and changes / 1. Update the estimate to reflect changes in technical or program assumptions or keep it current as the program passes through new phases or milestones. / Met. The estimate andschedule are being updated to reflect therevisedinternational contributions and changes in scope. Will complete for CD-2.
2. Replace estimates with EVM EAC and independent estimate at completion (EAC) from the integrated EVM system. / Will be done in practice prior to CD-2 and fully implemented after CD-2.
3. Report progress on meeting cost and schedule estimates. / Met. Cost and schedule progress is reported to the FPD and program monthly. Full EV reporting will be implemented after CD-2.
4. Perform a post mortem and document lessons learned for elements whose actual costs or schedules differ from the estimate. / Will be done. Not a requirement before CD-2.
5. Document all changes to the program and how they affect the cost estimate. / Will be done using a formal change management system after CD-2.
1
GAO Best Practices for Schedule
Appendix B
GAO Best Practices for Schedule
Table C-1 shows the project team self-assessment and the ICR team assessment of conformance with the GAO best practices for schedule.
Table C1. GAO Best Practices for Schedule(GAO-16-89G)Point / Description / Criteria / Project Team Assessment / DOE-PM notation
1 / Capturing all activities / Schedule reflects all activities as defined in the project work breakdown structure and scope. / Met. Schedule was developed using a comprehensive and well-defined WBS.
2 / Sequencing all activities /
- All activities logically sequenced in the order that they are to be carried out
- Schedule should not rely on constraints to keep schedule dates from changing
- Lags should only be used where appropriate. Leads are to be avoided.
- Summary or level-of-effort activities should not drive the schedule.
3 / Assigning resources to all activities / Schedule should realistically reflect what resources are needed to perform the work, whether all required resources will be available when needed, and whether any funding or time constraints exist. / Funding constraints are built into the RLS. Resource leveling will occur after CD-1.
4 / Establishingthe duration of all activities / Durations of activities should reflect how long each activity will take to execute. The same rationale used for cost estimating should be used for schedule estimating. Durations should be as short as necessary for accurate progress reporting. / Met: Durations are estimated by the L3 managers. In the case of planning packages, durations will be updated as the schedule develops.
5 / Integrating schedule activities horizontally and vertically / Schedule should link the products and outcomes associated with already sequenced activities. / Met. To be reviewed by the ICR and IPR.
6 / Establishing the critical path for all activities / The longest duration path through the sequenced list of activities should be identified. / Met by using the critical path method.
7 / Identifying total float / Schedule software computes total float so that schedule flexibility can be determined. Activities on the critical path have zero float in the absence of constraints. Large total float typically reflects incomplete logic. / Met: Primavera P6 is used to define critical path and float.
8 / Conducting a schedule risk analysis / Schedule risk analysis uses a good critical path method schedule to model risks and simulate results of schedule risks collected from project participants. Results are the level of confidence in meeting a project’s completion date. / Met. Impacts to CD-4 due to schedule risks was calculated using a Monte Carlo simulation at a 90% CL.
9 / Updating the schedule using logic and durations to determine the dates / Schedule updating requires recording actual start and finish dates, and remaining duration to project activities. Maintaining schedule logic through statusing is necessary to reflect true status. / Will be done in practice prior to CD-2, including status updates on progress.
10 / Baselining the schedule / The project schedule should be baselined for use in determining variance from schedule commitments. / Will be done in practice prior to CD-2 and fully implemented after CD-2.
1