Post Occupancy Evaluation
This guideline provides information on the general processes, which are to be followed for post-occupancy evaluation (POE) of the facilities where capital works have been carried out.
The post-occupancy process can be undertaken by a consultant or Agency/Health Service representative but should include representatives of the agency and be overseen by the Project Control Group (PCG) or governance committee. The Superintendent of the contract, generally the Project Manager or Principal Consultant, is responsible for ensuring that post-occupancy is effectively completed as programmed.
POE refers to the review of capital works facilities operational and maintenance activities undertaken by the consultant after commissioning has occurred and the facility has been fully operational for 12 months in line with the standards, policies and health requirements.
The main objectives of appropriately managing the POE of a capital works facility are to:
§ Assess fit-for-purpose ie the suitability of the asset to meet service delivery objectives
§ Assist continuous improvement by identifying positive and negative aspects of an in-service facility; and then by feeding this information into the planning and design of the future facilities
§ Prepare defect action plans, at an operational level. This entails the correction of deficiencies such as improving maintenance, minor works and/or management decisions.
Each project is individual and will have some issues unique to its own specific circumstances.
Overview
Post Occupancy Evaluation (POE) is a structured approach for the evaluation of the performance of a new or existing facility when it is fully operational – after at least 12 - 15months of occupancy.
A POE focuses on the users’ interaction with a facility and the degree to which the facility is supporting service delivery objectives. This includes measuring the effectiveness of the facility in terms of achieving asset planning requirements.
In addition, it should also examine whether the facility as built, is flexible or adequate to meet current requirements - especially where service objectives have deviated from the original brief and the extent that the original design is meeting current user expectations. These outcomes will give input into the benchmarking of future projects.
A POE may be carried out after at least twelve months of occupancy. This enables users to experience and adjust to their new environment. It also allows for a full cycle of the seasons that provides data for the evaluation.
When is a POE Conducted?
A POE is necessary where a capital works project exceeds $4 million, if required in the project consultant brief or significant innovative construction is involved.
A POE should occur when there has been an opportunity to assess performance through a complete cycle of operation. In most instances, this will occur after an asset has been in operation for over twelve months.
Responsibilities
As POE is necessary for the practice of good asset management, quality management and quality assurance, agencies are responsible for ensuring that:
§ Cost of the POE is budgeted as part of the project
§ Participation in the POE is part of the consultant’s brief.
Benefits
Some of the benefits of POE include:
§ Improved project briefing – that promotes increased functionality and cost-effectiveness
§ Improved asset management – that identifies maintenance and other recurrent costs
§ Improved operational processes – that better services and clients’ needs.
Risks
The risks associated with a POE include:
§ Inadequate definition and management of the POE
§ An undisciplined approach
§ Invalid or unreliable data collection
§ An exclusive focus on negative aspects
§ Unavailability of participants.
The consequences of not doing a POE include:
§ Lost opportunities to improve future facilities
§ Reduction of asset performance in the support of service needs
§ Repetition of current deficiencies in future facilities.
Aspects not Covered
The activities and outcomes are described in Section 4: The POE Study. The study does not specifically address the following issues:
§ Review of specific projects against the applicable technical standards at the time of the project development
§ Issues that require research or in-depth investigation. These would be flagged by the developed POE process but would require further specific investigation to evaluate the issues.
Initiating a POE
The responsibility for initiating a POE rests with the facility owner. A POE may be initiated by a number of different people, including:
§ Program Director
§ Development Manager
§ Asset Manager
Defining the Scope of a POE
The scope of a POE should be identified early. Failure to do so can lead to cost and time extensions and a potential diffusion of effort.
Selecting the Level of Evaluation
A POE needs to select a level of evaluation. An evaluation may be undertaken at either a “broad” or a “detailed” level.
A broad-level POE entails a high-level evaluation. It uses fewer resources than a detailed-level POE; and should be undertaken routinely on new facilities.
A detailed-level POE entails an in-depth level of evaluation. This level of evaluation is usually more resource intensive and should only be undertaken on an “as required” basis.
The combination of a broad-level POE followed by a series of detailed-level POEs may form a highly effective POE strategy. In this strategy, the detailed studies are identified by the outcomes of the broad study. This results in the focusing of resources upon priority issues.
Identifying Stakeholders
The identification of stakeholders in the POE process usually leads to the inclusion of:
§ The facility’s users, such as the staff and clients
§ The facility’s design team
§ The facility’s managers
§ Members of the community.
Other parties may also be involved. For example, in the case of an identified environmental issue the environmental protection authority may be involved with a POE.
Selecting an Evaluation Team
The selection and appointment of an evaluation team should be based on the scope, level and complexity of the POE, to determine the type of expertise required.
The selection of an evaluation team may be achieved by engaging external consultants, in association with “in house” resources. This provides a broad range of skills to address a wide variety of issues, whilst ensuring an appropriate degree of objectivity. Team membership should be based upon an individual’s expertise in relation to the anticipated issues.
A small POE may be undertaken by a single person.
When the appointment of a consultancy team is the chosen evaluation team tendering for the consultant according to the POE – consultancy Brief is required. The brief is included as Appendix C.
POE Templates
Data collection templates have been developed to correspond to the types and levels of data outlined in Section 4 (below): The POE Study.
The prescribed data collection templates must be used to ensure consistency in the collection of a minimum data set for all projects. This will enable comparisons to be made between projects of similar size, service profile and complexity. It will also ensure that data collected is comprehensive and useful for the informing of future benchmarking for the project and similar projects and updating of DHS processes, as required.
DHS POE templates are currently being developed. Refer to the NSW Health POE templates detailed in the NSW Health Post Occupancy Evaluation Guideline until further notice, in which they will be added the this guideline.
The POE Study
A POE is a structured approach to the collection and analysis of data in relation to facility performance and the translation of these findings into action plans.
Study Activities
The study should address the following activities:
§ Identify stakeholders
§ Communication with stakeholders
§ Develop an appropriate data collection methodology
§ Select data collection instrument/s
§ Conduct and analyse survey
§ Write a draft report
§ Review draft report with the participants
§ Prepare an action plan
§ Complete a final report.
Data Collection Methods
The information can be obtained by structured and semi-structured:
§ Focus Groups
§ Interviews
Structured, or loosely structured, interviews are an effective way of getting direct feedback from key staff/users. Experience would indicate interviews tend to be hard to control and a flexible approach is suggested. A checklist of target issues is recommended. Output tends to be verbatim quotes.
§ Survey / Questionnaire
Perhaps the most commonly employed technique is the structured questionnaire. This approach requires the development of a range of questions geared to measuring user responses to the required subject areas.
A major advantage of this approach is that it allows the survey of a large sample of users and thereby improves statistical reliability. Generally the best advice is “keep it simple”.
There are many pitfalls with questionnaire design. Before embarking on the cost of a major survey it is recommended that pilot questionnaires be tested. Also consider carefully how the questionnaires are to be analysed.
§ Workshops
The techniques of Value Management can be applied to PCR. A structured workshop can be organised with key participants to gather user responses. This approach has the advantage of being a focussed, short duration technique.
§ Financial analysis
§ Inspections, including physical facilities and records.
This approach employs a team of “experts” to visit the facility and assess its performance by observation. If the team is well selected, a significant amount of information can be gleaned simply by observation. Some amazingly powerful indicators of design problems can include pedestrian desire-lines across lawns where paths have not been provided, ‘temporary’ signage replacing the designed signs, posters and notices covering ‘observation’ windows, corridors used for storage purposes, windows propped open for ventilation, broken door hardware, offices or spaces accommodating more than their designed capacity, etc. Construction detail issues can include leaking roofs, overflowing gutters, cracking brickwork, excessive heat gain, defective door hardware, leaking taps etc.
While this approach is effective it is preferable to combine observation with interviews to check that the identified problems are correctly diagnosed. Similarly it is often desirable to allow users to participate in the “walk-through”. Identification of these “problems” raises further issues as to whether the cause was briefing, construction, supervision or user initiated.
The importance of the aspect under consideration, the time available, the cost and availability of the target groups will determine the choice of the data collection instrument/s. Target groups may include facility managers, facility staff and members of the community.
Aspects of a POE
A POE may address many aspects of a facility. A POE usually addresses a selection of the following aspects.
The aspects of a POE include:
Processes
§ Project identification
§ Project definition planning
§ Brief preparation
§ Project procurement
§ Facility commissioning
§ Facility operations
§ User consultation
Value and Costs
§ Capital cost
§ Valuation
§ Benchmarks
§ Social, historical and heritage value
§ Value for money
§ Replacement cost
§ Cost-benefit
§ Recurrent costs (including: utilities, maintenance, support services, staff, etc.)
Construction
§ Hospital operation compromised
§ Defects, warranties, staging/program, environmental issues
§ Plant or equipment impacts
§ Construction standards.
Risk Management
§ Service delivery risks
§ Project delivery risks
§ Other risks
Suitability
§ Effectiveness of the facility in achieving core business outcomes
§ Delivery of outcomes specified in the project definition business plan and concept stages
§ The need to overcome a difficult problem
§ Fitness for purpose, both current and future
§ Location, design and functional appropriateness
§ User satisfaction (staff, visitors, contractors)
§ Occupational health and safety, compliance issues.
Utilisation
§ Size, capacity, access and functionality
§ Current occupancy levels
§ Capacity for growth, modification and change
§ Adaptability.
Technical Performance
§ Energy and environmental
§ Human comfort
§ Structural, civil engineering
§ Building services
§ Car / ambulance access and parking
Materials, Fittings and Finishes
§ Efficiency, functionality and ease of maintenance of:
- Doors, sanitary fittings, windows, etc.
- Paint, floor coverings, etc.
Condition
§ Maintenance liability
§ Anticipated life expectancy
The POE Report
The aim of the report is to improve functionality and technical performance of planned facilities and any restorative actions required to improve asset performance. It should include both the positive and negative aspects of a facility. This enables the positive aspects of the facility to be repeated and the negative aspects to be addressed in future project documentation. Reporting findings and recommending actions.
A POE report should include:
§ An executive summary (minimum 2 pages)
§ Methodology used
§ Data and analysis
§ Findings
§ Recommendations
§ Action plans.
Recommendations and action plans may include actions that feedback into asset plans (including: project definition plans and project briefs), maintenance plans or minor works plans. In some instances further investigation may be necessary.
The report will be provided to both the agency and Department of Health initially as a draft, prior to the final report submission. The final report is to be provided as ten hard copies and an electronic copy (Microsoft Word 2000 format).
Two oral presentations of the report findings are to be presented to CPSP staff and the facility management and programmed areas concerned.
Post Evaluation Phase
Implementing Actions and Measuring Effectiveness
The post evaluation phase entails implementing recommendations and actions and measuring their effectiveness.
The activities of this phase include:
§ Seeking approval of the recommendations and action plans
§ Implementing recommendations and action plans
§ Reviewing the effectiveness of these actions.
Whilst the POE team will seek generate the recommendations and action plans, the implementation and review of these actions will be the responsibility of agency management, as they occur after the evaluation team has finished its work.
POE Performance Measurement
Measuring the performance of a POE requires an assessment of the following aspects:
§ Comparison of performance against original targets or appropriate benchmarks
§ Management of the process
§ Quality of the report and action plans
§ Degree to which the recommendations and action plans were implemented.