Annex 1: Our process for assessing serviceability
Our annual serviceability assessment is carried out at the level of the 4 sub-services: above and below ground assets for water and sewerage services. The assessment is based on detailed consideration of quantitative data and commentary submitted by the companies in their June returns, and data submitted to the Drinking Water Inspectorate and the Environment Agency. Table 1 below lists the current basket of indicators used to inform our assessment for each sub service.
The output of our analysis is a headline assessment of serviceability for each sub-service for each company, ranging from (best to worst), ‘improving’, through ‘stable’ to ‘marginal’ and ‘deteriorating’. We publish these assessments each year in our Financial Performance and Expenditure report. We elaborate on these terms below.
Table 1: the current basket of indicators
Water service / Sewerage serviceInfrastructure
(below ground) / Non-infrastructure
(above ground) / Infrastructure
(below ground) / Non-infrastructure
(above ground)
% of properties receiving pressure/flow below reference level - "DG2" at end of year / Water treatment works with determinations containing coliforms3 / Number of properties internally flooded “DG5” due to overloaded sewers / % Sewage treatment works failing numeric consents
% of properties affected by supply interruptions - "DG3", but only the >12 hour time band / Number of possible enforcement actions at water treatment works / Number of incidents of property flooding “DG5” due to sewer collapses / % of Population Equivalent (PE) served by non-compliant works based on look up table (LUT) 6
Number of mains bursts1 / % of water treatment works4 with leaving water turbidity samples’ 95 percentile above a threshold of 0.5 Nephelometric Turbidity Unit (NTU). / Number of sewer collapses5 / Sub-threshold indicators7 of forecast biochemical oxygen demand (BOD), suspended solids (SS) and ammonia8 compliance.
Water Quality: Iron (100% minus % mean zonal compliance)2 / Percentage of the number of service reservoirs with coliforms detected in more than 5% of tests. / Number of Category 1, 2 and 3 pollution incidents occurring at combined sewer overflows and foul sewers
1 Calculated from burst/1000 km and mains length
2 Before 2006 was percentage of zones with iron failures. DWI has provided additional historic data.
3 Number of water treatment works with coliforms as a percentage of coliform samples taken at works (avoids double counting of WTWs sample failures)
4 Total number of water treatment works where turbidity is measured. Number in relation to those works where turbidity is not measured informs assessment.
5 Calculated from collapses/1000 km and sewers length. Includes failures of rising mains.
6. Includes UWWTD failures from JR 2003.
7. Forecast is based on events where average, 95 percentile and maximum values of regulatory samples in last three years are more than 0.5, 1, and 2 times the relevant consent value respectively.
8. Ammonia included from JR 2006.
Steps in our process
Figure 1 below illustrates the steps in our process. We receive the June return information and data from quality regulators, and complete our early analysis, before writing to companies with our preliminary assessments. This provides companies with a further early opportunity to explain or elaborate on the data. We aim that the company should arrive at the same conclusion as ourselves. Where this is not so, companies are able to explain their views to us, enabling us to consider them and explore any issues in good time.
Figure 1: Steps in our process for assessing serviceability
We publish annually in August or September the serviceability assessments in our Financial Expenditure and Performance (FPE) report. From this year, we no longer call the assessments published in the FPE report ‘initial’, because we consider that our published assessment could only be changed in exceptional circumstances.
Our approach to the data
Our assessments are underpinned by annual monitoring of trends in the basket of indicators set out in table 1.
Trends in the metrics are more important than the actual values. Planned step changes in the metrics and related funding are determined through the periodic review process.
We recognise that circumstances and data capture practices do vary across companies. We may sometimes find it useful to compare and contrast performance in terms of absolute values across the industry, whilst recognising this has limitations. Such comparisons may inform our judgements about the regulatory action that may flow from a deteriorating serviceability assessment.
Indicators for each company are calculated from data received directly from the companies through the June Returns and indirectly from the quality regulators by the arrangements set out in MD109. The annual values are then normalised by expressing the actual values for each year as a ratio to the average for the full period for which data is available for that indicator. The data is then plotted on a graph for each sub-service. The example in figure 2, below illustrates this, for the sewerage infrastructure (‘below ground’) sub-service, for all England and Wales companies. We exercise judgement over the value of best historic levels of our serviceability indicators, for example by taking into account the consistency and variability of the data, which might otherwise yield an unrealistic minimum for reference purposes.
Figure 2: Illustrative example of serviceability trend monitoring
If the trend in the time series is upwards, then the trend is adverse, if horizontal, then the trend is stable, if downwards then the trend is improving.
Temporary influences
We have outlined how serviceability assessment is based around trends. Time trends require that data must be consistently defined and generated over the period of time that is assessed. A change in definition introduces a discontinuity in a time trend and makes interpretation difficult. Reliable and defensible time trends are key to our analysis.
Where companies have changed their method of measurement, then we require companies to expose the discontinuity in trend produced by the change. This may be through reporting parallel data series, on the ‘old’ and ‘new’ basis for a period, or through backcasting of data. We also require companies to provide a full explanation, and this must be subject to reporter scrutiny.
Serviceability indicators may display a degree of natural volatility, and are often affected by shorter-term influences such as the weather. Company operational practices, as well as defective assets, can also affect performance and service trends. Our assessments take into account shorter-term influences on performance, including benign or hard years, temporary operational factors, and exceptional events. We look hard to understand the consistency in trends. Companies’ asset systems should be robust to handle variations in the demands placed on them, so it is for companies to provide full justification if they believe we should take account of any such temporary influences in our assessments.
Making the assessment
We look at the trend in each indicator and assess both the direction of any trend, and the degree to which a trend is sustained. We make comparisons with the best level achieved in past, and look for significant changes.
We look to the company’s commentaries in the June Returns for explanations to inform the serviceability assessment. Company Reporter’s reports on the June Return add value, often with more detail and similarly inform the assessment. We also make use of statistical analysis to inform the existence of trends in data, and to inform our view on whether any change should be judged significant. We also submit our assessments to external quality control by an independent reviewer.
Serviceability assessments range from (best to worst) ‘improving’, through ‘stable’, to ‘marginal’, and ‘deteriorating’. Where serviceability is less than stable, there is usually a progression from ‘marginal’ to ‘deteriorating’. Thus, for example, serviceability may be assessed as ‘marginal’ where bursts have steadily been increasing over successive years, but where indicators of service to customers remain stable or improving. Or a marginal assessment might be made where there appeared to be an adverse trend developing, but there had not yet been a significant cumulative change from the reference level. A rule of thumb would be that if there were three successive increases in an indicator, then, subject to it being a significant cumulative change from the reference level this would be considered a deteriorating trend. Two successive increases that, together, approach a significant step up from the reference level would inform a marginal trend.
To reach a stable assessment, from a deteriorating or marginal position, we will generally expect data in successive years to confirm that adverse trends have been reversed. Thus the general pattern of recovery from an assessment of deteriorating serviceability comprises an initial arresting of deterioration, followed by a recovery step, or steps, to achieve performance broadly in line with the reference level. Where companies are recovering from deteriorating serviceability we will continue to look for performance in line with reference levels to confirm stability.
Our use of the indicators
There is some overlap between the data used to inform serviceability and levels of service indicators that inform the overall performance assessment (OPA). Our expectations on levels of service outputs have been set down in each company’s final determination. Companies must deliver these defined outputs, as well as achieving overall stable serviceability.
In reaching our overall serviceability assessment for each sub-service we consider all of the indicators together. In doing this we do not use a formal weighting scheme. We expect each to be at least stable, and will seek further scrutiny, and appropriate follow up in relation to a sustained trend in any one indicator. But, for each sub-service there is a key indicator that we place particular reliance on, in guiding our overall assessment, as set out below.
For the infrastructure sub-services, we take particular account of the asset performance indicators burst mains (water service) and sewer collapses (sewerage service). These indicators are particularly important in informing future service capability, and in our view provide the clearest evidence about the quality and extent of a company’s interventions to maintain the long-term integrity of its assets.
In non-infrastructure, for above ground assets, compliance carries particular weight in our assessment because these are statutory requirements, and asset maintenance failings would be evidenced by compliance failure.
Compliance may also be achieved for a short time through additional operational activity, masking underlying deterioration in the ability of assets to deliver serviceability. So, whilst we give primacy to treatment works compliance – coliform compliance (water service) and numeric consent compliance (sewerage service), we also look at the trends in the other indicators that underlie the headline performance to further inform our judgement.
Sub-threshold indicators
The additions of sub-threshold indicators to inform future performance have been largely successful. In 2002 we introduced turbidity threshold monitoring at water treatment works. This was not because we wanted companies to achieve better standards than the permitted concentration or value (PCV) at water leaving treatment works. It was because the crossing of a particular threshold value (0.5 Nephelometric Turbidity Units) is seen as a precursor to a problem with the treatment. We recognise that in some cases this can be discounted because there is no filtering process required and it is a reflection of a change in raw water quality.
In 2002 we introduced sub-threshold indicators for sewage treatment. These look at the final effluent sample results in respect of three thresholds (half, 95 percentile and twice the consent) to detect underlying trends in biochemical oxygen demand (BOD) and suspended solids (SS) performance. In JR06 we have added ammonia (NH3) performance. We give a little more weight to the 95%ile events, since they align more with look up table (LUT) compliance.
Non-infrastructure assets comprise not just treatment plant, but also storage (water service), pumping (both services) and sludge treatment (both services, sewerage service predominating). There is scope for adding to the basket of indicators for non-infrastructure assets to inform future serviceability (for example, trends in breakdowns) which are considered in annex 3.
Annex 2: Our process when serviceability is less than stable
We expect service to be restored in line with reference levels of service. In the absence of robust justification to the contrary, the best historic levels of service and performance achieved by the company provide the reference level for achieving stable serviceability. In considering the best historic levels of service and performance, we would refer to a period of time where data was of acceptable quality and consistency. Best historic levels would not be defined by reference to a single anomalous data point.
Where serviceability is less than stable, we deploy a staged approach to regulatory action to secure corrective action by the company. This includes requiring companies with deteriorating serviceability to set down action plans with milestone dates for activities aimed at restoring stable serviceability for our scrutiny. This requirement is supplemented by regular progress reporting, usually at six monthly intervals, following guidance from Ofwat’s Capital Maintenance Team, and with reporter scrutiny.
This approach aims to bring defaulting companies back on track and to prevent a situation developing that could cause a more serious failure of service. It also informs our judgement about whether other regulatory action is required.
If the milestones in the company’s action plan are not achieved, or if we judged that insufficient progress was being made to achieve the regulatory requirement, then we can also consider taking formal action. This could be through a legally binding undertaking accepted by us, or through an enforcement order. In either case these would clearly set out the actions that must be taken by the company to restore stable serviceability.
We expect all companies to achieve stable serviceability regardless of the relationship between actual expenditure and that assumed in setting price limits. The company, in accepting a determination, also accepts the requirement to maintain (or achieve and maintain) stable serviceability. If the company does this through a set of investments that cost less than Ofwat has assumed, it can retain the financial savings as part of its efficiencies in the usual manner. But, companies must invest to maintain stable serviceability, even if this may require more expenditure than assumed at price setting.
As set out in MD212, financial adjustments are also likely to be appropriate if a company fails to achieve stable serviceability.
Annex 3: Ofwat response to UKWIR technical recommendations
The Report makes many recommendations for company specific indicators. We believe that they are potentially very useful in informing day to day and forward planning. We leave these to companies to decide which to use. The tables below are our detailed response to the indicators recommended for general reporting.
Water infrastructure
UKWIR recommendation / Indicator / Ofwat viewRetain / DG2 / Agree. We note that this indicator may relate more to lack of network capacity than the need for capital maintenance, but remains as a ‘system fit for purpose’ indicator.
Modify / DG3: include interruptions over 6 hours to align with OPA measure. / We will review the case for this change, and will reach a view in time for JR08. Inclusion of over 6 hour interruptions may tend to mask trends in more serious events. However, it may be a fuller representation of service continuity.
Bursts: Dis-aggregate into reported and detected. / We are not convinced on this proposal, since the total bursts figure reflects the level of activity needed to maintain the asset system fit for purpose. It is for each company to demonstrate the economic optimum for stable serviceability. Companies may explain their views on the implications of reported versus detected bursts in their JR commentary.
Drop / Iron pick up.
Nr Zones failing iron PCV. / We agree, and have dropped this indicator for JR06. We accept that this indicator has shortcomings at company level. It does have application at zonal level, particularly if supplemented with additional sampling. Companies should not be dissuaded from applying this technique to inform their investment planning.
Propose / OPI (turbidity,iron manganese) developed by DWI
(the arithmetic mean of mean zonal compliance for turbidity, iron and manganese.) / We will explore this further with companies before reaching a view. We recognise that OPI (tim) does at least throw a net around three ‘failure modes’. It is not clear from the report what turbidity and manganese compliance tells us about the state of the pipeline assets. It is not clear whether a trend in the combined indicator provides better information than, say trends in the separate components. We would expect companies to be monitoring each parameter as part of its DOMS.
Iron: Mean zonal compliance. / We have adopted this measure as a replacement for zones failing iron PCV from 2006, with back-cast data obtained from DWI. The trends are broadly similar, and we are content to accept this new indicator.
Future / Natural rate of rise in leakage (NRR). / We support future work in this area, and we encourage companies to consider this in the light of the review of economic levels of leakage. In principle, evidence of deterioration comes from an increasing frequency for new leaks to develop. But there are calculation and NRR definition issues.
We do not think it is helpful to consider NRR as only the break out rate of leaks that occur in the absence of proactive leakage control intervention. Since companies have to implement active leakage control (ALC) this is so rare as to be unrealistic and therefore impractical. Suitable screening of all minimum nightline data for demand led and operational (e.g. rezoning) led variations may help to derive a suitable indicator.
In the meantime we will continue to review trends in distribution losses and renewal activity alongside bursts in appropriate cases.
Water non-infrastructure
UKWIR recommendation / Indicator / Ofwat viewRetain / Nr works where enforcement action considered. / Agree.
WTWs coliform compliance. / Agree.
WTW Turbidity - measured works. / Agree.
SR coliform compliance. / Agree.
Drop / WTW turbidity – all works / We find it useful to inform the completeness of measurement. But we will not publish (as now).
Booster pumping stations mean time between work orders / Agree. We have dropped this indicator from JR06. We will work towards a more suitable indicator based on failures, for trial in JR07.
Future / Booster stations Mean Time Between Failures. Suggested based on Failures in calendar year – i.e. don’t try to measure hours run, or include scheduled work. Report says such data can come ‘readily’ from company maintenance systems. / Agree need to develop the measure, which can be extended to all non-infrastructure assets. We will work with companies to develop a suitable measure for unplanned maintenance, to trial at JR07. We will aim to back-cast data for earlier years, using existing 'failure' data extracted from company systems.
Treatment works asset performance measure. Could be MTBF, but needs some form of criticality explanatory factor. / We note the report’s recommendation, and intend to explore this with companies following the trialling of a failures based indicator for pumping assets.
Sewerage Infrastructure