What can we learn from Benchmarking?

Mr. Chairman, Ladies and Gentlemen. My name is Bård Uthus and my speech is based on paper 6.37 called “What can we learn from Benchmarking?”

2) BKK

At first, I will give a short introduction of the company I am working for. BKK is one of Norway’s largest companies in the field of generation, sale and transport of electrical power, and is situated in the western part of Norway. The company is owned by local authorities, and by Statkraft, the largest power producer in Norway.

The distribution company has about 550 employees whichserves approximately 170000 customers with almost 6 TWh. We operate 16 000kilometres of power lines and cables. This is quite much for Norway, but I think little in a European context.

3) Introduction

This speech is about Benchmarking, and I will start with some background information on why Benchmarking is important to us.

A lot has happened in the Norwegian electricity industry during the last decade. It started with the new Energy Act in 1991, leading to the Norwegian electricity market being deregulated, and supply and transmission unbundled. At the same time the grid was defined as a natural monopoly and hence regulated financially through a Rate of Return regulation. We were guaranteed cost recovery and therefore given incentives for inefficient activities, which the customers had to pay for. Looking back, even we agree with this.

Our regulator wanted to create a drive for higher efficiency and introduced an incentive based regulation in 1997. We call it a revenue cap regulation, but it is quite similar to a price cap regulation as you probably know better.

The new regulation increased profit possibilities. Of course, the owners like the taste of money, and became more profit seeking. At first, the model did not include incentives to maintain quality of supply. To resolve this problem, the Regulator introduced quality dependent revenue from 2001, which is based on a compensation for energy not supplied. As a curiosity, we have to compensate some industrial customers 12 Euro pr kWh, I repeat 12 Euro. Many are surprised, or shall I say shocked over the high cost, but it certainly gives us incentives to maintain quality of supply for these customers.
Now we are in the second regulatory period, and we are quite satisfied with the regulation. We certainly don’t want to go back to a rate of return regulation.

Benchmarking is an important element in this model, but the new regulation also led to an increased interest for benchmarking in general. We were asking ourselves two questions, First, and perhaps most important, what we can do to improve ourselves. And, of course, get arguments to criticise the model. Criticising the regulator is a part of the game.

4) The DEA model

As previously mentioned, the Norwegian Regulator uses Benchmarking to reduce each company’s revenue. This reduction varies from 1,5 % for the most efficient grid companies to 6,0 % for the least efficient ones.

I have called the Data Envelopment Analysis model, or the short term DEA,a regulatory benchmarking tool. We often see that regulators around Europe prefer this benchmarking model when they want create a drive for higher efficiency in the industry. I know that the model is used in Norway, Sweden, Finland, Austria and the Netherlands, but probably in other countries as well.

In my opinion it is quite natural that Regulators prefer the DEA as their benchmarking tool. The model is based on a linear programming technique, which is a standardised and objective method. This is important to note, as the Regulator then can be seen as impartial. Another advantage with the model is that it allows several inputs and outputs, without deciding for example cost functions. This allows the Regulator to benchmark the industry without having to discuss every internal or external cost driver.And believe me; each company claim that their individual cost drivers must be regarded. But frankly, we can’t demand the Regulator to know every internal or external cost driver for each company.

The cost drivers are described as outputs in the model, and the Regulators often describe them with variants of customers, energy, network and quality.If you want to know more, the model is described in detail in paper 5.35 from Finland.

As the slide shows, the envelop curve is drawn through the most efficient companies. This is the reference companies,which the other companies are compared to. It is important to notice that several reference companies decide the efficiency for the rest of the companies. There are as much as 40 reference companies in the Norwegian model, and all the other companies have more than one reference companies. Incorrect data for one reference company is normally not critical to the others.

Still, there is no other part in the Norwegian model that has been criticized as much as the benchmarking model. Arguments such as "it’s unfair" or "that it doesn’t measure efficiency" always arise. I must admit that I have been criticising as well. However, I have realized that the alternatives are worse. The question is; should benchmarking be used at all? If you choose to, I strongly believe DEA is one of the best solutions. Still the Regulator must take into consideration that Benchmarking is not exact science.

One of the reasons for the criticism is probably that the DEA is difficult to understand. The model does not inform us on how to improve efficiency, only that we should reduce costs. In fact, the model is often referredto as a black box. We put something into the model and get an answer out, without knowing why the result became as it did. Consequently, we have participated in several other benchmarking studies to answer the most important question. How can we become more efficient?

5) Parametric studies

Common for the studies we have participated in, is that they are based on parametric benchmarking methods. My experience is that the methods require accurate knowledge about conditions within a company; which can be both difficult and time-consuming to obtain even for the companies themselves. On the contrary, the results are more detailed and indicate how we can become more efficient.

Cost function models may be quite sensitive for poor data quality. As I told you, the DEA had 40 reference companies. As showed on the slide, parametric methods often give only one reference company. The model presumes that all information given from this company is correct. If not, all the others will of course get wrong efficiency measurement.

Our experience is that it is difficult to establish a reliable method.

6) Experiences from BKK

I will now continue with our experiences. First, and most important, we have to admit that Benchmarking is not an exact science. A Benchmarking study always seems to be followed up withnew studies. My experience is that this is a never ending story, and the goal seems to a more exact model instead of starting to improve. Remember that the benchmarking study is only a preliminary work; the tricky part is what we must do in order to improve. Many companies do not realise this, and then they don’t finish it properly. The result is that many companies blame the model, instead of starting to improve.

You must remember that your company has qualification to start actions of improvement without benchmarking results. In fact, a lot of these actions are already known before the company begins the benchmarking. I clearly recommend that every company should start making improvements and not wait for even more exact benchmarking studies to come. I do believe many grid companies need a push to begin action in order to improve. A benchmarking study may be a key to this push, but then you don’t need an exact one.Therefore, our experience is that the benchmarking studies themselves have limited benefit for BKK

7) Regional benchmarking study

To exemplify this argument, I will show you results from a benchmarking study carried out inside the companies owned by Statkraft. This particular benchmarking study was introduced in order to exploit synergy effects. Before beginning the benchmarking study, we discussed whether precise benchmarking results were needed or not. Our conclusion was that it is better to do a simple benchmarking and consequently start actions of improvement, instead of spending too much time on the benchmarking itself. Therefore, we decided to use a simplified cost function based on a replacement value of the network, being aware of that this is an imprecise size. Our main idea was that all components must be maintained and operated. If you have many components, the replacement value is high and the operating costs should also be correspondingly high.

As shown in the figure, our benchmarking study indicated that we spent too much money on medium voltage substations. Instead of demanding more exact results, we started looking for different actions of improvement. As a consequence, we decided to change from time-based to condition-based maintenance strategy for MV substations. But, I also think we would have done this even without the benchmarking. However, the study accelerated actions of improvements.

8) Internal monitoring of the cost development.

The incentive based regulation led to a demand for more precise monitoring of the economical development inside our company. As the regulator decides the income, it is natural to benchmark the cost development year by year. However, we have bought several grid companies in Western parts of Norway during the latest years. This makes it difficult to compare the costs year by year. To measure the effect of the improvement, our owners have asked us to report the ratio between the operation costs and the replacement value. This value is relatively stable and independent of mergers.

As you see, the ratio shows a declining tendency, or in other words, reduced operation costs. However, we also experienced a cost increase from 2001 to 2002. Obviously, our owners will not be satisfied if the development continues. We must certainly do something to turn this development around.

9) International benchmarking

The last topic I would like to comment on is international benchmarking. National regulators across Europe seem to exchange ideas and experience, and some try to establish a certain level of international benchmarking. Our Regulator has informed us that international benchmarking might be used in Norway. If the regulator does so, it can be quite dramatic to us. Therefore, we wanted to participate in an international Benchmarking due to two reasons:

  • First, to ensure that we are a qualified discussion partner if this happens
  • Second, to give us an indicative benchmark, compared to European companies.

I find the results rather interesting. The challenges of international benchmarking became clear to us. A large number of problems arise when we compare the efficiency of grid companies across borders. In example, cost levels vary among countries, from high cost countries to low cost countries, which influence the total cost of operating a distribution grid. If we want a fair comparison, this must be dealt with. Also differences in accounting principles, tax and labour legislation make it difficult to compare accounts across the borders. Many countries do not have unbundled transmission and supply, which makes it easy to misallocate the costs. Data access and quality also varies across boarders.

All the problems I have mentioned, is difficult to handle. I do belive the results from the European benchmarking indicates this. The European benchmarking shows a great difference between the most efficient and the least efficient company, which makes the problems with international benchmarking visible. In fact, the least efficient company could have reduced their costs six times in order to become efficient. Do you believe that a cost reduction of 80 % is realistic? Even if the electricity industry has been criticised for being inefficient the result indicates a problem with the benchmarking study as well as a problem for the specific company.

10) Conclusions

Finally, I will summarise my conclusions. It seems reasonable that the regulators around Europe prefer DEA. As a consequence of this, you should know this model to ensure that you are a qualified discussion partner.

Second. You don’t need exact benchmarking studies to start actions of improvements. Get some ideas at this Cired conference and just go home and do it!

Third, international benchmarking seems to be too difficult. Go home and tell your regulator not to do it. Thank you for your attention.