Developing key national indicators for the united states

katherine wallman, KENNETH prewitt, and susan schechter

Comments by John Barber

This is a project with ambitious objectives which deserves our support and which other countries will hope to emulate. However it faces a number of problems and pitfalls many of which are referred to in the paper. I propose to comment on these from the perspective of a potential user. In my time I have worked as a macro-economic forecaster, as an energy analyst and latterly as an innovation policy maker. Recently I have been engaged in a study for the OECD which among others covers the role of Science, Technology and Innovation indicators in the formulation of technology and innovation policy.

2. The authors define an indicator as a statistical measure that tracks change over an extended period of time. They set out a number of strict criteria which any series to be included in the “Key National Indicators Initiative” (KNII) must meet. These include ‘accuracy’, lack of variation unrelated to real changes in the underlying phenomenon being measured and completeness which reflects the degree of match between what the indicator measures and the phenomenon of interest.

3. The use of indicators to understand what is happening, by policy and business analysts and by researchers, requires that the phenomenon which the indicator represents must be embedded in some kind of structural explanation, theory or model describing the system in which that phenomenon is embedded and which determines the value of the indicator and the influence that the phenomenon concerned exerts on related entities. The significance of the indicator can only be properly understood in relationship to the system concerned and in the movements in the other indicators/variables which help to describe the functioning of the system. The authors provide an excellent of this when they refer to climbing US divorce rates.

4. The divorce example throws up another important issue. Because the world in which we live is complex and our understanding of most economic and social phenomena is somewhat limited, there are often competing explanations or hypotheses attempting to explain the same set of related phenomenon and indicators. In the absence of perfect knowledge there is scope for politicians, analysts and researchers to hold a range of different prior beliefs. This tends to further increase the variety of explanations on offer. The same indicator may have a different significance for different groups or individuals depending on which explanation they subscribe to. Moreover individual explanations evolve over time, new ones arise and others go out of fashion as circumstances change. For example I heard stockbrokers say in at different times in the 1970s that rapidly rising supply of money caused both rising and falling share prices in London. Thus, even if an indicator continues to closely represent an unchanging phenomenon, its significance for analysts, policy-makers and researchers will nevertheless change over time. [Since the 1960s the principle focus of UK macro-economic policy has varied among the current balance of payments, the money supply, the exchange rate, the growth of money GDP and the rate of inflation.]

5. Competing explanations require different data sets and focus on different indicators and thus make different demands on official statisticians. Some important and relevant phenomena can be measuredin line with the criteria set out by the authors for inclusion in the KNII but this is not possible for others. For example it is possible to obtain reasonable data for R&D undertaken by firms but not for other forms of technology generation suchengineering development. As the relative importance between R&D and other forms of technology generation vary systematically across sectors our inability to measure the latter can significantly distort the making of innovation policy. We need to be careful that the part of economic and social development which we can measure does not cause us to ignore equally important phenomena for which measurement is either more difficult or not possible at all.

6. The boundarybetween Business R&D and other forms of technology developmentis described in the OECD Frascati Manual which underpins the definition of R&D for tax purposes in a number of countries. Despite the OECD’s best efforts there are suspicions both in the industrial and the policymaking communities that the definition is not applied consistently across advanced industrial countries and that caution needs to exercised when comparing R&D performance across countries. International comparisons of the results of innovation surveys, answers to which are often subjective, involves significant difficulties. These sorts of problem can without doubt be found in other areas of policymaking.

7. National Indicators are by their nature aggregates across sectors, regions or different groups of the population. Even if the phenomenon being measured remains unchanged within each subcategory the relative importance of the various categories can change significantly over a longish period of time. If behaviour differs across subcategories then the aggregate behaviour of the indicatorwill change even though behaviour remains unchanged within the individual categories themselves. Both providers and users of data need to be constantly on their guard against such ‘aggregation bias’. In the 1960s Professor Terence Gorman set out the qualities required of aggregate economic indicators for use in estimate structural behaviour. While his approach may have represented a counsel of perfection it served to demonstrate the kinds of restrictive assumptions which are implicit in estimating behavioural relationships between aggregate indicators.

8. Macro-economic analysts and forecasters often focus on the correlation between various indicators (so-called reduced form relationships). This is often used to calibrate less well founded data sources against more established indicators. Such data sources can provide useful ‘lead indicators’ of key official statistical sources. Similarly a consistent pattern of reduced form relationships betweenrelated data series can generate confidence in the picture they present even if the quality of each of the individual series is indifferent.

9. Some indicators such as infant mortality or GDP represent social or economic outputs which are desirable in the own right. Others are mainly instrumental, for example we target the money supply in order to control inflation. Both kinds of indicators may be used as policy targets or measures of policy success encountering a number of possible problems:

  • Some indicators may be subject to ‘Goodhart’s Law’ in that using them as a target may change their behaviour and significance e.g. money supply, number of articles published in academic journals;
  • Focussing on one indicator to the exclusion of others may mean that the measured success of a policy may be bought at the expense of other desirable outcomes e.g. hospital weighting lists, proficiency of school children at certain core subjects;
  • Some indicators may not be as complete a measure of the corresponding phenomenon in which e.g. the long-running debate on the deficiencies of GDP as a measure of economic welfare.

10. The authors see an important rolefor indicators in allocating resources to different public uses. One might perhaps distinguish between the justifying a given allocation of resources to a wider public and the process of deciding what that allocation of resources should be. The latter involves the weighting of different desirable outcomes which is difficult and contentious process as illustrated by Professor Kenneth Arrow’s work on the impossibility of constructing a socialutility function. It also involves comparison ofincommensurable outcomes e.g. educational attainment of young children against the health of old people.

11. I would like to finish by urging those responsible the KNII to closelyinvolve expert users at all stages. Their work requires them to take a close interest in the construction and significance of the data and they are often the first to spot any unforeseen errors,any changes in the behaviour and significance of an indicator and to identify any special factors which may distorting its movement. They often have access to other data sources which can validate the indicator or warn of emerging problems. They are also likely to be an important source of demand for new indicators.

12. Finally I want to wish the authors and their colleagues success in constructing the KNII. None of the above is intended to deter them in their task but instead to help ensure that the indicators are used as effectively as possible.

John Barber

11th November 2004