The Existing Statistical Information (We Refer to the Eurostat CVTS2 Survey on Continuing

Indicators and benchmarking as a support to the decision-making process: the Italian experience in active employment policies

DRAFT

Aviana Bulgarelli ([1])

1.Benchmarking in the public decision-making process

Benchmarking bears at least two meanings and can be helpful in two ways in the decision-making process.

 On the one hand, it can be used for a coordinated, consistent and systematic analysis of performance, possibly tied to a system of targets/objectives; in this case, the choice of indicators is strictly linked to the identification of a system of criteria/targets which are considered relevant by public decision-makers, involving, as a first step, the definition of a set of significant and feasible indicators.

 On the other hand, it can be used to identify the most effective intervention policies, models and instruments for achieving the objectives fixed and enabling good/best practices to be pinpointed and transferred in various contexts.

Clearly, this distinction does not imply that both approaches should be considered separately. In fact, the definition and measurement of performance gaps in a system of targets/objectives is a necessary basis for choosing the most suitable good/best practices for a given policy issue. Likewise, the ex-post analysis of results achieved through the policies adopted represents a reference for a possible redefinition of the decision-making process.

However, our intention here is to focus mainly on the first approach, not only because statistical information plays an essential role in this case, but also because, as we shall see, the experience gained over these last years within the European Employment Strategy (EES) appears particularly significant.

To understand better the difficulties encountered and to contribute to a definition of policy-makers’ needs, it is important to remember that, in order to provide an effective support for the decision-making process, benchmarking should be integrated in a broader process, which could be ideally summarised in the following subsequent and iterative stages[2]:

1. Identification of macro-objectives and corresponding performances to be considered as really significant for comparing different countries/regions in order to measure the success of implemented policies.
2. Identification of a system of elementary variables/indicators referable to the macro-objectives and representative of the different possible channels (national/regional) through which they can be achieved (different territories, population targets, production sectors, etc., should be given a different weight).
3. Comparison, on this basis, between the national/regional situation examined and the reference benchmark. This is particularly important since performance gaps can be measured with reference to either a system of predefined targets or the values presented by best performers.
4. At this point, it is necessary to understand and explain the reasons for the differences observed in performance, pinpointing those on which it is more important to act, and identifying as well actions to be undertaken; here, benchmarking is used to find best/good practices with better effectiveness/impact taking into account the current context conditions (sociocultural, political, institutional , etc.).
5. Monitoring policy implementation, analysing benchmarking results and, possibly, correcting the entire process, as a final stage.

Unlike its utilisation in the business field, this approach obviously involves different and particularly complex problems in the case of public policies (for instance in the case of the EES). In fact:

 multiple objectives and, at the same time, specific intervention instruments and policies make this approach complex and often difficult to “systematise” within a single decision-making model;

 objectives sometimes can be conflicting and it is not always possible to identify the corresponding trade-offs;

 it is often difficult to find connections between individual policies/instruments available and the different objectives identified, which are strongly influenced by the specific context conditions;

 the plurality of actors contributing to the identification of objectives and to the implementation and monitoring of policies (these latter having an impact on expenditure decisions) make this scenario even more complex.

Therefore, the measurement of performance gaps only represents the first stage of a decision-making process based on benchmarking, which requires not only measuring but also understanding, planning, testing, monitoring and, if necessary, reconsidering the policy choices made. Within this framework, the need for good statistical information is only one of the necessary conditions; nonetheless, as the EES experience has demonstrated, benchmarking and its related system of statistical indicators have often become a “surrogate” for evaluation, with an appreciable influence on the decision-making process.

2.The European Employment Strategy and benchmarking in the Italian experience

The EES is an example of multi-level strategic programming and represents, at the same time, the reference framework of a complex benchmarking exercise based on the open method of coordination. It is, in fact, a process of “guided convergence” which, starting with a reference framework of guidelines, uses benchmarking as an instrument to support a programming process based on objectives that are also defined quantitatively. For non-Europeans, this brief example could be useful[3]:

As a matter of fact, this process is somehow similar to that initiated with the Maastricht stability pact, but inevitably with a greater degree of freedom:

- firstly, because of the greater complexity of social and labour policies (in terms of objectives, targets and effectiveness of intervention tools);

- secondly, because it is part of a particularly complex decision-making mechanism involving a large number of actors, at different decision-making levels with high political, managerial and instrumental responsibility (from the EU level to the local level).

However, by identifying a reference system of objectives anchored to a set of quantified indicators, the EES has had an influence on the action of policy-makers, at a national level in the first place, but also at a regional and, in some cases, at a local/provincial level.

Fabrizio Barca[4] has already explained in his contribution the influence EU policies have had in Italy on the development programming and governance model, fostering a strengthening of instruments – both cognitive and methodological – for defining policy choices. The EU programming model, based on the meta-objective of “economic and social cohesion”, has triggered a profound reorganisation of structural policies within a “global” model. This model now also constitutes a benchmark for Italian programming at both national and regional level.

Looking specifically at active labour policies, the constant reference to the EU framework has had a considerable influence on public decision-makers. Some examples seem particularly significant in this regard, but we will consider here only their institutional aspects, leaving aside the more specific, albeit important, political aspects of the decision-making process.

We know there is a large number of indicators proposed for monitoring EES targets. They have been partly changed already and will undergo further modifications over time, thanks to the identification of better quality and more relevant indicators with respect to identified priorities. In particular, employment indicators are linked to the different key aims defined by the Lisbon European Council referring to strengthening employment levels, equal opportunities for men and women and, more in general, greater participation in the labour market of the older population and women.

A first example of the influence of the benchmark mechanism refers to the recent in-depth reform process of the labour market in Italy, and it is quite significant since it concerns a typically national policy sphere. Many of the targets set by the EES have already been assimilated into the national and often regional context. This resulted in stronger emphasis, for example, on the female component or on the older population, even when – as in many regions in southern Italy – the high priority of the “elderly” component of the workforce does not fully correspond to the local labour-market conditions.

Again, at a national level, this impact has also emerged during the implementation of the labour policy monitoring system, whose reorganisation has been partly inspired by the European benchmark model, with all the difficulties ensuing from the aim of coherently linking up a system involving multiple actors at various levels and with different degrees of involvement. As already mentioned, the Italian labour policy programming, implementation and evaluation process is articulated at many decision-making levels with accentuated forms of subsidiarity, both vertical (central administrations, regions, provinces, etc.), and horizontal (employers associations and trade unions). This complexity also inevitably concerns, both from the producer and the user side, the statistical information system which is necessary to monitor and evaluate the policies adopted. The objective of compatibility with the Lisbon benchmark has inevitably made the entire process more complex and required the creation of a large number of “technical units”, involving a multiplicity of actors.

Among the Lisbon benchmark indicators, that concerning lifelong learning represents another emblematic case of how having to comply with a system of indicators (and performance gaps) has acted as a catalyst on the attention of policy-makers, prompting extensive corrections in targeting and rebalancing the policy mix.

In Italy, the indicator used for lifelong learning, represented by the participation rate of population aged between 24 and 65 years in permanent and continuing training, shows a severe structural lag: against the objective set to a participation rate of 12.5% by 2010, our country only reached 4.7% in 2003, compared to an EU average of 9.7%. This context is moreover characterized by strong disparities, not only among generations (mainly to the detriment of older groups) but also among regions (southern Italy is particularly affected). Over and above the interpretative problems that this indicator also presents – and we shall come back to this later – it is clear that public decision-makers are well aware of this fact; however, once integrated within the Lisbon framework, this indicator has had a significant impact on policy makers, who only had little room for manoeuvre. In a situation where resources are limited, redirecting them towards lifelong learning policies means reorganising alternative intervention policies, with consequences:

- not only from the political point of view but also regarding the management of the governance system referring to active labour policies;

- from an organisational point of view, due to the need to redirect the “machine” towards greater needs pertaining to the programming, management and implementation of interventions.

In this context, greater attention has therefore been devoted to adult education and training policies, both in general and specifically addressed to the employed population, by developing new types of training supply. However, it was also clear that, because of the gap with the reference benchmark and given the existing budget and time constraints, it was going to be a “mission impossible”. Even with greater financial efforts and making adjustments by redefining the beneficiaries covered by the main policy tools available, the only real “room for manoeuvre” had clearly to be found in trying to make the most of available resources.

The existing statistical information – we refer to CVTS2, Eurostat’s survey on continuing vocational training in enterprises – showed that, within the broader adult population target, training activities for the employed population had a particularly negative ranking in the European context: in fact, only one Italian firm out of four (around 24%) carried out any form of training activity, compared to a 62% average participation of European enterprises. An in-depth analysis of the data provided by this survey also showed how this could be explained – at least partly – by the peculiarity of the Italian industrial system, based on small and medium-sized enterprises and on so-called traditional sectors (clothing, footwear, wood and furnishings, etc.).

In particular, the existence of smaller-sized enterprises, seemed to be a key factor explaining the low propensity ofenterprises to offer training to their employees: investing in human capital – just as in Research & Development – does not usually produce tangible effects in the short term and requires a strong corporate culture and strategic vision. Factors that tend to discourage smaller firms from investing in human capital include a greater incidence of training costs (not only direct costs but also those deriving from missing workers in the production process), a lesser organisational and logistical capacity, and greater uncertainty with regards to returns of training, especially in a context of growing labour flexibility and mobility. Within this framework, introducing innovative forms of intervention seemed more important than financial incentives to ensure greater effectiveness of policies for continuing training.

To achieve such an objective, new bodies have been created (“Interprofessional Funds for continuing vocational training” - Fondi paritetici interprofessionali per la formazione continua) where, for the first time, social partners would be directly involved in the programming and management of a significant part of the resources allocated to continuing vocational training. The aim was to encourage a greater dissemination of the training culture, especially with regards to smaller-sized enterprises, and contribute to steer financing towards initiatives expected to be more in line with the real requirements of companies.

Interprofessional Funds are now fully operational. Starting from 2004, they have an independent financing channel as well as a significant supply of resources in addition to those allocated in 2003 by the Ministry of Labour and Social Policies for their start-up. Interprofessional Funds are part of the existing system for financing and managing continuing training initiatives (i.e. the European Social Fund as well as the national/regional programmes and funds). The real difference today is the fact that social partners share responsibility for the operational management, whereas previously, the social partners’ role mainly consisted of cooperating with public administrations in order to define intervention strategies and priorities within the framework of concerted actions. They now have actually to manage financial resources, to plan and direct interventions, to organise and reconcile local and sector needs and to monitor the outcomes of activities. The close link between the funds and the enterprises involved acts as a facilitator, and one can reasonably expect a greater and prompter ability to grasp the needs of enterprises.

Also in the case of lifelong learning, the construction of a system for collecting and producing the statistical information needed for monitoring and evaluating the policies implemented is an important related aspect. In the light of our previous explanations, the promotion of lifelong learning involves a set of particularly complex policies, where not only institutional actors but also other players – such as social partners – are directly engaged in the programming and management of considerable financial resources (i.e. not only the Fondi Interprofessionali but also the national funds for the vocational training of temporary workers). As far as continuing vocational training is concerned, the achievement of an integrated monitoring and evaluation system in line with the European benchmarks thus requires significant efforts from the administration.

The example of lifelong learning is undoubtedly emblematic for national policies, but what has happened in recent years at a regional level, for instance during the ESF programming and reprogramming, is equally significant. In addtion, it is also worth mentioning the experience gained at a strictly local/provincial level of programming, which I will briefly illustrate.

The Italian Provinces are required to play a leading role in active labour policies and have also been involved, more or less directly, in the benchmarking system proposed with the EES. A typical example is the experience of Local Action Plans for employment, where many local authorities have set up integrated strategic programming tools for training, education and labour, within the framework of EES guidelines. These local entities have directly measured themselves with the European benchmark when defining active labour policies and, more in general, local development policies.

All these are only brief examples and many more could be presented, but they obviously give a positive picture of the “stimulating” role of the benchmarking system. However, the processes involved, currently and in the past, are very complex and require the participation of a large number of both national and local stakeholders, as well as a firm commitment of the administrations concerned. Therefore, the identification of a significant and feasible common system of statistical indicators represents a strategic goal that seems to be however only partly achieved. In fact, the need to dispose of a wider and better information – in terms of timeliness, comparability, relevance, etc. – is not the only crucial aspect to be dealt with.

3.Indicators, benchmarking and evaluation

From the experience I have summarised previously, we can identify some of the main difficulties, both methodological and technical, linked to the quality of the statistical information system and to the indicators used for the construction of benchmarking. Since data not only constitute the basis for the knowledge of different phenomena, but also play a key role in steering policy makers’ choices, their imperfection is a potential source of serious problems for public decision-makers and could make their work more difficult if not “damaging”.

Of course, benchmarking makes sense insofar as the accuracy of data is really significant in terms of coherence and above all comparability. In particular, when the implicit goal is to evaluate the progress of policies in different countries/regions, it is obviously better to avoid making any comparisons at all than to make “wrong” ones. Without going too much into technical details, I would like to report here a small example concerning lifelong learning.

Until the recent introduction of the new continuous labour force survey, the number of trained employees was estimated with reference to a four-weeks period preceding the interviews. As it is well known to statisticians, such choice could potentially produce biases due to the poorer statistical visibility of short training courses. For countries with short average duration of training courses, the shorter is the reference time of interviews, the lower is the probability to observe the phenomenon over a given observation period. It is of course difficult to know how significant these biases can be in the Italian case, but it has to be mentioned that our country has an average duration of training courses decidedly below the European average.