EET/S3/10/23/2

EMBARGOED UNTIL 09H30, 8TH SEPTEMBER 2010

The Economic Evaluation of “Homecoming Scotland 2009”

A Review for the Economy, Energy and Tourism Committee of the Scottish Parliament

August 2010

Summary

About this report

This report has been produced by Dr Geoff Riddington of GRID Economics for the Scottish Parliament’s Economy, Energy and Tourism Committee. This was commissioned by the Committee to provide an independent assessment of the work undertaken by consulting firm, EKOS, on behalf of VisitScotland and EventScotland into the success of the Homecoming Scotland 2009 initiative and also of particular events, such as The Gathering.

An assessment of the research undertaken by EKOS

  • The specific finding of the research by EKOS that there were around 72,000 additional visitors as a result of Homecoming Scotland 2009 (HS09) is accurate and robust.
  • The finding that resulting additional expenditure (including VAT) in Scotlandfrom HS09 was £31.6m is realistic and robust.
  • The additional expenditure associated with HS09 includes the additional expenditure of event organisers made from fees from additional visitors. It does not includeexpenditure from public grants. This, in the author’s view,is appropriate but could be misleading when comparing figures with analyses in which grants are included.
  • In contrast, organisers’ expenditure is included in “The Gathering” review which double counts fees. Consequently, the author’s opinion is that the study conducted by EKOS for VisitScotland and EventScotlandoverestimates the expenditure and output associated with this specific event.
  • The consultant’s methodology for allocatingthe expenditure by a visitor to Scotland for the whole of their visitto Scotland as a result of their attendance for a single day at an HS09 event is logical.However,it is recommended that this expenditure is also subdivided, on an activity day basis, to allow the impact of secondary activities (e.g. visits to historic sites) and regional impacts to be shown.
  • The presentation of a “local” analysis, as for Edinburgh in “The Gathering”analysis is fundamentally misleading if the expenditure is on a whole trip basis,since much of this expenditure may well not have been in the “local” (Edinburgh) area.
  • The reportedpublic sector budget that underpinned the HS09 campaign was £5.5m. However, this figure does not include funding of promotional campaigns by VisitScotland (VS) and VisitBritain (VB), which amounted to an additional £3m. Since this represents an Opportunity Cost (because it diverts attention away from, for example, promotion of wildlife holidays) this should be included in the budget associated with HS09,resulting in a total cost of £8.5m for the public sector for HS09.
  • Including all the written-off costs of loans to The Gathering (£291,500) would have been inappropriate.However a proportion of around £83,000 could logically have been included.

An assessment of the economic modelling undertaken by EKOS

  • An industry standard agreed set of multipliers was used to undertake the impact analysis. Unfortunately,the underlying model used by EKOSto evaluate HS09 is, in the opinion of the author, both out of date and incorrect and substantially overestimates the actual impact. Rather than 1536 jobs generated by HS09the true figure is probably no more than 900 jobs (i.e. a near 70% overestimate has been made in terms of the numbers of jobs said to have been created or supported through the success of HS09).
  • Confusion is likely because of the terminology used:

“Output” or “Additional Expenditure”, is poorly understood and,incorrectly, suggests a direct measure of economic activity

The measure used for evaluation “Output/Cost” has been labelled “Return on Investment” (ROI). This is misleading insuggesting some affinity with the Rate of Return on Investment (ROI) used by investment analysts. This often leads to misleading reporting in the media and elsewhere that HS09 or The Gathering has resulted in a rate of return of £X million for the £Y million of public investment.

  • Impact analysis also forecasts inter alia employment and gross value added. It is suggested that the “cost per job”, the measure traditionally used by development agencies, is both meaningful and often the most appropriate.
  • Because this measure can be easily understood, the limitations of Impact Appraisal are more apparent. The value of employment varies between regions and it is recommended that regional analysis of impact is normally carried out. It is also important to identify the employment that is long lasting and high quality and will be important to Scotland’s future. It is possible to extend the model to provide estimates of employment impacts by industry, gender, FT/PT and value added.

Recommendations for evaluating future events

  • In general,the author’s view is that future decisions should be based on economic value and not impact however measured. Ideally Impact Analysis should be embedded within a Cost Benefit framework, with the value of a project (which covers both short- and long-term) in education, culture, community and physical health set alongside economic health. Ideally, all elements should be valued to provide a single benefit/cost ratio. The other aims of HS09 (celebration, pride, involvement of the Diaspora) would be seen as having an explicit monetary value for the Scottish people.
  • Recognising the difficulties in generating acceptable values it is suggested that as an alternative, a standard appraisal check list, similar to those used in Transport Appraisal, be established.
  • Currently the appraisal of a programme, such as HS09 or The Gathering, is commissioned by the agency responsible for it. Information for future planning thus becomes confused with a judgement on operations. In the author’s view, an independent body such as Audit Scotland may well be more appropriate to conduct such evaluations given that they would have no other policy interest in the success or otherwise of a programme such as HS09.

The Economic Evaluation of “Homecoming Scotland 2009” and “The Gathering”

A Review

1Introduction

In 2009, the Scottish Government, through its tourist agencies VisitScotland and EventScotland, decided to promote a series of linked events under the title Homecoming 2009. The campaign linked social and cultural events, often related to celebrations of the life of Robert Burns, with a “home of golf” sporting theme and a Highland Gathering of the type originally conceived by Walter Scott. In all, over 400 events took up the Homecoming brand.

The economic rationale was based around the generation of extra visitors, particularly affluent long haul passengers from the Scottish Diaspora in North America and Australasia. It was believed that expenditure on promotion and advertising in these areas would generate visitor expenditure significantly larger than the costs. Work commissioned by VisitScotland/EventScotland to identify if that had happened was carried out by EKOS.

Economic Impact Analysis is a forecast of what might happen in the economy if an action is taken. In the case of Homecoming Scotland, it is a forecast of the difference between what actually happened in 2009 and what might have happened had public sector agencies, specifically VisitScotland and EventScotland, not devoted additional expenditure to the event.

Inevitably assumptions have to be made by any person evaluating the programme and it is the validity of the assumptions that are the subject of debate, particularly when they deviate from standard practice. The work produced for the Scottish Parliament’s Economy, Energy and Tourism Committee in this report reviews the methods used and the assumptions made by the independent consultants EKOS in their studies of the Economic Impact of “Homecoming 2009” and “The Gathering”.It then reviews the use of Economic Impact Analysis in decision making and makes recommendations about how it can be used and extended.

2The Main Elements

The methodologyused by EKOS to evaluate HS09 has 4 elements:

  1. Estimation of the number of extra visitors generated by HS09
  2. Estimation of the gross expenditure by these visitorsthat could reasonably be said to result from the investment or activity (e.g. expenditure generated as a result of a new music festival);
  3. Identification of the extent that the expenditure simply displaces expenditure from existing activities/investments (the number of people who would have gone to alternatives) and consequently the net new additional expenditure; and,
  4. Estimation of the effects in the real economy (jobs and income) as the additional expenditure ripples through the economy via increased demands.

In the author’s opinion,the study by EKOS follows this methodology in a highly professional manner, using standard and often conservative assumptions. At each stage the consultants have been anxious to ensure that only the additional expenditure that could unequivocally be associated with the additional costs was considered. In some cases it can be argued that too little was in fact attributed whilst at other times a critic could argue that the assumptions tend to suggest a more substantial impact than is justified. The details of these are now discussed.

3Estimation of Visitors attracted byHomecoming 2009

Three methods were used to estimate visitornumbers resulting from the campaign:

  1. by collecting data directly from each of the events associated directly with Homecoming Scotland 2009 (HS09) and identifying, by survey, those who were there only because of that campaign;
  2. by survey of all tourists to identify the percentage that were there only because of that campaign;
  3. by using an econometric model linking the number of tourists to factors like income, and exchange rates with a dummy variable to represent the additional numbers who were induced by the HS09 campaign.

One difficulty, however, is that HS09 was both a series of events and a marketing campaign. The data in the EKOS report suggests that a significant number of visitors (11,316) appear to have been attracted to Scotlandonly as a result of the marketing campaign and did not then go to any of the 400 plus events associated with that campaign. Conversely there were four times as many(44,789) who were at events only because of HS09 but were not influenced by the marketing campaign. The figures are shown in Table 1.

Table 1: Influence of Campaign v Events for those who only came to Scotland because of HS09

Only Reason for Visiting Scotland / Number / Percent
Events and Marketing of HS09 / 15,829 / 22.0%
Events not Marketing of HS09 / 44,789 / 62.3%
Marketing not Events of HS09 / 11,316 / 15.7%
TOTAL / 71,934 / 100.0%

This surprising finding warranted further investigation and the data sources used by EKOS in its work for VisitScotland/EventScotland are now discussed.

3.1Estimation of Visitor Numbers by Attendance

The first surveyinvolved collecting visitor number data from each of the events in the HS09 programme. It is undoubtedly the most reliable. Where data was unavailable, projections were employed and downsized to allow for “optimism bias”.

The events programme of HS09 attracted a total of 1,782,493 visits. The EKOS study took into account the difference between the number of visitors and the number of visits. Anumber of people visited more than one event; hence the number of visitors is smaller than the number of visits. The visitor survey established that those attending main events such as “The Gathering” attended on average1.4 events in total; whilst those attending partner events such as “The Homecoming Open” attended 1.1 events on average.

A proportion of the visitors came to Scotland and attended the events onlybecause they were in the HS09 programme. EKOS identified from the same visitor survey that, on average,16% of those at funded events and 5% of those at partner events were in Scotland purely as a result of HS09. Applying this proportion to the aggregated audience figures suggests that some 70,882 event visits were as a result ofHS09 only.After applying the number attending two or more events,EKOS estimate that the actual number of “HS09 only” visitors to Scotlandis 60,617.

It should be emphasised that the consultants were very conservative in allocating to “HS09 only visitors” those that they were firmly convinced would not have attended the event had it not been forthe campaign. As such,in the author’s opinion, there is no reason to question the final small percentage of visitors to Scotland in 2009 attributed to HS09 by EKOS.

3.2Estimation by Survey

Ideally a survey would have been conducted of all visitors to Scotland in 2009 to establish the importance of both the HS09 marketing campaign and events programme. Because of the number of visits (15m) and the relatively small numbers associated with HS09 (72,000 or 0.5%) such a survey would have needed to be huge and incredibly expensive to be remotely credible.

A much cheaper alternative utilised the electronic enquiry database held at VisitScotland and the planned internet survey of the Homecoming Campaign.

The database consists of five types of enquiry. By far the largest type consists ofenquirers who in previous years had requested an e-newsletter and consequently were exposed to publicity on the Homecoming campaign of events. The other four were those in the current year that requested newsletters, brochures, and/or those seeking information on the “Expeditions Scotland”and the “Fly your family home” programmes. The numbers are substantial e.g. 177,000 for the US alone. A request to complete the internet survey was issued to 6,000 contacts of each of the 5 enquiry types, chosen at random (30,000 targets). Responses were received from 10%, totalling about3,000.

Within the survey three questions were critical:

a) Have you or will you definitely visit Scotland this year?

b) What is the Main Reason for the visit?

c) Those who had visited were also asked if they had visited a Homecoming Event.

The percentage of enquirers that actually made a visit varied dramatically according to the classification and country. For example, for the US, only 9% of previous enquirers actually visited Scotlandin 2009 whilst 44% of those requesting a brochure made the trip. An estimate thatabout 84,000 family groupson the database visited Scotland in 2009was obtained by grossing the conversion rates by numbers of each type. The average size of the family group of around 2.2 gave an overall estimate of visitor numbers of 219,458.

Of those, some 12.4% identified HS09 as the main reason for their visit to Scotland, equating to 27,145 visitors. Of those, some 60% were identified having attended an HS09 event.

It might be argued that the database held by VisitScotland was not representative of the population of visitors to Scotland and that the 10% who replied to the survey were not representative of the population surveyed. Clearly there will be substantial numbers of visitors in 2009 that made the trip without opting to go onto a marketing database and many of these may well have been influenced by the HS09 campaign. If they subsequently did not go to any events, this would lead to a substantial under-estimate of the effect. Conversely it may well be argued that those most likely to answer a survey request from the Scottish National Tourist Organisation are those who have visited Scotland and consequently the conversion rate is too high and could lead, in turn, to an overestimate of the influence of HS09.

As discussed earlier, any statistically better alternative would have been extraordinarily expensive and therefore, it is the author’s view that whilst this estimate of just over 27,000 visitors to Scotland because of HS09 might not be considered robust it is not unreasonable.

The survey also provided insight into the most critical question, i.e. the number of people who made the trip for emotional or family reasons as a result of the HS09 campaign although they did not attend organised events. This was estimated at 11,316.

3.3Estimation by Statistical Model

The third apparent method of identifying the numbers who came to Scotland specifically for HS09 is to use tourism statistics obtained from the UKNational Tourist Survey (NTS) and the International Passenger Survey. Tourist numbers vary quite substantially depending upon such factors as the state of the national economy, the strength (or otherwise) of the pound against the euro and dollar (amongst others) and expectations about the weather. In 2009, for example, the weakness of the pound against the euro, long range predictions of a “barbeque summer” and a struggling domestic economy led to increased interest in holidaying in the UK both by UK and foreign nationals. In general, we would expect these factors to be similar in Scotland and the Rest of the UK i.e. it might be expected that we could predict, with reasonable accuracy, holiday visits in Scotland from a regression model relating Holiday Visits to Scotland and Holiday Visits to England. Any significant effect of HS09 might be expected to register as additional to the normal forecast based on visits to England.

A statistical analysis of this nature was carried out by the author using such regression models and dummy variables. This found no significant difference for 2009 that could be associated with HS09.However, there are three reasons why we cannot conclude that HS09 had no effect despite this analysis.

Firstly, the most important markets for HS09 were long haul, white English speakers from North America and Australasia. After the collapse of Zoom airlines in 2008, direct flights from these countries to Scotland were limited to Continental Airlines fromNewark, USA and with Globespan from Canada. It is not unreasonable to suppose that a lot of holiday makers passing through London possibly on their only trip to the UK, might spend some time there. Thus, if HS09 “worked”, it probably also generated an increase in visitors to England as well as Scotland.

The second and, in terms of public policy, potentially more important reason is the effect of not promoting other aspects of Scottish Tourism. Concentrating the £3.5m recurrent budget of VisitBritain and VisitScotland on HS09 will have had a negative effect on visitor numbers, particularly from short haul and UK markets. The statistics supply the net effect. It could be argued that because the UK and Europe are numerically far more important than the Diaspora, any diversion of marketing effort will have negative effects overall[1]. Table 2 shows the source of visitors to Scotland

Table 2: Home Country of Scottish Tourists

Visits '000 / %Non Scottish
Diaspora / 674 / 8.0%
Rest of Overseas / 1884 / 22.4%
Total Overseas / 2558
Rest of UK / 5850 / 69.6%
Total Non-Scot / 8408 / 100.0%
Scotland / 6620
Total UK / 12470
Total / 15028

The third and related reason is that the final estimate for visitor numbers as a result of HS09 (71,934) represented only 0.5% of total visits (15m) to Scotland in 2009. For the US, which was the recipient of 54% of the marketing effort, the extra visitors for HS09 would amount to only 1.6% of the normal number from that country. On this evidence it is difficult to argue that the Scottish tourist industry was dependent upon the success of “Homecoming Scotland”.