SHS Survey Review 2010

TTSAC 2010 – 28th April 2010 Paper 2

Purpose

This paper presents key findings from the ongoing SHS Survey Review 2010 project. The key themes coming out of the review so far are identified along with a summary of the key issues and discussion points going forward.

Background

The SHS Survey Review 2010 project has been set up to assess the operation of, and uses made, of the survey with the primary aim of assessing the achieved benefits, quality and continued need for the survey and to maximise the benefits through appropriate question and topic content.

The review focuses on:

·  The purpose of the survey

·  Alignment with broader analytical work within and beyond Scottish Government

·  Topic coverage

·  Survey design

·  Project ownership and management

The core key stages of the review undertaken so far include:

·  Interviews with key stakeholders (such as SHS Lead Analysts)

·  A web-based survey (targeting respondents within and outwith Scottish Government)

·  Focus groups (with participants outwith Scottish Government)

·  A consultation exercise targeting policy stakeholders within Scottish Government

Summary of Key Themes

The SHS Survey Review 2010 project has highlighted a number of key issues, with the key message coming out so far being that the SHS is good but could be better. It is recognised as being a survey with considerable potential which at the moment is not being fully harnessed.

In summary, the key issues around the current design and deliverables are:

1.  The SHS is seen as a valuable and useful resource within and outwith Scottish Government.

2.  Stakeholders draw upon the whole range of topics covered by the survey and the data supports a variety of uses.

3.  The impact of SHS data on Scottish Government policy varies.

4.  The data provided is considered to be of a high quality.

5.  A large proportion of users undertake their own research alongside SHS data

6.  If the survey ceased to provide data there would be an expectation on the Scottish Government to meet the resulting data gaps.

7.  The survey meets the needs for many stakeholders from a range of backgrounds but could do so more effectively.

Whilst some of the key issues going forward are:

8.  The strategic governance of the survey needs to be improved.

9.  The survey is not fully aligned with the priorities of Scottish Government articulated within the National Performance Framework.

10. The current design of SHS struggles to meet the multiple demands for data and analyses across a wide range of stakeholders.

11. The demands for additional content cover a disparate breadth of topics.

12. There are demands for changes to be made to the survey design to address issues including the timeliness of data, the level of detail provided and the geographies of the data.

13. There is scope for the survey to adopt a core and modular design.

14. There is a need to explore alternative mechanisms for short term social survey data provision.

15. There is a need to better engage with survey users particularly outwith Scottish Government.

For discussion

·  What are the implications of these findings for future options for the survey?

·  Should the survey maintain a distinctive identity or align itself to the changing priorities of government?

·  How can transport statistics requirements best be delivered?

·  How could transport requirements best be serviced through a core and modular design?

·  General discussions and enquiries welcomed.

The Key Themes of the Review

A.  The SHS is seen as a valuable and useful resource within and outwith Scottish Government.

The survey clearly plays a significant role for many and the range of stakeholders reflects the breadth of the topics that the survey covers. Three out of four respondents to the web-based survey identified the SHS as useful and a very low proportion identified the SHS as not useful (totalling 2%). In some instances the SHS represented the primary data source for Scottish Government analysts. Further, the Local Outcome Indicators Project identified the SHS as the source of 19 (out of the 60) indicators used in Single Outcome Agreements. The role of the SHS as a support to local authorities was reinforced by direct responses from authorities and through links with Scottish Government staff.

If the SHS was disbanded it would result in a substantial information gap for a number of topic areas and would be considered a great loss across a range of users. Importantly, if the survey was stopped there would be an expectation on the Scottish Government to meet the resulting data needs of stakeholders.

B.  Data provided by the survey is of a high standard.

The survey is seen as providing robust and reliable data for Scotland that covers a range of issues and topics. The ability of the survey to provide data at local authority level is also considered valuable. However, there can be problems with this given the low numbers involved in certain categories within the survey. Similarly, although the survey is seen to provide up-to-date information there were some demands to improve the timeliness of the data particularly amongst some local authority users.

C.  Use of the survey varies.

The data provided by the SHS has a range of uses including monitoring trends, supporting and evaluating policy, allowing comparisons (local authority and international) and supporting additional analyses. However, the web based survey conducted as part of this review suggests that the SHS is most commonly used infrequently as a source of background information and to raise general awareness. This is likely to reflect the broad focus and remit of the survey.

The use of the survey also varies within Scottish Government with some SHS lead analysts dependent on the data whilst others rarely use it, if at all. The use of the survey is affected by the availability of alternative data sources, alignment with the National Performance Framework and the ability of the survey to provide data that meets user needs. Based on this there is a need to consider the appropriateness of removing content duplicated by other sources (particularly if the duplication is with a Scottish Government source); and in policy areas where use is limited, either modifying the content to encourage use or removing it altogether if the costs of change are deemed excessive.

D.  The impact of SHS data upon policy varies.

There are a number of examples of Scottish Government policies that have been supported by SHS data. However, the use of survey data to inform policy varies between policy divisions. In teams such as transport, finance and community safety SHS data is used frequently and identified as a key information source whilst in areas including housing, internet, local services and regeneration use is infrequent. SHS lead analysts were often unable to provide examples of policies supported by SHS data. In response, it is suggested that lead analysts should be regularly required to demonstrate the impact of SHS data in their policy area.

E.  The strategic governance of the SHS needs to be improved.

The current governance structures of the survey contribute towards some of the problems identified by the review so far. Whilst the SHS team are able to effectively run the survey day-to-day their workload leaves little time for strategic consideration of the survey. This is exacerbated by the high turnover of both SHS project team staff and of SHS lead analysts. The frequent turnover of staff is considered to promote inertia within the survey and restricts utility maximisation of the survey data. There is commonly a lack of awareness of the survey in a broader sense, limited use of the survey beyond the area for which a lead analyst is directly responsible and limited collaboration between analysts. There can also be reluctance to modify questions within the survey if they are not meeting the needs of an analyst which can be explained by a desire to maintain time series data but also reflects an assumption that a question is phrased in a particular way for a historical reason unknown to the analyst. The biennial questionnaire review process is also considered frustrating internally by lead analysts and unclear externally by other stakeholders. This is particularly problematic given the demands for content to be added to the survey.

There can also be uncertainty regarding the division of responsibilities between the SHS team, contractors and lead analysts.

F.  The current design of the SHS struggles to meet the multiple demands for data and analyses across a wide range of stakeholders.

Despite the high overall levels of satisfaction with the survey there is considerable demand for additional content to be added to the survey to meet the specific needs of stakeholders. These demands are from stakeholders within and outwith Scottish Government and cover a huge range of topic areas. As a result there is greater demand for the inclusion of questions than the current provision of the survey can accommodate. Some of these demands would require the introduction of new questions or the re-phrasing of existing questions which would take the survey beyond a 45 minute running time; others would require revisions to the way the survey is carried out for example in terms of sample sizes and who responds to particular questions.

A single survey is unlikely to be able to meet all of the data and analysis requirements of stakeholders. As such there is a need to manage these demands and prioritise survey content. Although some of the revised content would potentially benefit a range of users some of the demands are likely to be too niche to be of value to a wider audience.

Consideration could be given to expanding the duration of the survey beyond a 45 minute running time to increase the volume of content. Further, to free up space within the survey it is suggested that current content which is not extensively used is removed. Similarly, content that may be better suited to alternative Scottish Government surveys (such as the Scottish Health Survey) could be moved to make more space available within the SHS. In making these decisions it would be important to consider the potential cross-cutting value of data or whether small changes to questions would increase use. Discussions with lead analysts suggest that at present, cross-cutting use of data is limited within Scottish Government. However, the SHS team respond to a number of cross-cutting ad hoc requests and the potential to carry out such analyses is identified as an important strength of the survey.

G.  The survey is not fully aligned with the priorities of Scottish Government articulated within the National Performance Framework.

Since 2007, the priorities of the government have been expressed through a National Performance Framework. Historically, the SHS has concentrated on supporting the work of the Scottish Government’s transport, local government and communities policy areas. Over time the topics covered by the survey has widened to incorporate other data needs. If the survey were to better reflect the priorities identified by the National Performance Framework there would need to be a substantial shift in the balance of survey content with areas such as the environment expanded with transport, communities and local government content reduced. However, this is likely to be a problem given the restricted survey running time and the already high levels of demands for revised and additional content. This again highlights the need to identify the key intended users of the survey and whether the survey should maintain the existing core topics (communities, local government and transport) or continue to expand its content coverage.

H.  The survey has the potential to respond to changes in the government landscape.

The survey is seen to serve the needs of various stakeholders. To some extent this reflects the capacity of the survey to respond, at least in part, to the changing governance landscape seen in developments such as the shift towards the National Performance Framework and the concordat between Scottish Government and local authorities. The survey also has the potential to respond to the emerging and developing governance needs in the future. For example, within Scottish Government, SHS data has a potential to meet the data needs arising from Equality Impact Assessments (EqIA’s)[1]. However, it is likely that survey content would need to be regularly modified to respond to such developments.

I.  There is a need to identify who the primary users of the survey should be.

The considerable pressure on the survey to respond to disparate users needs coupled with the diversity of the users themselves make it highly unlikely that a single survey design will be able to meet the needs of all users. As a result there is a need to identify who the primary users of survey are intended to be. This incorporates the range of stakeholders within Scottish Government (divisions, professions etc…) and externally (such as local authorities, the voluntary and community sector, NDPB’s etc…) who currently use the survey. The resolution of this question will have clear implications when considering appropriate survey content and design for the future.