Evaluating the Effectiveness of the National Special Needs

1

Evaluating the Effectiveness of the National Special Needs

Information System in Scotland

Executive Summary

November 2004

Lead Grant-holder: Dr Z Dunhill

This Project was funded by NHS Quality Improvement Scotland

Researcher: Anne Wood

Report re-drafted by Emma Stevens (ISD Scotland)

NHS Quality Improvement Scotland

Edinburgh Office

Elliott House

8-10 Hillside Crescent

Edinburgh

EH7 5EA

Copies of this report and other publications produced by NHS Quality Improvement Scotland are available on the NHS Quality Improvement Scotland website.

Evaluating the Effectiveness of the National Special Needs Information System in Scotland

1

Introduction

Information technology in theNHS plays an important part in the maintenance, planning and monitoring of the progress of care for children in Scotland. Improvements in technology, coupled with the formation of the Child Health Information Systems Project Board, fuelled the decision to develop a national system, specifically designed to record data about children with special needs. A focused working group of paediatricians therefore developed the national Special Needs System (SNS), building on a project funded by the Chief Scientist Office (Dunhill & Zealley, 1991). This group consisted of representatives of senior clinicians working in the area of disability across Scotland. At the time of this study, eight Health Boards across Scotland were using or implementing the SNS.

Delays in the publication of this evaluation are unfortunate, but do not mean that its findings are no longer relevant or useful. Since the time of the study, Boards using SNS have generally successfully progressed their use of the system – registering growing numbers of children and, importantly, capturing them at an increasingly young age. Several Boards have suffered from a lack of administrative support or from a lack of available clinical input, but personnel are keen to overcome these problems and continue using the system. Also, another NHS Board (Fife) is now implementing SNS (autumn 2004).

As well as changes within the Boards using SNS, advances in policy and related strategies have introduced new ‘stakeholders’ with an interest in the findings of such an evaluation. These new stakeholders include the Maternal & Child Health Strategy Group (directed from the Scottish Executive and working to produce an over-arching strategy for children’s e-health) and the Hall 4 Implementation Group (also directed from the Scottish Executive and working to implement changes to child health surveillance in accordance with recommendations from the Hall 4 “Health for all Children” report) – as well as bodies at a local level who are interested in integrating data, such as some of the eCare demonstration sites and the West Lothian Changing Children’s Service Project.

Aims

The aim of this evaluation is to assess the effectiveness of the recording of information about children with special needs in Scotland. Consideration was given not only to SNS, but also to any locally devised special needs registers - whatever their format.

Methods

In order to successfully undertake the evaluation, site visits were arranged during spring 2002 to NHS Boards throughout Scotland. During the visits, discussions took place with clinicians and administrative staff using both SNS and other local systems. The importance of this was to gather qualitative information about the operational processes and current recording practices used across Scotland.

Two qualitative questionnaires were developed as part of the evaluation. These were circulated during summer 2002. One was distributed to clinicians and administrative staff that use special needs systems on a daily basis. The other was sent to Allied Healthcare Professionals (AHP), who should be receiving regular outputs from these systems.

Focus group discussions with parents of special needs children were also carried out during summer 2003. The groups were organised by a trained facilitator and parents were accessed through voluntary groups and charitable organisations across Scotland. The facilitator concentrated on issues such as the feedback received by parents from professionals, information sharing between professionals dealing with different aspects of their child’s care, co-ordination of their child’s care and general communication.

Additionally, analysis of data held on children with special needs was undertaken. Data extracts were received from those NHS Boards that use SNS in April 2003. Unfortunately, due to the limited capabilities of the non-user systems, it was not possible to obtain extracts of information from other systems. Local systems either did not electronically record the data required or could not extract data suitable for analysis from their electronic systems. Analysis was therefore only carried out on data from the NHS Boards that use SNS.

The analyses undertaken include comparisons with data from other sources to assess the completeness, consistency and ascertainment of SNS data recording. Specific analyses concentrate on five indicator conditions: Autistic Spectrum Disorders (ASD), Down’s Syndrome, Neural Tube Defects, Sensorineural Hearing Loss and Cerebral Palsy.

Summary of Results

Overall, the results of this evaluation indicated that the effectiveness of the Special Needs System was not uniform.

At the time of the study, there were eight NHS Boards using SNS, all of which were at different stages of implementation and/or utilisation. This study identified significant differences in the completeness and consistency of the data recorded in the SNS, dependent on the data item (for example: condition or disability) and on the NHS Health Board being considered.

Responses to qualitative questionnaires and discussions during site visits highlighted that areas not using SNS had significantly different recording practices, and tended to collect less detailed information. Unlike the SNS, none of the non-user boards used the integral Read code browser or ICD classification system for coding diagnoses; none had a detailed section on the disability characteristics of a child; and none had a ‘shared clinical notes’ section, where (with parental consent) information could be distributed to other professionals involved with the child. Some Boards did not electronically record any information about children with special needs. Importantly, none of the non-user Boards were able to supply data for analysis comparable to that carried out on SNS data.

The relative complexity of SNS did however mean that some users found the system more difficult and time-consuming to use – and reported that they therefore omit to complete parts of the forms. The site visits to user Boards and questionnaire responses from SNS users revealed mixed levels of enthusiasm towards current use of the system. This appeared to stem from variations in the training and knowledge of SNS and also in the communication between professionals using the system. Users of SNS were however predominantly positive about its future use in their NHS Board area.

Whether or not the SNS was capturing a reasonable proportion of children with special needs also seemed to be variable at the time of the study. It was estimated by the Joseph Rowntree Foundation (Stalker, 2000; that approximately two percent of children in any given child health population should be eligible to be registered on a special needs system. The results of analysis of SNS data however revealed that the majority (5 of 8) of the NHS Boards using SNS have significantly lower figures than this estimate. Rates of indicator conditions recorded in SNS compared with estimates of national prevalences were also variable, but again it appeared that at the time of the analysis SNS was generally under-recording. However, this could be explained in part by the fact that each NHS Board was at a different stage of implementation and/or utilisation. This appears to have had a direct impact on the numbers of children registered on the system at the time of this study.

The analysis of SNS data also revealed that the completeness and consistency of the data recorded on SNS varied significantly across and between NHS Boards. For example, essential child details such as their surname, sex and postcode were well recorded (with a minimum of 95% completeness), but the family details section was poorly recorded (as low as 10% complete in one NHS Board area). Variations in the completeness and consistency of data entered by different types of Healthcare Professionals were also observed across all of the NHS Board areas, perhaps related to differing clinical practice for particular conditions: significant differences in the rates of particular types of conditions and/or disabilities were noted.

Finally, the focus group discussions revealed that participants of focus groups in all areas had similar concerns and were disgruntled about similar issues. These parents / carers talked about their need for more information at diagnosis and about care or treatment thereafter. Additionally, they called for better co-ordination of care and improved information sharing between health professionals. These findings are intended to inform improvements in care and support for special needs children and their families. They cannot be taken as direct evidence as to whether or not children registered in SNS are better followed up than in non-user areas.

Discussion: The Effectiveness of SNS

Variations in the training and knowledge of SNS and in the communication between different types of users undoubtedly have a negative impact on the consistency and completeness of recording in SNS. However, despite the difficulties observed, SNS is the most advanced and structured system that has ever been available for recording of data on children with special needs across Scotland. Credit should be given to those NHS Boards who have persevered with local implementation and utilisation difficulties.

In spite of difficulties obtaining information about locally derived systems, it was evident during discussions at site visits and from questionnaire responses that SNS contains substantially more detailed and potentially useful information than the locally derived systems. The fact that a number of NHS Boards are using the SNS means that problems can be easily shared and speedily resolved and that analyses between NHS Boards can be carried out. Also, the very fact that non-user Boards were unable to supply data for analysis is a clear indication of the limitations of other local systems.

The apparent under-recording of some types of children in SNS should not be seen as an indication that the system has no use for estimating prevalence. The ‘Capture – Recapture Methodology’ is increasingly recognised as an accurate and efficient way to estimate prevalence. According to the underlying theory, a single system should not be expected to capture all relevant children, but should be used in combination with other sources to ensure that all children are identified (Hook et al, 1995). It has been applied to a number of different health and social situations, including the epidemiology of autism (Harrison et al, 2005 (in press)).

All NHS Boards in Scotland using SNS should continue to collaborate through the National User Group to further tailor the system toward meeting national priorities for data on performance, trends and outcomes. This may also encourage non-user Boards to implement the system, or at the very least, have an information system in place that is similarly capable of tracking children in the community with special needs.

Conclusions and Recommendations

The results of this study indicate a number of ways in which the SNS system could be improved and also aspects of local systems that could be improved. Recommendations are intended to inform and guide the SNS National User Group regarding further development and improvement of the system nationally and to highlight to a variety of audiences the need for accurate and consistent capture across all NHS Scotland Boards of information related to children with special needs. They should also be of interest to stakeholders such as the Maternal & Child Health Strategy Group.

Summary of Conclusions

• The overall effectiveness of SNS:

(i) The effectiveness of the SNS system was not uniform, both across and within the NHS Boards using it.

(ii) SNS is the most advanced system available nationally to users in Scotland for electronic recording of information on children with special needs.

(iii) Systems other than SNS were unable to supply data for analysis – a clear indication of their limitations.

(iv) The attention and collaboration of members of the SNS National User Group has been crucial to the development and effective use of the system.

• Data quality: Analysis of SNS data revealed that incomplete data did not appear to be random. Also, the accuracy of data held in SNSis unknown.

• The SNS dataset: Some users found filling in SNS forms time-consuming and complicated.

• Implementation of SNS: Some users of SNS had found implementation of the system difficult.

• Effective use of SNS: SNS was not always being used to its full potential and some types of clinicians were not feeding into the system.

• Reports on SNS data: Regular national reports on data held in SNS were not produced.

• Recognising the needs of parents and families: Information about family and social issues surrounding a child’s support more often need to be taken into account and shared.

• Access to information for parents and carers: Parents felt they were not supplied with enough information about their child’s condition and needs.

• Low ascertainment: SNS recording of children with particular conditions and / or disabilities was disappointingly low – and reasons for this were unclear.

• Coding of conditions: Identification / coding of conditions was sometimes inconsistent.

Summary of Recommendations

• The overall effectiveness of SNS: The SNS National User Group should continue to play a central role in the use and development of the system.

• Data quality: The importance of regular and consistent data entry should be stressed to users of SNS. A validity audit, comparing data held in the electronic system with information written on paper forms, might give useful insight into the accuracy of data held in SNS.

• The SNS dataset: Streamlining the SNS dataset and populating parts of SNS with data from other Child Health Information systems would reduce the complexity of forms and the time taken to complete them. Information about the streamlined dataset should also be made available to non-user Boards in an attempt to achieve consistent capture of information across all Boards.

• Implementation of SNS: The experience of SNS users in the National User Group should be called upon to complement documented guidelines for implementation.

• Effective use of SNS: Users should be made fully aware of the capabilities of SNS and the National User Group should consider making SNS available to a wider range of clinicians.

• Reports on SNS data: Regular national reports should be produced centrally (by ISD for example). Dissemination of these would also help to raise the profile of SNS.

• Recognising the needs of parents and carers: Clinicians should be encouraged to more often record information about the wider social context of a child and their family – and, with parental consent, share this information with other professionals involved in the child’s care.

• Access to information for parents: It should be standard practice for parents to be supplied with appropriate information about their child’s condition and needs at the time of diagnosis and thereafter. Partnerships with charitable organisations might benefit this.

• Low ascertainment: Further investigation into the factors influencing the recording rates of particular conditions and disabilities in SNS should be undertaken. Also, collaboration with other organisations (for example, specialist charities) might help to improve registration rates of children with particular conditions.

• Coding of conditions: The SNS National User Group should discuss the applicability of using standard coding in SNS. Users of other local systems should also be made aware of the importance of consistent coding.

REFERENCES

Dunhill, Z. Zealley, H. Donaghy, M. (1991) Evaluation of a revised systematic approach to the assessment, registration and monitoring of children with special needs. Chief Scientists Office (CSO).

Harrison M K, O'Hare A E, Campbell H, Adamson A, McNeilage J. (2005). Prevalence of Autistic Spectrum Disorders in Lothian, Scotland and estimate using the capture-recapture technique. Archives of Disease in Childhood. In press.

Hook EB, Regal RR. (1995). Capture-Recapture Methods in Epidemiology: Methods and Limitations. Epidemiol Rev 17; 243-64.

Stalker, K. (2000) Supporting disabled children and their families in Scotland: a review of policy research. York: Joseph Rowntree Foundation / Findings. Available from

Evaluating the Effectiveness of the National Special Needs Information System in Scotland