Epidemiological Cut-off Feasibility Study
June 2017
AUSTRALIAN BUREAU OF STATISTICS

© Commonwealth of Australia 2017

This work is copyright. Apart from any use as permitted under the Copyright Act 1968, no part may be reproduced by any process without prior written permission from the Commonwealth. Requests and inquiries concerning reproduction and rights in this publication should be addressed to The Manager, Intermediary Management, Australian Bureau of Statistics, Locked Bag 10, Belconnen ACT 2616, by telephone (02) 6252 6998, fax (02) 6252 7102, or email: <>.

In all cases the ABS must be acknowledged as the source when reproducing or quoting any part of an ABS publication or other product.

Produced by the Australian Bureau of Statistics.

Inquiries
For further information about these and related statistics, contact .

Epidemiological Cut-off Feasibility Study, 2017

CONTENTS

KEY FINDINGS / 4
BACKGROUND / 5
METHODS / 6
Sensitivity and specificity tables / 6
Epidemiological cut-offs used in this study / 7
Receiver operating characteristics curves and area under the ROC curve / 8
CHARACTERISTICS OF PEOPLE WHO HAVE EVER SERVED IN THE AUSTRALIAN DEFENCE FORCE / 10
Demographic characteristics / 10
Mental health status / 11
RESULTS / 13
Areas under the curve / 13
Epidemiological cut-offs / 18
Optimal screening cut-offs / 18
Optimal epidemiological cut-offs / 22
Applying optimal epidemiological cut-offs across time periods / 27
FURTHER CONSIDERATIONS / 27
Potential methods to improve data reliability / 27
Sampling methods / 28
CONCLUSIONS / 29
SOURCES / 30
1997 National Survey of Mental Health and Wellbeing of Adults / 30
2007 Survey of Mental Health and Wellbeing / 30
2007-08, 2011-12 and 2014-15 National Health Surveys / 31
DATA QUALITY / 32
GLOSSARY / 33
REFERENCES / 36
APPENDIX 1 – TABLES OF RESULTS / 37
APPENDIX 2 – SUMMARY TABLES / 55

KEY FINDINGS

The Epidemiological Cut-off Feasibility Study aims to assess the reliability of the Kessler Psychological Distress Scale (K10) as a predictor of mental disorders in the total Australian population and the population of persons who have ever served in the Australian Defence Force. Analysis was conducted using five ABS population surveys, three of which included information on whether respondents had ever served in the Australian Defence Force.

MENTAL HEALTH

  • In 2007, when age structure is taken into account, 21% of people aged 16-85 years who had ever served in the Australian Defence Force (ADF) had a mental disorder in the previous 12 months, similar to that of the total Australian population of the same age (20%)(2007 Survey of Mental Health and Wellbeing).
  • For persons who had ever served in the ADF, anxiety disorders were the most common disorder type (14.0%), followed by affective disorders (8.3%) and substance use disorders (4.7%). This was similar to rates of anxiety disorders, affective disorders and substance use disordersexperienced in the total Australian population (14.5%, 6.3% and 5.3% respectively (2007 SMHWB).

RELATIONSHIP BETWEEN PSYCHOLOGICAL DISTRESS AND MENTAL DISORDERS

  • For the total Australian population, the 1997 and 2007 SMHWB showed the strongest association between psychological distress as measured by the Kessler Psychological Distress Scale (K10)and mental disorders experienced in the past 30 days.
  • For the total Australian population, K10 has excellent discrimination in predicting the existence of affective disorders, good discrimination for anxiety disorders and fair discrimination for substance use disorders (as assessed through analysis of the 2007 SMHWB).
  • For persons who have ever served in the ADF, K10 has good discrimination in predicting the existence of affective disorders, excellent discrimination for anxiety disorders and fair discrimination for substance use disorders (as assessed through analysis of the 2007 SMHWB).

OPTIMAL SCREENING CUT-OFFS

  • For the total Australian population, the 2007 SMHWB indicates that the optimal screening cut-off for affective disorders is 21, for anxiety disorders 18, and for substance use disorders 17.
  • In comparison, the Mental Health in the Australian Defence Force: 2010 ADF Mental Health Prevalence and Wellbeing Study (MHPWS) found that optimal screening cut-offs for the ADF population were 19 for affective disorders and 17 for anxiety disorders. The higher cut-offs for the total Australian population may imply that for persons showing increased levels of psychological distress, these levels are less likely to be associated with the presence of either affective or anxiety disorders in the general population than the ADF population.
  • For persons who have ever served in the ADF,the 2007 SMHWB indicates that the optimal screening cut-off for combined affective, anxiety and substance use disorders is 18.

OPTIMAL EPIDEMIOLOGICAL CUT-OFFS

  • For the total Australian population and persons who have ever served in the ADF, optimal epidemiological cut-offs derived from K10 scores in each ABS survey provide predicted prevalence rates of mental disorders similar to the actual prevalence rates from the same survey.
  • However, when optimal epidemiological cut-offs determined from one ABS survey were applied to K10 data in other ABS surveys, predicted prevalence rates did not match actual rates from those surveys. This may be due to differences in methods used to measure mental disorders across surveys, as well as changes in reporting patterns by respondents over time.

BACKGROUND

In 2010 the Centre for Traumatic Stress Studies (CTSS) at the University of Adelaide, on behalf of the Department of Defence (Defence) and the Department of Veterans’ Affairs (DVA), undertook the Mental Health in the Australian Defence Force: 2010 Australian Defence Force Mental Health Prevalence and Wellbeing Study (MHPWS), which researched the wellbeing of currently serving Australian Defence Force (ADF) personnel (McFarlane et al 2011).

One aim of the MHPWS was to refine methods for detecting mental health conditions in the ADF population. Participants were screened for Post-Traumatic Stress Disorder (PTSD), psychological distress and alcohol use disorders using three instruments routinely used by the ADF, including the Kessler Psychological Distress Scale (K10). Analysis of the screening indicated comparable trends between self-reported mental health condition prevalence measures compiled by CTSS and the diagnostic interviews conducted(McFarlane et al 2011).

To maximise the number of ADF members who could be identified for early intervention, the MHPWS recommended screening cut-offs for the K10 and other instruments to be employed. The study also recommended the use of epidemiological cut-offs to accurately monitor prevalence of disorders over time(McFarlane et al 2011).

The Epidemiological Cut-off Feasibility Study was undertaken to add to this field of research. Aims of the study include:

  • exploration of existing information from Australian Bureau of Statistics (ABS) health surveys to understand the relationship between levels of psychological distress based on the K10 and prevalence of mental health conditions.
  • assessment of the accuracy of the K10 to determine the likelihood of whether someone in the general population or the population who had ever served in the ADFhad anxiety, affective or substance use disorders.
  • determining whether different screening and epidemiological cut-offs are applicable to the general populationand the population who had ever served in the ADF.
  • assessmentof the feasibility of using epidemiological cut-offs as a proxy measure of prevalence of mental disorders.

METHODS

Results in this study are based on analysis of five ABS population surveys that include the Kessler Psychological Distress Scale (K10) and questions on mental disorders:

  • 1997 National Survey of Mental Health and Wellbeing of Adults
  • 2007 Survey of Mental Health and Wellbeing
  • 2007-08 National Health Survey
  • 2011-12 National Health Survey
  • 2014-15 National Health Survey

Populations included in analysis are the total Australian populationand persons who have ever served in the Australian Defence Force (ADF), disaggregated by sex and selected age groups.

Data for the ADF population are available from the2007 Survey of Mental Health and Wellbeing, 2011-12 National Health Survey and 2014-15 National Health Survey. The question asked to identify this population is ‘Have you ever served in the Australian Defence Force?’.

Mental disorders considered for analysis are:

  • Affective, anxiety and substance use disorders (that is, persons with any of the three conditions)
  • Affective and anxiety disorders (persons with any of the two conditions)
  • Affective disorders
  • Anxiety disorders
  • Substance use disorders

The particular types of disorders included in these categories are listed in detail in the Sources section.

SENSITIVITY AND SPECIFICITYTABLES

Epidemiological cut-offs are a statistical tool used to make judgements about the likelihood of a person having a particular characteristic depending on another characteristic (Altman Bland 1994). In this study, cut-offs provide an assessment of whether persons have or do not have mental disorders based on their K10 scores.Typically, for persons with K10 scores equal to or above a particular epidemiological cut-off this may indicate the existence of a mental health condition, while for persons with scores below the cut-off this may indicate the absence of a condition. While some people with scores equal to or above the cut-off will not have a condition, and others with scores below the cut-off will, the epidemiological cut-off methodology minimises these errors.

Epidemiological cut-offs were determined using sensitivity and specificity tables, which grouppersons in a populationaccording to their K10 score and by whether or not they have a mental health condition.Sensitivity and specificity tables were created for the mental disorders mentioned above, sex and selected age groups.

In this study, sensitivity refers to the ability of a person’s K10 score to correctly identify whether a person has a mental disorder, while specificity refers to the ability of K10 to correctly identify whether a person does not have a mental disorder. A standard epidemiological method, referred to as ‘epidemiological cut-off 1’ in this study (see below), is to take the point at which specificity exceeds sensitivity to be the epidemiological cut-off (that is, the K10 score at which specificity exceeds sensitivity) (Andrews and Slade 2001).


In Figure 1below using 2014-15 NHS data, the point at which specificity exceeds sensitivity is a K10 score of 16.

EPIDEMIOLOGICAL CUT-OFFS USED IN THIS STUDY

In this study, three different criteria were used to determine cut-offs from ABS health surveys.

‘Epidemiological cut-off 1’as described in Figure 1 above refers to the K10 score at which specificity exceeds sensitivity; that is, the score at which the proportion of people who are correctly identified as not having a mental health disorder according to their K10 score exceeds the proportion of people who are correctly identified as having a mental health disorder according to their K10 score.

The ‘optimal screening cut-off’ replicates the method used in the Mental Health in the Australian Defence Force: 2010 ADF Mental Health Prevalence and Wellbeing Study (MHPWS), in which the cut-off is the K10 score that maximises the sum of the sensitivity and specificity (that is, the proportion of people who, according to their K10 score, are correctly identified as having or not having a mental health disorder respectively). This method was recommended for use by the MHPWS to identify individuals in the Australian Defence Force whomight need care, and was designed to be more inclusive for use in screening(McFarlane et al 2011).

The ‘optimal epidemiological cut-off’ also replicates the method in the MHPWS in which the cut-off is the K10 score that brings the numberof false positives (persons for whom their K10 score incorrectly indicates they have a mental health disorder) and false negatives (persons for whom their K10 score incorrectly indicates they do not have a disorder) closest together, thereby counterbalancingthese sources of error. This method was recommended for use by the MHPWS in studying trends of prevalence of mental disorders in the Australian Defence Force(McFarlane et al 2011).

In this study, epidemiological cut-offs derived from the 1997 SMHWB are based on mental disorders with symptoms experienced in the last four weeks while epidemiological cut-offs from the 2007 SMHWB are based on mental disorders with symptoms experienced in the last 30 days. Epidemiological cut-offs from the National Health Survey are based on current and long-term mental disorders.‘Mental health conditions’ and ‘mental disorders’ are used interchangeably in this report.

RECEIVER OPERATING CHARACTERISTICS CURVES AND AREA UNDER THE CURVE

Receiver Operating Characteristics (ROC) curves and the Area Under the Curve (AUC) are used in this study to provide a measure of the strength of the relationship between K10 scores and the prevalence of mental disorders in a selected population.

ROC curves plot true positives (that is, persons for whom their K10 score correctly indicates they have a mental disorder) against false positives(persons for whom their K10 score incorrectly indicates they have a mental disorder). The greater the area under the curve, the better K10 is at discriminating between people with and without mental disorders (Sunderland et al 2011).

In general, areas under the curve of 0.90-1.00 indicate K10 has excellent discrimination in predicting the existence of mental disorders,0.80-0.90 indicates good discrimination, 0.70-0.80 indicates fair discrimination, and 0.60-0.70 indicates poor discrimination. An area under the curve of 0.50-0.60 indicates there is no relationship between K10 and mental disorders.

In Figure 2 below,the area under the curve is 0.82,indicating K10 has good discrimination when predicting the existence of affective, anxiety or substance use disorders (combined) for persons aged 18-85 years. The dotted line represents an area under the curve of 0.50 for comparison.


CHARACTERISTICS OF PEOPLE WHO HAVE EVER SERVED IN THE AUSTRALIAN DEFENCE FORCE

A number of characteristics of people who have ever served in the Australian Defence Force are presented in this section to provide context to the study results.Results are taken from the 2007 Survey of Mental Health and Wellbeing unless otherwise stated.See Appendix 2 for summary tables.

DEMOGRAPHIC CHARACTERISTICS

In 2007 there were 837,000 persons aged 16-85 years who had ever served in the Australian Defence Force (ADF), representing 5% of the total Australian population of the same age. Of these, 85% were male and 15% were female. In comparison there were 672,900 persons in 2011-12 aged 18-85 years who had ever served in the ADF (2011-12 National Health Survey), and 678,000 persons in 2014-15 of the same age (2014-15 National Health Survey).

The age structure of persons who had ever served in the ADF was considerably older than the Australian population aged 16-85 years (with a median age of 59 years compared with 44 years respectively). Around 38% of persons who had ever served in the ADF were aged 65-85 years in 2007, compared with 15% of the total Australian population of the same age.


In 2007, of persons who had ever served in the ADF, around half (45%) were not in the labour force while 7% were in defence industry employment and 48% worked in other industries. Less than 1% were unemployed.

MENTAL HEALTH STATUS

The 2007 SMHWB collected information on prevalence of mental disorders comprising affective, anxiety and substance use disorders, as well as levels of psychological distress.Note that data for 2007 presented in this section refer to mental disorders experienced in the previous 12months, while epidemiological cut-offs for 2007 presented in the Results chapters refer to mental disorders experienced in the previous 30 days.

In 2007, 16% of people aged 16-85 years who had ever served in the Australian Defence Force had experienced a mental disorder in the previous 12 months. When age structure is taken into account, 21% of people aged 16-85 years who had ever served in the Australian Defence Force had experienced a mental disorder in the previous 12 months, similar to that of the total Australian population of the same age (20%).

Age and sex standardised results from the Mental Health in the Australian Defence Force: 2010 ADF Mental Health Prevalence and Wellbeing Study (MHPWS) found that 22% of the ADF population(that is, persons currently serving in the Australian Defence Force, excluding trainees or reservists) had experienced a mental disorder in the previous 12 months, similar to that of the Australian population(McFarlane et al 2011).

Similar to the total population, in 2007 rates of mental disorders amongst people who had ever served in the Australian Defence Force were highest amongst young age groups and lowest at older ages.


Amongst persons who had ever served in the Australian Defence Force, anxiety disorders were the most prevalent (12.1%), followed by affective disorders (5.9%) and substance use disorders (3.1%). This was similar to rates of anxiety disorders and affective disorders experienced in the total Australian population (14.4% and 6.2% respectively), and lower than rates of substance use in the Australian population (5.1%) (2007 SMHWB).

Taking age structure into account, the prevalence of each disorder was similar for persons who had ever served in the Australian Defence Force andthe total Australian population: anxiety disorders (14.0% and 14.5% respectively), affective disorders (8.3% and 6.3%; note this difference was not statistically significant) and substance use disorders (4.7% and 5.3%).


In 2007, 9.7% of persons who had ever served in the Australian Defence Force experienced high or very high levels of psychological distress (K10 scores of 22-50). A further 21% experienced moderate levelsof distress(K10 scores of 16-21) while70%experienced low levels (K10 scores of 10-15).


When age structure is taken into account,around 11.1% of persons who had ever served in the Australian Defence Force experienced high or very high levels of psychological distress, similar to that of the total Australian population (9.5%).