Type: Original Research Article

Type: – Original Research Article

TITLE: - Reducing bias in incidence estimation due to membership fatigue in voluntary reporting health surveillance schemes

Main Author - Matthew Gittinsa

Co-Authors - Roseanne McNameea, Fiona Hollanda, Lesley-Anne Cartera

a Centre for Biostatistics, School of Health Sciences, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK

Corresponding Author - Matthew Gittins,

Centre for Biostatistics,

University of Manchester,

Jean McFarlane Building,

University Place,

Oxford Road,

Manchester, M13 9PL, UK

Phone – (+44) 0161 275 5664

Fax – (+44) 0161 275 5205

E-mail -

Ethics - Multicentre Research Ethics Committee approval has been granted to THOR (MREC 02/8/72)

Conflict of interest – There are no known conflicts of interest.

ABSTRACT

Objective: Accurate estimation of the true incidence of ill-health is a goal of many surveillance systems. In surveillance schemes including zero reporting to remove ambiguity with non-response, reporter fatigue might increase the likelihood of a false zero case report in turn underestimating the true incidence rate and creating a biased downward trend over time.

Study Design and Setting: Multilevel zero-inflated negative binomial models were fitted to incidence case reports of three surveillance schemes running between 1996 and 2012 in the UK. Estimates of the true annual incidence rates were produced by weighting the reported number of cases by the predicted excess zero rate in addition to the within scheme standard adjustment for response rate and the participation rate.

Results: Time since joining the scheme was associated with the odds of excess zero case reports for most schemes, resulting in weaker calendar trends. Estimated incidence rates (95% C.I) per 100,000 person years, were approximately doubled to 30 (21-39), 137 (116-157), 33 (27-39), when excess zero rate adjustment was applied.

Conclusion: If we accept, that excess zeros are in reality non-response by busy reporters, then usual estimates of incidence are likely to be significantly underestimated and previously thought strong downward trends overestimated.

Key Words -, surveillance, excess zeros, zero-inflated negative binomial, work-related ill- health, voluntary reporting

Running Title – Reporter fatigue in surveillance schemes.

What is new?

- The odds of a reporter responding with a false zero increases as membership time of the scheme increases. Reporters with responses that are designed to be intermittent e.g. 1 per year are more likely to respond with a false zero than a reporter responding continuously throughout the year.

- Accurately estimating the true incidence of ill-health is the goal of many surveillance schemes. Requesting zero reporting helps to remove ambiguity with non-response but health surveillance schemes that request zero case reports are vulnerable to the inclusion of false zero report i.e. a zero case report even if cases have occurred.

- Failure to adequately account for excess zeros at the design or modelling stage in any current or future surveillance scheme can result in underestimation of the true incidence rate and overestimation of any trends in incidence over time.

Acknowledgements – The authors are grateful for the advice and support of all those involved in the THOR project including Professor Raymond Agius, his team at the Centre for Occupational and Environmental Health, and those physicians who report to each individual scheme.

Financial Support –This work and the THOR schemes was supported by the Health and Safety Executive (HSE) and charitable donations. This paper expresses the views of the authors and not necessarily the funding body.

1. Introduction

In 2006, the World Health Organisation (WHO) produced a set of recommendations regarding standards in surveillance of disease and health conditions.(1) This included the presence of zero reporting within the scheme in order to remove any ambiguity between a zero occurrence report and a non-response. One prominent surveillance scheme currently operating within Great Britain, The Health and Occupational Reporting network (THOR),(2) requests a report of new cases regardless of whether any new cases occurred.

THOR is a national occupational health reporting scheme consisting of specially trained occupational health physicians, hospital consultants, or general practitioners (GPs) with some occupational health training who voluntarily report new cases of work-related ill-health (WRIH) on a monthly basis.(2) The case reporting methodology has been explained in more detail elsewhere.(2-6) However briefly, a monthly count of new WRIH cases is reported by physicians designated to be either a ‘core’ or ‘sample’ reporter; where ‘core’ reporters are asked to report every month of the year and ‘sample’ reporters on one randomly chosen month per year. ‘Sample’ reports are then multiplied by 12 and added to the ‘core’ reports in order to estimate the total yearly number of cases reported by all participating reporters. The THOR network in various forms has been active since 1989 with a number of reporters being members for many years. While it is accepted that there may be underestimation of the true incidence these data have been used to estimate the true trends in new cases of WRIH.

Participation in the scheme for extended time periods may cause reporting fatigue especially if an automated or efficient time management system has not been implemented by the reporter. This reporting fatigue may manifest in a non-response or an increase in zero case reports as reporters are asked to submit a response even if zero cases have occurred.

In a paper investigating time trends in the incidence of work-related mental ill-health and musculoskeletal disorder, Carder et al. (2012) noted over time an increase in both non-response and the return of zero case reports in the THOR scheme Surveillance of Occupational Stress and Mental Illness (SOSMI) which might be attributable to reporter fatigue.(7) McNamee et al. (2008) also noted in three THOR schemes a link between reporter membership time and both non-response and zero case reports.(8) To account for increasing fatigue, McNamee estimated calendar time trends from Poisson models adjusted firstly by membership time as a covariate, and secondly by only including reports from within 5 years of first reporting month. Neither method was considered adequate as including two highly correlated time variables resulted in very wide confidence intervals, and restricting the data to within 5 years dramatically reduced the sample size.

Equally, incidence rates may have been overestimated. ‘Sample’ reporters may feel the need to justify their inclusion by possibly harvesting cases occurring in the month or months prior to the designated reporting month. McNamee et al. (2010) attempted to investigate the accuracy of estimating the incidence using a randomised crossover trial of continuous sampling (‘core’) vs time-sampled (`sample’) reporting. Results indicated that over reporting present in the ‘sample’ group may increase the total estimate by 26%, and that rates declined gradually in the ‘core’ group over time.(9) McNamee commented that the 26% increase may equally have been due to under reporting in the ’core’ reporters due to fatigue and excess zeros.

The aim of the work presented here is to look for evidence of excess zero case reports. A zero-inflated regression model explicitly models the probability of a ‘false’ or ‘excess’ zero under the influence of membership time increases, as would be expected if reporter fatigue is present. Estimates of true calendar time trends are generated after accounting for the presence of excess zeros within the data. Finally, model predictions of an excess zero are used to estimate the true WRIH incidence rates corrected for the presence of excess zero case reports along with estimates of the true change over time.

2. Methods

Monthly reports of new cases of WRIH are collected in three THOR schemes under a multilevel structure comprised of the repeated monthly counts (level 1) of newly observed cases per reporter (level 2 clusters), with 12 per year for each ‘core’ and one per year for each ‘sample’ reporter. THOR originally focused on occupational lung disease but has since expanded into specialists reporting networks for specific causes of WRIH such as skin & respiratory problems. A full description of the three schemes and the methodology behind them can be found elsewhere,(2-6) but in brief EPIDERM (Occupational Skin Surveillance) employs consultant dermatologists to report the incidence of occupational skin disease, for example contact dermatitis. Cases of occupational respiratory disease such as occupational asthma are submitted to SWORD (Surveillance of Work-Related and Occupational Respiratory Disease) by specialist chest physicians. As of 1996, some 800 occupational physicians report a range of work-related disease to the OPRA (Occupational Physicians Reporting Activity) scheme. OPRA may include both skin and respiratory cases but also, for example, musculoskeletal, mental ill-health, or infectious diseases. Each scheme also recorded the year and month of report along with first month as a reporter, time as a member of scheme, season, peak holiday season, and bank holiday months.

The schemes then calculate the incidence of WRIH by dividing the total number of new cases occurring by the total working population exposed within a time period, usually annually.(10) To improve the estimate the true incidence of WRIH, THOR multiplies the observed number of new cases reported by 12 for sample reporters, and the reciprocals of the participation rate per scheme of all eligible UK Physicians, and the observed response rate.(11;12) The response rate is calculated from the number actual reports divided by the number of expected reports given the number of participating reporters. The participation rates per scheme were previously estimated to be 65%, 56%, and 72% for EPIDERM, OPRA, and SWORD respectively.(11;13)

2.1 Statistical Analysis

The monthly counts of WRIH are assumed to follow a Negative Binomial (NB) distribution which contains overdispersion. The NB regression model accounts for overdispersion by allowing additional variation in the expected number of cases i.e. the Poisson mean, to vary randomly over a gamma distribution.(14) To accommodate heterogeneity between reporter response clusters a multilevel NB model which includes an additional random effects intercept parameter is used.(15) Here the variation between reporters in the log expected number of cases is assumed to follow a normal distribution with mean zero and constant variance. Though not required here, additional random effects parameters may be added to represent variation associated with the ‘slope’ of the model covariates or additional cluster levels within the data structure.(16)

To account for excess zero case reports within the data a multilevel Zero-Inflated Negative Binomial model (ZINB) also with random effects intercepts is fitted.(17;18) The ZINB is a mixture of two models.(19) A logistic regression models the binary ‘false-zero’ vs ‘true-case’ report where the probability of success is the probability of a ‘false-zero’ occurring, and a NB model represents a conditional distribution of true monthly counts of WRIH including the zero case reports.(20) The zero-inflated model allows for a differing set of predictors and covariates for each portion of the model.(21) To address the study objectives the primary predictors of interest are mean centred membership year in the zero-inflated portion and mean centred calendar time in the negative binomial portion. Covariates were chosen a priori to be peak holiday season for the zero-inflated portion and first month as a reporter and bank holiday months in the negative binomial portion of the model. The multilevel ZINB includes a random effects intercept in each portion of the model such that they are assumed to be independent and normally distributed with mean zero and a unique variance.(18) For comparison the standard NB model is also fitted and the goodness of fit compared using the Akaike Information Criteria (AIC).(22;23) All modelling is repeated for ‘core’ reporters only, ‘sample’ only, and both together with and without an interaction term between reporter type and time. Odds Ratios (OR) illustrate the predictors’ influence on the probability of a false zero event occurring whereas Incidence Rate Ratios (IRR) illustrate the influence of the predictor on incidence rate of WRIH, both of which are expressed on the exponential scale. All analysis is performed in the statistical package R 3.0.2(24) using the package “R-Multilevel ZINB” created in S-Plus by Drs A.H. Lee and K.K.W. Yau(18) and later modified for R by Dr Atkins.(22)

To illustrate the influence of excess zeros on estimates of the true annual incidence of WRIH the incidence rate is reported firstly, in its raw format (i.e. ‘sample’ reports multiplied by 12 only), under the standard THOR adjustment (i.e. additional weighting for participation and response rates), and finally after a further adjustment for excess zeros. As excess zeros are a form of non-response the true incidence can be estimated by treating these reports as under reporting. The inflated portion of the ZINB model estimates the probability of an excess zero (p) being present, and subsequently the probability of the true number of cases (1-p) being reported. To estimate the true incidence when excess zeros are present the observed total number of new cases multiplied by the reciprocal of the probability a true case has been reported. A formula for calculating 95% C.I. associated with the incidence of WRIH is not currently available within count data that has a multilevel structure, has experienced weighted adjustment, and requires a finite population correction. Therefore approximate 95% confidence intervals are provided using complex survey design methodology that uses a first order Taylor series variance estimator.(25)

3. Results

The monthly counts of WRIH reported to each THOR scheme are summarised in full in Table 1 and the distributional properties illustrated in histograms provided in Figure 1. The membership period tended to be shorter and the percentage of zero case reports greater (63%, 51%, and 72% for EPIDERM, OPRA, and SWORD) in the ‘sample’ reporters compared to ‘core’ reporters.

Table 1 – Descriptive statistics of the three THOR schemes.

Figure 1 – Histogram illustrating the distribution of new cases of work-related ill-health reported to each of the THOR schemes by core and sample reporters.

For comparison, Table 2 gives the WRIH incidence rate ratios (IRR) and corresponding 95% Confidence Intervals (95% C.I.) from the standard multilevel negative binomial model split by scheme and the four reporting scenarios; ‘core’ only, ‘sample’ only, ‘core’ and ‘sample’ together, and reporter type interaction with calendar time.

Table 2 – Work-related ill-health incidence rate ratios (IRR) from a standard negative binomial model with random effects, repeated across the three THOR schemes.

Table 3 gives the corresponding IRR (95% C.I.) for the NB portion and OR (95% C.I.) for the inflated portion of the ZINB models. In both cases an increase in calendar year, membership year, or change from a base category (e.g. ‘core’ compared to base ‘sample’ reporter) corresponds to a change in incidence (IRR) of new cases or odds (OR) of a false zero case reported. OR and IRR can also be expressed in terms of percentage changes. For example, if we look in the top panel of Table 3 (EPIDERM), and in the columns headed "Core and Sample", we see an OR of 0.04 against rep type (core). This indicates that EPIDERM "core" reporters are 96% less likely compared to "sample" of producing a false zero. The accompanying confidence interval tells us that this difference is between 92% and 98%. Further down the same column, we see an IRR of 0.98 against "calendar year", and this indicates that the incidence of WRIH reported by "core" and "sample" reporters is decreasing by 2% per year (95% C.I. = 2%,3%). The reduced AIC goodness of fit measure in all ZINB models compared to their corresponding NB model indicates an improvement in model fit has occurred in all schemes when accounting for excess zeros, with the strongest improvements seen in the ‘sample’ reporters. In all cases described the overdispersion parameter is significantly greater than zero and less than 0.2019 (max=1) indicating that overdispersion is present in the data but not due to model misspecification, confirming the negative binomial as the preferred model.

Table 3 – Excess zero odds ratios (OR) and adjusted work-related ill-health incidence rate ratios (IRR) from the zero-inflated negative binomial models with random effects, across the three THOR schemes.