APPENDIX A

Detailed description of the method section

To develop a quality indicator set for RA disease course monitoring, we applied a systematic approach: the RAND modified Delphi method. The procedure is presented in figure 1 and consisted of the following five steps: 1) search for indicators and recommendations about disease course monitoring, 2) first questionnaire round, 3) consensus meeting, 4) second questionnaire round and 5) draw up the final set.

Step 1 Search for indicators and recommendations about RA disease course monitoring

We based our study on a systematic literature search of papers and published (inter)national guidelines including the, at that time preliminary, RA guideline ofThe Dutch Institute for Healthcare Improvement (CBO). It was sought for studies that examined the influence of RA disease course monitoring on patients’ outcome, for guidelines, for opinions from experts regarding RA disease course monitoring, and for studies in which indicators for disease course monitoring were developed. The literature search was performed in Pubmed using search terms including quality indicators, guidelines, monitoring and rheumatoid arthritis (see Box 1). Two researchers independently included or excluded every publication based on the abstract (JF or MH, LvH). Articles were in- or excluded on the basis of the exclusion-criteria presented in table 1. Discrepancy between the two researchers was discussed until consensus was reached.

Included articles and guidelines were searched for recommendations or indicators for RA disease course monitoring. Two researchers (LvH and MH), independently selected recommendations and indicators aboutRA disease course monitoring. Any discrepancy was discussed until consensus was achieved.

For all selected indicators and recommendations the level of supporting evidence (A-D) was determined.Grade A1 recommendations and indicators were directly included. Grade A2, B, C, and D recommendations and indicators were further processed in the first written questionnaire round.

Evidence
level / Definition / Example of a study providing
the specified level of evidence
A1 / A good systematic review of studies designed to answer the question of interest / Systematic review of randomised, controlled trials
A2 / One or more rigorous studies designed to answer the question but not formally combined / Randomised, controlled trial
B / One or more prospective clinical studies that illuminate but do not rigorously answer the question / Prospective cohort study; underpowered or poor quality randomised, controlled trial; nonrandomised, controlled trial
C / One or more retrospective clinical studies that illuminate but do not rigorously answer the question / Audit or retrospective case-control study
D / Formal combination of expert views or other information / Delphi study, expert opinion; informal consensus

Step 2. First questionnaire round

Selected recommendations and indicators were translated into QIs by describing who, should do what, to whom and when. These (potential) indicators were included in the questionnaire. Rheumatologists who were involved in the development of the Dutch RA guideline (n= 7) and rheumatologists from the quality committee of the Dutch Rheumatology Association (NVR) (n=6) were asked to be a member of an expert panel.They were asked to rate potential indicators for the extent in which the description adequately reflected the quality of RA disease course monitoring on a 1-9 Likert scale (score 1=not at all to 9 = excellent). In figure 2 an example is given. The experts were also asked to make adjustments if necessary. Indicators with a median score lower or equal to 3 were not discussed and directly excluded if there was ’no disagreement’ (<33% in the opposite highest 3-point region). Indicators with a score between 4 and 6 in the first round were discussed during the consensus procedure and altered (see below, step 3). QIs with scores between 7 and 9 with ´no disagreement´ (less than 33% in the opposite lowest 3-point region) were directly included. For some of these indicators rheumatologists were asked to further define elements of the indicator (e.g. ’persistent disease’ and ’periodically’) during the consensus meeting.

Step 3 Consensus meeting

To achieve consensusabout so far not accepted QIs and definitions of QI elements, all experts were invited for a consensus meeting.Rheumatologists received a personalized rating sheet on which the responses of all experts were visible: their own score was marked with an X (see figure 2). Goals of the consensus meeting were to achieve consensus on the items in which there was disagreement or where the median score was between 4 and 6 (the middle 3 point-region), and to redefine elements from accepted as well as not accepted indicators in the previous round.

For the meeting 1.5 hours was scheduled. One of the authors was chairman (AdB). The discussion was audio-taped and analysed afterwards by LvH and MH. Information from the audio-taped conversations was used to adjust the indicators and to define elements more clearly.

Step 4 Second questionnaire round

Potential indicators that were discussed in the panel meeting were re-scored on the 1 to 9 scale. For those potential indicators that had already been accepted after the first round and only needed re-definition, participants were asked whether they agreed (yes or no) with the new definition. The median score from the first round and a summary of the comments during the consensus meeting were shown per potential indicator.

Items with a median score of 7 or higher without disagreement were included in the final set of quality indicators. Redefined quality indicators were accepted if at least 80% of the experts agreed with the new formulation.

Step 5 Draw up the final set

The pre-final set of quality indicators for disease course monitoring was determined and communicated to rheumatologists during a congress for all Dutch rheumatologists organised by the NVR. Rheumatologists could comment on the set and final adjustments were made. The numerator and denominator descriptions were formulated for each QI: in the denominator the target group was described and in the numerator the actual performance in the target group.