The Journal of Cystic Fibrosis

The Journal of Cystic Fibrosis

Review File

VALIDATION OF THE TREATMENT SATISFACTION QUESTIONNAIRE FOR

MEDICATION IN PATIENTS WITH CYSTIC FIBROSIS

Antoine Regnault, PhD; Maria-Magdalena Balp, MD; Károly Kulich, PhD; Muriel Viala-

Danten, MSc

Review timeline:

Submission date: 26 July 2011

Editorial Decision: 12 December 2011 (1st decision), 25 March 2012 (2nd decision)

Revision received: 10 February 2012 (1st Revision), 11 April 2012 (2nd Revision)

Additional Correspondence: 12 April 2012

Accepted: 13-Apr-2012

Transaction Report:

(Note: With the exception of the correction of typographical or spelling errors that could be a source of ambiguity,letters and reports are not edited. The original formatting of letters and referee reports may not be reflected in thiscompilation.)

1st Editorial Decision

Reviewers have now commented on your paper. You will see that they are advising that you revise your manuscript. If you are prepared to undertake the work required, I would be pleased to reconsider my decision.

Finally, I would appreciate if you could submit your revised paper by Feb 10, 2012.

Yours sincerely

Editor-in-Chief

Journal of Cystic Fibrosis

2nd Editorial Decision

Finally, I would appreciate if you could submit your revised paper by April 24, 2012.

Yours sincerely

Editor-in-Chief

Journal of Cystic Fibrosis

REFEREE REPORTS

Referee 1 (Remarks to the Author):

The study reports results of the psychometric analyses of a Questionnaire for Patient Satisfaction with Medication in a sample of a clinical study with patients with CF receiving inhaled antibiotics. As patient reported outcomes are considered increasingly important to evaluate treatments and to identify barriers to treatment in patients with chronic diseases, it is important to develop and validate instruments on patient satisfaction. So far, there is no CF-specific instrument available, and generic instruments such as the TSQM have seldom been validated for CF. The paper is well written, concise and based on the large database of a clinical trial. However, I have several concerns with the study and with the methodical approach.
Major Comments
1) As far as I understand, the TSQM was developed for adults with different chronic diseases and not for children and adolescents. This raises questions on the feasibility and validity of the instruments for minors with chronic diseases. It is very challenging to develop a self-report measure across developmental phases from childhood to adulthood, with respect to different cognitive and language development which is clearly associated with the construct patient satisfaction. The authors report that patients <13 years completed the questionnaire with the assistant of a parent or guardian. They controlled for an effect of assisted response by statistical analyses. However, the absence of remarkable differences between responses of minors <13 years and adolescents and adults does not imply that both approaches are equivalent, in terms of measuring the same construct. The parent perspective is probably included in the children's responses. I would suggest to avoid the problem of assessing children with an instrument for adults by restricting the whole study on participants from 12 years (formal stage of cognitive development). The authors might refer to the consensus statement regarding the assessment of quality of life in clinical trials with CF (Abbot et al. . J Cystic Fibr 10 Suppl 2:S82-5), as QoL assessment has s challenges than the assessment of patient satisfaction.

2) The second and even more important concern than the need for developmental adjustment of an instrument is the language problem. The authors should report how many different language versions of the TSQM they used. Each language version should be a carefully forward and backward translated instrument which then has to be validated in each language to assure equivalence of the different language versions. A minimal requirement, however still insufficient standard of cross-cultural validation would be to control for language in the statistical analyses.

3) The theoretical background for the proposed associations between satisfaction, convenience and compliance in CF should be described in the introduction. At the end of introduction, hypotheses about these associations should be stated. The first sentence of the SEM results should be moved to the introduction and explained in detail.

4) In the sample description, it would be helpful to break down age (number of children, adolescents, adults), countries and languages.

5) The description of the original instrument should be more detailed. What was the theoretical background of the developers? How many items per scale are there, and how do the items refer to dimensions? What are the psychometric properties (validity, reliability) of the original instrument? How is the scoring procedure? Which is the format of the response scale? How is a score on side effects calculated if no side effects are reported (and consequently no further concerns on side effects occur)?

6) It is not clear how compliance was assessed. Also the rationale for an 80% cutoff between good and poor compliance should be explained.

7) Why would the authors conclude that the four dimension structure of the questionnaire was confirmed (p 7, line 35)? Usually, a confirmatory factor analysis would be needed to base this statement. Global satisfaction cannot be considered an independent factor.
Minor comments:
8) The affiliation of the first and last authors (company) should be stated more clearly. Mapi Values company?

9) I am not familiar with the rules of the JCF how to name products or pharmacological agents (as a trademark or as substance). The results of this study are based on a pooled sample across the treatment arms of the study. It might not be necessary in the context of this paper to use the trade names the products, as the questionnaire might be used for any inhaled medication in CF.

10) Please report in detail how the instruction of the questionnaire was adapted for CF.

11) A separate section on procedures should be incorporated in the methods section.

12) More details on the disease severity of the patients should be reported, e.g. the number of pulmonary exacerbations before and during the trial.
Referee 2 (Remarks to the Author):

This manuscript presents important validation data for the TSQM, a treatment satisfaction outcome measure that was used in the EAGER trial of dry powder tobramycin in CF. The data has relevance to both CF clinicians and researchers.
I have some comments and questions for the authors to address in a revision:
-In the abstract and the introduction, the authors state that the goal of their study is to demonstrate "relevance" or to demonstrate that the measure is "appropriate." Neither of those words reflects what the actual goal seems to be - namely testing validity and reliability of the measure. Thus I would remove those words in place of the more appropriate statistical aims and hypotheses. Relevance is a matter that is open to interpretation. A measure can be valid but not relevant, for instance.
-Can the authors provide some more detail on the actual instrument? Is it a Likert scale for responses? How is the transformation to 0-100 completed? How many items are mapped to each of the 4 domains? How long does it take a subject to complete the measure?
-The psychometric data is presented for week 5 data. Were similar cross-sectional analyses performed at weeks 13 and 21? Do the cronbach's alpha and correlations remain similar if later weeks are used for data? That would be an important measure of the stability of the responses over time. Did the authors also evaluate test-retest reliability?
-Were any longitudinal analyses performed to assess for the stability of the measure scores over time?
-In the EAGER manuscript published by Konstan et al in JCF, data is presented on TSQM scores showing differences in the TIP and TOBI arms of the study. This data needs to be references in this publication, particularly since the goal of the measure is to help discriminate treatment satisfaction in a research setting. Were the psychometric analyses performed separately by treatment arm and did the measure have similar properties in the two arms?
-In the discussion, the authors state that 4 additional questions were added to be more CF specific. I would like to see these questions and the data presented in this paper. This would allow for a better understanding of whether the TSQM should be modified by other CF researchers, and if so, what potential benefit additional questions might have on the instrument as a whole. I would encourage the authors to include this data in the current manuscript, or at least include the additional questions and the baseline scores and ranges for these questions.
-Figure 1 needs a more complete legend, explaining what the data presented in the figure represents. The methodology is explained in the text, but the figure should stand alone for a reader to interpret the data.

1st Revision - authors' response

Reviewer 1

Major Comments

1) As far as I understand, the TSQM was developed for adults with differentchronic diseases and not for children and adolescents. This raises questionson the feasibility and validity of the instruments for minors with chronicdiseases. It is very challenging to develop a self-report measure acrossdevelopmental phases from childhood to adulthood, with respect to differentcognitive and language development which is clearly associated with theconstruct patient satisfaction.The authors report that patients <13 years completed the questionnaire withthe assistant of a parent or guardian. They controlled for an effect ofassisted response by statistical analyses. However, the absence of remarkabledifferences between responses of minors <13 years and adolescents and adultsdoes not imply that both approaches are equivalent, in terms of measuring thesame construct. The parent perspective is probably included in the children'sresponses. I would suggest to avoid the problem of assessing children with aninstrument for adults by restricting the whole study on participants from 12years (formal stage of cognitive development).The authors might refer to the consensus statement regarding the assessment ofquality of life in clinical trials with CF (Abbot et al. . J Cystic Fibr 10

Suppl 2:S82-5), as QoL assessment has s challenges than the assessment ofpatient satisfaction.

Authors‟ response:

We fully agree with the reviewer about the specific care needed for measuring Patient-

Reported Outcomes in children and the challenging task of developing an instrument to becompleted by children, in particular to integrate the variability of children experience due tothe developmental process. In particular, we think that, for an instrument to be consideredfully appropriate for self-administration in children, its development should have involvedchildren in a quantitative research phase to demonstrate the content validity of the instrumentin the children population.

Our purpose was not to warrant the use of the TSQM in children with CF. For this purpose, aspecific work with both qualitative research involving children and a specific analysis of themeasurement properties in a homogeneous sample of children would be needed. Forexample, such work might start by exploring whether a complex concept as satisfaction isrelevant for children and could be measured in this population. If so, it might be interesting toexplore whether satisfaction is characterized in the same way for children than for adults andadolescents.

Our perspective was actually slightly different: in the EAGER study, the children did notrespond alone to the TSQM, they were helped by a parent or tutor - because the difficulty ofthe measure of satisfaction in children was anticipated. We agree that the responses to the

TSQM items in this context probably largely reflect the parent‟s perspective. So the conceptbeing measured in this context is not perfectly straightforward and the question pooling thedata from the two mode of completion was asked. The differential item functioning analyseswere performed to investigate whether there was any systematic difference between the twomodes of completion. No major difference was detected. While we agree that this does notguarantee that the same concept is measured, it is nonetheless a positive signal for thepooling of the data. The possibility of using further analysis (like for example confirmatoryfactor analysis to compare the relationships of items to the TSQM domains depending on thecompletion mode) was considered, but the small number of children younger than 13 madetheir application impossible. Finally, it was added that this fairly small number of youngerpatients (42 patients out of a total number of 454) would not drive the results and includingthe small sample in the analysis could actually only be deterrent to the measurementproperties by adding noise –so they could not invalidate the good measurement properties ofthe TSQM.We modified the discussion of the manuscript to make our approach clearer and in particularto acknowledge more clearly that our work does not warrant the specific use of the TSQM toassess treatment satisfaction in children, and that further work would be needed for thispurpose.

2) The second and even more important concern than the need for developmentaladjustment of an instrument is the language problem. The authors should reporthow many different language versions of the TSQM they used. Each languageversion should be a carefully forward and backward translated instrument whichthen has to be validated in each language to assure equivalence of thedifferent language versions. A minimal requirement, however still insufficientstandard of cross-cultural validation would be to control for language in thestatistical analyses.

Authors‟ response:

We agree with the reviewer that cross-cultural difference can have substantial impact on

PRO measures. The EAGER study was a multinational trial conducted in 15 countries so thequestion of the equivalence of the different language versions of the questionnaire is animportant one.

The TSQM questionnaire had undergone rigorous linguistic validation (with forwardtranslation, backward translation, and tests with patients). Linguistic validation is therecommended approach to optimize the equivalence of language versions of PROinstruments (see for example the ISPOR principles of good practice for translation andcultural adaptation process for PRO measures, Wild et al. Value in Health 2005).

More specifically to our study, we considered conducting statistical analyses aiming toinvestigate cross-cultural equivalence of the TSQM (for example like those reported inRegnault et al. Clinical trials 2009). Unfortunately the composition of the sample of theEAGER trial made such analyses very difficult since about two third of the sample was fromthe US and the remaining third was split in very small sample from the other 14 countries.Thus, it would have been impossible to reliably apply statistical methods at the country levelbecause of extremely small samples and, even grouping similar countries into homogeneoussubsamples (at the region level for example) appeared to be challenging, and potentially notvery informative. Consequently, it was decided not to perform such analyses.We added as a limitation in the discussion that the cross-cultural equivalence between thedifferent language versions could not be tested with the available data.

3) The theoretical background for the proposed associations betweensatisfaction, convenience and compliance in CF should be described in theintroduction. At the end of introduction, hypotheses about these associationsshould be stated. The first sentence of the SEM results should be moved to theintroduction and explained in detail.

Authors‟ response:

The hypotheses regarding the association between satisfaction, convenience and compliancethat were tested in the analyses, as well as the theoretical work from which they werederived, are specified in the introduction of the revised manuscript, as requested by thereviewer.

4) In the sample description, it would be helpful to break down age (number of children, adolescents, adults), countries and languages.

Authors‟ response:

As requested by the reviewer, a description of age in class (children, adolescent and adults),and countries was added to table 1. Unfortunately the actual language version of thequestionnaire completed by the patients was not recorded in the database and thereforecould not be described.

5) The description of the original instrument should be more detailed. Whatwas the theoretical background of the developers? How many items per scale arethere, and how do the items refer to dimensions? What are the psychometricproperties (validity, reliability) of the original instrument? How is thescoring procedure? Which is the format of the response scale? How is a scoreon side effects calculated if no side effects are reported (and consequentlyno further concerns on side effects occur)?

Authors‟ response:

The development and validation of the TSQM have been comprehensively described anddiscussed in the original articles on the questionnaire (Atkinson et al. 2004; Atkinson et al.2005), which are cited in our paper. Given the constraints in terms of article length, it wasimpossible to repeat this information in our paper. Nonetheless, we understand that theamount of information given about the questionnaire was insufficient to the reviewer so wedeveloped the section presenting the TSQM in the revised version of the manuscript.Specifically, we added some information about the development of the questionnaire, theresponse scale and how the side-effect score is calculated when no side effects are reportedby the patient (in this case, the score is set to 100). In addition, figure 1 shows how the itemsof the TSQM relate to the domains.

6) It is not clear how compliance was assessed. Also the rationale for an 80%cutoff between good and poor compliance should be explained.

Authors‟ response:

Used and unused capsules or ampoules were returned at each follow-up visit (Week 5, 13and 21) as well as a Patient Dosing Log. The number of capsules or ampoules returned perpatient was used to compute compliance. This is now specified in the text.The cut-off of 80% is often used in the literature to characterize good compliance (Cramer etal. Value in Health 2008; Osterberg & Blaschke, N Engl J Med 2005). Nevertheless, we agreewith the reviewer that this threshold does not have any clear justification. This is nowmentioned in the discussion and the relevance of further research about this threshold iscommented.