Eli Lilly and Company

Lilly Corporate Center

Indianapolis, Indiana46285U.S.A.

Phone 317 276 2581

September 13, 2012

Dr. Joe Selby

Executive Director

Patient-Centered Outcomes Reserach Institute

1828 L St., NW, Suite 900

Washington, DC 20036

Dr. Sherine Gabriel

Professor of Medicine and Epidemiology, William J. and Charles H. Mayo Professor

Mayo Clinic

PCORI Methodology Committee Chair

RE: Patient-Centered Outcomes Research Institute (PCORI) public solicitation for comments on Draft Methodology Report: “Our Questions, Our Decisions: Standards for Patient-Centered Outcomes Research”

Dear Dr. Selby, Dr. Gabriel, and Members of the Methodology Committee:

The following comments on the PCORI Draft Methodology Report are submitted on behalf of Eli Lilly and Company (“Lilly”). Lilly is one of the country’s leading innovation-driven, research-based pharmaceutical and biotechnology corporations. The company is devoted to seeking answers for some of the world's most urgent medical needs through discovery and development of breakthrough medicines and technologies and through the health information we offer. Ultimately, the company’s goal is to develop products that save and improve patients’ lives.

Lilly shares PCORI’s desire to improve patient-centered health outcomes and appreciates the opportunity to comment on this draft. The draft methodology report takes steps to establish standards as guidance to improve the scientific validity and generalizability of research while ensuring patient-centeredness. The draft report is a considerable undertaking and important advancement in meeting the mandate of the Patient Protection and Affordable Care Act (PPACA). We are pleased to provide technical comments in response to the draft methodology report and offer the following input on specific chapters:

Chapter 4:

Methodological Standards for Patient-Centeredness of Research Proposals and Protocols

  • Increasing the relevance, meaningfulness, and interpretability of data to all stakeholders (including patients) is an important progression in conducting and reporting scientific research as it pertains to drug development. An important component of this is including measures that are most relevant to patients and recruiting representative populations. Consideration of patient-centeredness during study design is very important in understanding patient’s needs, deciding inclusion/exclusion criteria, developing the data collection forms, and evaluating the feasibility of the study.
  • The proposed standard (4.1.1), states “At a minimum, patient informants should be engaged in formulating research questions; defining essential characteristics of study participants, comparators, and outcomes; monitoring study conduct and progress; and disseminating results.” Although it is key to include patients in determining the relevant questions for research, there are several issues regarding further discussion with participants following the initiation of a study. These issues include, but are not limited to, patient confidentiality, ethics, legal, and regulatory concerns, comprehension of medical practices and study design, as well as methodological complications with ‘intervening’ with patients under Good Clinical Practice. PCORI should consider how these issues should be addressed in the context of the proposed standard.
  • We recommend that the standards specifically state that researchers should follow all applicable laws and regulations.

Chapter 5:

Methods for Prioritizing Patient-Centered Outcomes Research and for Peer Review

  • This section provides a solid framework for research and will help focus research where it is needed. However, the use of systematic reviews as a means for identifying gaps in research priorities is a complex concept, and the guidance provided in the draft methods report is vague. While systematic reviews are a key component of PCOR, patient-level data may also be used to perform hypothesis-generating analyses using methods such as regression models, cluster analysis, CART analysis, etc. Therefore, this section should include additional methods for studying efficacy and effectiveness at the patient level in addition to systematic reviews.

Chapter 6:

Choosing Data Sources, Research Design, and Analysis Plan: Translation Framework and Development of a Translation Table

  • The non-randomized study section of this chapter should incorporate observational study designs. Furthermore, PCORI should update Figure 6.2 entitled Decision Tree for Comparative Effectiveness of Therapeutics to reflect the appropriate nature of observational study designs. Observational research does not rely on recruitment strategies and instead considers inclusion/exclusion criteria.

Chapter 7:

General and Crosscutting Research Methods

  • It would be a benefit to have PCORI endorse standard definitions that would apply to all patient-centered outcomes research (e.g. definition of patient sample, definition of intervention group, and definition of relevant outcomes).

Causal Inference:

  • The draft report on causal inference provided sound scientific rationale and a clear example supporting the need for proper bias adjustment techniques in non-randomized studies – as well as a few suggested key principles for analysts to follow. The section outlining causal inference methods should be expanded to address the strength, weakness, and scope of different methodologies in regards to adjusting for selection bias.
  • There are various causal inference methods which may not apply in every situation, make different assumptions, may be more appropriate than another, and may demand different sensitivity analyses in each situation. We recommend PCORI provide more clarity in order to provide guidance that will lead to trusted and reliable information. The information provided in the draft methods report is a starting point, but much more detail is needed. Without more specificity or guidance, there is a risk that conflicting results on the same question will occur without clarity as to why or how to resolve the differences.
  • The causal inference standards are limited and only focused on a small number of analytical methods. There is only a brief mention of multivariate regression, propensity scoring, and instrumental variable, meaning more information is necessary to provide clearer directions and standards for researchers. More guidance is needed in regards to retrospective data issues, pre-specification, multiplicity, and replication.
  • There is little guidance and little understanding of the operating characteristics of causal inference research based on observational data. Without an understanding of causal inference, it is difficult to appropriately interpret and make healthcare decisions on this data. Detailed guidance would be a beneficial step toward appropriate interpretation of such data.

Heterogeneity of Treatment Effects (HTE):

  • The recognition of the importance of HTE in the draft methods report is instrumental for advancing patient-centered research.The proposed standards are most applicable in subgroup identification and HTE analyses. They are a logical and defensible set of standards, analyses types, and criteria for each of the HTE study types. Descriptive HTE analysis is a novel concept, which will provide more robust data to enable future meta-analyses.
  • The draft report emphasized the importance of an interaction test to support the evidence of heterogeneity of treatment effects, however, it did not provide practical instructions regarding implementation of the test. PCORI should clearly define the level of evidence required to support a meaningful interpretation of the interaction effect (e.g. subgroup mean, least square mean) as well as best practices for selecting variables of interest and conducting the tests. Furthermore, we recommend that PCORI should state that interactions beyond two-way interactions are to be avoided. The interpretation of interactions beyond two-way becomes more difficult to interpret and implement in practice.
  • Not all subgroups in HTE are equally important or meaningful for use in treatment decisions and predictive models. This should be acknowledged, and the use of causal diagrams to address this issue should be included. Causal graphs (also known as causal diagrams), such as Directed Acyclic Graphs (DAGs) or Path Diagrams, offer researchers a better insight into causality and confounding by visually representing the apriori assumptions about the underlying biological mechanisms or causal pathways that relate exposures, outcomes, and covariates.
  • The standards for HTEs list a few examples of variables such as gender, age, co-morbidity, and lifestyle. There should also be consideration for race, ethnicity, and/or regional factors. The FDA recognizes and has done workshops in the past that address the multi-regional HTE. Finally, disease state specific variables should also be included in the standards, and a clear rationale for collection and analysis should be provided.

Preventing and Handling Missing Data:

  • The draft methods report makes a causal statement that clinical studies often are based on restricted populations because the researchers are attempting to avoid missing data which might not be realistic. Clinical studies often have restricted enrollment, but avoidance of missing data is not the primary cause for these design choices. The restricted population enables easier causal interpretation. The standards for missing data should emphasize statistical methods for handling such data and that evaluating the effect of missing data on the interpretation of results should be carefully planned and executed.
  • Guidance for trials is included in this section, and should incorporate methods related to observational study designs, including database studies. The proposed standard (7.4.4) details reasons for dropout and missing data, and we recommend PCORI include an additional section within this standard that includes possible reasons for missing data when conducting a study using secondary datasets. Issues such as enrollment in claims databases (ICD-9 codes) and data gaps when using electronic health records are critical when applying statistical methods to these data with missing gaps especially as it pertains to the various categories of missing data such as Missing At Random (MAR), Missing Completely At Random (MCAR), and Missing Not At Random (MNAR).

Data Networks:

  • This section captures the critical elements in data networks. It describes the core aspects to consider for quality data from a data network in a high-level comprehensive manner: Data Harmonization (structure and content), Data Privacy, Data Ownership, Intellectual Property, Data Access, Network Establishment and Network Maintenance. The rationale for standards is also logical and compelling.
  • Standard 7.5.5. outlines the need for standardized terminology encoding in data networks. However, multiple terminology systems for electronic health care databases exist with different terminology such as ICD codes and READ codes. It is unclear if standardized terminology encoding refers to within one terminology system or across different systems. A standard set of terminology would be ideal, but it would be very difficult to achieve broad agreement and use. In addition, mapping standards are needed to convert codes from one system to another and should be considered for future work.
  • On report page 78, the first sentence of the second paragraph mentions the importance of “quality of data,” but no standard specifically addresses the quality of incoming data to networks. PCORI should consider including a standard that addresses the quality of incoming data.
  • In addition to the recommended actions on page 79, for future work, we propose PCORI consider promoting a priori industry standards (including structural, syntactic, and semantic standards) for electronic health records to support easier harmonization of data elements across the spectrum of care. For example, the Certification Commission for Health Information Technology (CCHIT) has established certification requirements and standards for electronic health records that PCORI could recommend.

Chapter 8:

Design-specific Methods

Adaptive and Bayesian Trial Designs:

  • The section and supplemental materials are in line with current practice and raise the awareness of adaptive designs and Bayesian methods in general. The supplemental materials are written by some of those on the forefront of adaptive designs and include relevant references, etc. The use of Bayesian trials methods has broad potential and more information about methods for this type of trial should be included in the final report. In addition, we recommend PCORI consider including additional sections and standards for various studies including: newly emerging methods for observational studies primarily evaluating safety issues including high dimensional propensity scores, self control study design, and sequential probability testing.

Data Registries:

  • The justification of the use of data registries is relatively complete. Defining minimal standards for data registries to ensure quality, standardization, and rigor is a step in the right direction. For low-prevalence conditions, registries many be the only opportunity to have meaningful power to analyze treatment interactions within subgroups that may be a key to patient-centered approaches. Randomized trials and other observational studies with the ability to detect interactions with sufficient power that accounts for multiplicity of testing multiple subgroups may be rare, and registries may fill that important need.
  • Additional benefits of registries include: ability to provide data on populations not typical in RCTs (e.g. children, elderly, minorities, pregnant, and co-morbidity); ability to examine impact of physician practice behaviors on quality of care, prescribing preference, consequences of healthcare decisions, etc.; and usefulness for signal amplification, particularly for rare outcomes.
  • The proposed minimal standards for data registries tend to be more directed at good practices for designing new data registries. Furthermore, there are three uses of data registries for PCOR studies that call for somewhat different quality considerations and standards:

(1)Observational PCOR studies for which a Registry is being newly designed to provide data.

(2)Observational PCOR studies which propose to use data elements from an existing ongoing Registry to address a research question.

(3)Observational PCOR studies which propose to modify an existing ongoing Registry to address a research questions.

We recommend that PCORI consider including standards in the report that account for these types of registries and address related quality requirements for each.

  • When using an existing registry, the standard should require a feasibility assessment of a proposed registry based on its history of operation (taking into account potential sponsor or clinician biases) and quality to date.
  • There are numerous centralized computer assisted registry technologies that are based in phone, internet, laptop/computer, home interview, daily diary/experience sampling, and smart phone technologies. These registries are less tied to immediate treatment setting and patient visits, and may offer more control in minimizing follow-up attrition and missing data. In addition, these types of registries may be more suited not only to patient needs, but also collecting patient-reported outcomes. The proposed data registry standards should recognize the emerging use of non-traditional registries and survey panel registry designs versus site-based registries.
  • A comparison group should be included in the standards for conducting research using data registries.
  • There is no guidance in the standards for data registries relating to whether registry patients for proposed PCOR can also be simultaneously enrolled in clinical trials or not. Clinical trials impose specific treatment protocols, potentially confounding the registry sample and possibly changing the generalizability to real world outcomes.
  • When outlining the use of registries, the document should include an emphasis on the advantage of having a larger sample size in a data registry. Particularly for conditions and rare diseases with small patient populations, registries may be the only opportunity to have meaningful statistical power to analyze treatment interactions within subgroups that may be a key to patient-centered approaches.
  • In standard 8.2.4 regarding the safety and security of data registries, the inclusion of a process for reporting serious adverse events associated with treatments should be included.
  • Standard 8.2.9 should include the proper methods to address biases and effect modification as well as confounding. Sensitivity analysis to assess the impact of key assumptions in studies should be added. In addition, an issue with many registries is whether individuals can be enrolled multiple times within a site or between sites, and standards should discuss how this can be avoided while protecting patients’ privacy.
  • Standards for data registries should include information to assist researchers in ensuring the registry population used within a study is comparable to the target population.

Studies of Diagnostic Tests:

  • The report should endorse the Grace Principles (2010), which describes a hierarchy of evidence for observational research on comparative effectiveness that can be used by decision-makers, as well as key elements of good practice including defining research questions and methods a priori; collecting valid, clinically relevant data; analyzing, interpreting and reporting data, including sensitivity analyses and alternative explanations for findings; and conducting these studies in accordance with accepted good practices. Currently, there are no clear standards or requirements for what infromation should be captured or how long it should be stored and made available.
  • The report should differentiate between diagnostic or screeninglab tests for biomarkers versus subjective assessment tools where an objective biomarker is not available for patient-reported outcomes (e.g. measurements of pain, depression, or anxiety).
  • We recommend that PCORI consider discussing the significance of understanding the sensitivity and specificity of diagnostic tests and the importance of having this information readily available for medical personnel and patients alike.This was not included in the draft report. Finally, it would be beneficial to reference in the draft report that the current process of drug and diagnostic tests approval lacks a systematic approach to benefit-risk analysis, leading to inconsistency, lack of transparency, and an inability to challenge or defend decisions- in both the pre-approval and post-marketing setting. The risk-benefit analysis that currently goes into regulatory decisions appears to be ad hoc, informal, and qualitative, without defined and tested algorithm or summary metric that combines benefit and risk data that might permit straightforward quantitative comparative analysis. This is an important gap that needs to be addressed.

Lilly values the opportunity to comment on the future of PCORI. We look forward to continuing to work with PCORI to improve the standards for patient-centered outcomes that improve patients’ lives. If you have any questions or need additional information, please feel free to contact me. Thank you for your attention to this very important matter.