Communication and Dissemination Research PFA Cycle 1, 2017

Offline Reviewer Critique Template

Communication and Dissemination Research

PCORI seeks to fund projects that address critical knowledge gaps in the communication and dissemination process—both the communication and dissemination of research results to patients, their caregivers, and clinicians, and the communication between patients, caregivers, and clinicians—in the service of enabling patients and caregivers to make the best-possible decisions in choosing among available options for care and treatment.

Important Reminders:

  • Before beginning your first critique, read the CDR PCORI Funding Announcement. Key aspects of the funding announcement can also be found in the reference table at the end of this document.
  • Use this offline template to draft a written critique for each criterion you have been assigned to review.The offline reviewer critique template begins on page 3.
  • Please use the scoring rubric on page 2 to ensure that your comments and numeric scores align. This helps other reviewers and Program staff to better understand your scores and to use that information in preparing for the in-person meeting.
  • Once you have completed your critique based on the review criteria, double-check to make sure that you have addressed the key points listed on page 10.
  • Once you have finished drafting your critique, please copy/paste your comments into PCORI Online.

Reference: PCORI Scoring Chart – Criterion Scores

Definitions of strength and weakness modifiers:

  • Major Strength: An attribute that is likely to lead to improvements in healthcare and/or outcomes
  • Moderate Strength: An attribute that would probably lead to improvements in healthcare and/or outcomes
  • Minor Strength: An attribute that could lead to improvements in healthcare and/or outcomes
  • Minor Weakness: An easily addressable weakness that does not substantially lessen the impact of the study’s results on healthcare and/or outcomes
  • Moderate Weakness: A weakness that would lessen impact of the study’s results on healthcare and/or outcomes
  • Major Weakness: A weakness that would seriously limit impact of the study’s results on healthcare and/or outcomes

Please use this template only as a placeholder for your critiques. Once you have finished your critique, please copy/paste your comments into the PCORI Online Critique Form. (To access the Online Review form for each assigned application, navigate to the Reviewer Dashboard in PCORI Online.)

Keep comments for each criterion to no more than 5000 characters, including spaces.Comments for each criterion should list out strengths and weaknesses using a bulleted format. Comments for the Protection of Human Subjects and Overall Narrative sections should be written in paragraph form.

Request ID:

Criterion 1. Potential for the study to fill critical gaps in evidence (Not required for Patient and Stakeholder Reviewers)

The proposal addresses the following questions:

  • Does the application convincingly describe the clinical burden?
  • Does the application identify a critical gap in current knowledge as noted in systematic reviews, guideline development efforts, or previous research prioritizations?
  • Does the application identify a critical gap in current knowledge, evidenced by inconsistency in clinical practice and decision making?
  • Would research findings from the study have the potential to fill these evidence gaps?

Strengths:

Weaknesses:

Criterion 1 Score:

Criterion 2. Potential for the study findings to be adopted into clinical practice and improve delivery of care (All reviewers)

The application should describe how evidence generated from this study could be adopted into clinical practice and delivery of care by others. The application should also address the following questions:

  • Does the application identify who will make the decision (i.e., the decision maker) or use (i.e., the end-user) the study findings (not the intervention) this study produces, such as local and national stakeholders?
  • Does the application identify potential end-users of study findings—such as local and national stakeholders—and describe strategies to engage these end-users?
  • Does the application provide information that supports a demand for this kind of a study from end-users?
  • Would this study’s research findings have the potential to inform decision making for key stakeholders? If so, provide an example. How likely is it that positive findings could be reproduced by others, resulting in improvements in practice and patient outcomes? Identify the potential barriers that could hinder adoption of the intervention by others.
  • Does the application describe a plan for how study findings will be disseminated beyond publication in peer-review journals and at national conferences?

Strengths:

Weaknesses:

Criterion 2 Score:

Criterion 3: Scientific merit (research design, analysis, and outcomes) (Not required for Patient and Stakeholder Reviewers)

The application should show sufficient technical merit in the research design to ensure that the study goals will be met. The application should also address the following questions:

  • Does the application describe a clear conceptual framework anchored in background literature which informs the design, key variables, and relationship between interventions and outcomes being tested?
  • Does the Research Plan describe rigorous methods that demonstrate adherence to the PCORI Methodology Standards?
  • Is the overall study design justified?
  • Are the patient population and study setting appropriate for the proposed research question?
  • Does the application provide justification that the outcome measures are validated and appropriate for the population?
  • Are each of the comparators (e.g., active intervention arm and comparator arm) described clearly and well-justified? If “usual care” is one of the arms, is it adequately justified and will it be sufficiently measured?
  • Are the sample sizes and power estimates appropriate? Is the study design (e.g., cluster randomized design, randomized controlled trial, or observational study) accounted for and is the anticipated effect size adequately justified?
  • Is the study plan feasible?Is the project timeline realistic, including specific scientific and engagement milestones?Is the strategy for recruiting participants feasible? Are assumptions about participant attrition realistic, and are plans to address patient or site attrition adequate?

Strengths:

Weaknesses:

Criterion 3 Score:

Criterion 4: Investigator(s) and environment (Not required for Patient and Stakeholder Reviewers)

This criterion should assess the appropriateness (e.g., qualifications and experience) of the investigator(s)/team and the environment’s capacity (e.g., resources, facilities, and equipment) to support the proposed project. It should not be an assessment of the institution’s quality. The application should also address the following questions:

  • How well-qualified are the PIs, collaborators, and other researchers to conduct the proposed activities? Is there evidence of sufficient clinical or statistical expertise (if applicable)?
  • Does the investigator or co-investigator have demonstrated experience conducting projects of a similar size, scope, and complexity?
  • If the project is collaborative or dual-PI, do the investigators have complementary and integrated expertise? Are the leadership, governance, and organizational structures appropriate for the project?
  • (Dual-PI Option Only) Does the Leadership Plan adequately describe and justify PI roles and areas of responsibility?
  • Is the level of effort for each team member appropriate for successfully conducting the proposed work?
  • Does the application describe adequate availability of and access to facilities and resources (including patient populations, samples, and collaborative arrangements) to carry out the proposed research?
  • Is the institutional support appropriate for the proposed research?

Strengths:

Weaknesses:

Criterion 4 Score:

Criterion 5: Patient-centeredness (All reviewers)

The application should demonstrate that the study focuses on improving patient-centered outcomes and employs a patient-centered research design (i.e., a design informed or endorsed by patients). (Note: The study can be patient-centered even if the end-user is not the patient, as long as patients will benefit from information.) The application should also address the following questions:

  • Does the application include a thorough description about which outcomes (both benefits and harms) are important to patients, and are those outcomes included in the study plan?
  • Does the application provide information that indicates that closing the evidence gap is important to patients and other stakeholders?
  • Are the interventions being compared in the study available to patients now, and are they the best options for comparison (including whether they would be chosen by patients and their healthcare providers for managing the condition being studied)?

Strengths:

Weaknesses:

Criterion 5 Score:

Criterion 6: Patient and stakeholder engagement (All reviewers)

The application should demonstrate the engagement of relevant patients and other stakeholders (e.g., patients, caregivers, clinicians, policy makers, hospitals and health systems, payers [insurance], purchasers [business], industry, researchers, and training institutions) in the conduct of the study. Quality of engagement should be evaluated based on scope, form, and frequency of patient and stakeholder involvement throughout entire research process. The application should also address the following questions:

  • Does the application provide a well-justified description of how the research team incorporates stakeholder involvement? Does the study include the right individuals (e.g., researchers, patients, caregivers, clinicians, policy makers, and other healthcare system stakeholders) to ensure that the projects will be carried out successfully?
  • Does the application show evidence of active engagement among scientists, patients, and other stakeholders throughout the research process (e.g., formulating questions, identifying outcomes, monitoring the study, disseminating, and implementing)? Is the frequency and level of patient and stakeholder involvement sufficient to support the study goals?
  • Is the proposed Engagement Plan appropriate and tailored to the study?
  • Are the roles and the decision-making authority of all study partners described clearly?
  • Are the organizational structure and resources appropriate to engage patients and stakeholder throughout the project?

Strengths:

Weaknesses:

Criterion 6 Score:

Protection of Human Subjects:(Not required for Patient and Stakeholder Reviewers)

Does the application have acceptable risks and/or adequate protections for human subjects? (Yes/No)

Please provide comments related to human subjects protections, if any.

Overall Score:

Overall Narrative:

Please provide your overall narrative here. The narrative should be written in paragraph form and provide a summary of strengths and weaknesses that drove the overall score (provided above).

Reference: Key Points

Once you have completed your written critique based on the review criteria, double-check to make sure that you have addressed these key points:

Criterion 2

Does the proposed research study the benefits and harms of interventions delivered in real-world settings, and will the study provide practical information to help patients make informed healthcare decisions?

Does the application describe a plan for how study findings will be disseminated beyond publication in peer review journals and national conferences?

Criterion 3

Does the proposed research compare at least two viable alternatives, each supported by efficacy data?

Does the proposed research include a clear conceptual framework, theory, or model that anchors the background and significance and informs the design, key variables, and relationships being tested?

Are each of the comparators (e.g., active intervention arm and comparator arm) clearly described and well justified? If usual care is one of the arms, is it sufficiently justified, sufficiently described and will it be sufficiently measured?

Does the proposed research have sufficient statistical power to detect clinically meaningful effects? Are proposed sample sizes justified? Are assumptions used in power calculations explained?

Is there a solid plan to recruit, enroll and retain study participants?

Criterion 5

Does the proposed research include health outcomes that are meaningful to the patient population, their caregivers, and family members under study, and that are likely to guide their decisions?

Are the interventions being compared in the study available to patients now, and are they the best options for comparison?

All Criteria:

Does the proposed research adhere to all relevant PCORI Methodology Standards? (Note relevant Review Criteria for addressing each standard.)

  • Identify a true gap in evidence (RQ-1; Crit. 1)
  • Provide a rationale and define “usual care,” if it is proposed (RQ-5; Crit. 2 & 3)
  • Show that the outcomes chosen are relevant to patients and end users (RQ-6; Crit. 5)
  • Show that study participants represent target populations and are properly recruited, enrolled, and retained (PC-2; Crit. 3)
  • Describe rigorous data analysis plans (IR-1, IR-3; Crit. 3)
  • Describe their plan and statistical methods for handling missing data in their project (MD-1, MD-2; Crit. 3)
  • Address heterogeneity of treatment effects and state the aims of HTE analysis (HT-1; Crit. 3)

Notify your MRO if the application proposes to:

Measure cost-effectiveness or conduct a formal cost-effectiveness analysis as a primary aim.

Develop, test, and/or validate individual decision aids, tools, or mobile applications.

1

Reference: CDR PFA Summary

What is CDR interested in?
The CDR program is seeking applications that study the comparative effectiveness of communication and dissemination strategies aimed at informing and empowering patients, caregivers, and other healthcare decision makers so that they know what questions to ask and have the information needed to provide support in shared decision making.
CDR is interested in comparative effectiveness research that
  • Identifies and compares:
  • Communication strategies to promote the use of health and healthcare CER evidence by patients and clinicians
  • Dissemination strategies to promote the use of health and healthcare CER evidence by patients and clinicians
  • Explanation of uncertain health and healthcare CER evidence to patients and clinicians
  • Compare:
  • Strategies that increase knowledge on how to communicate complex information to patients and caregivers.
  • Strategies meant to generate conversations between patients and providers about what is appropriate and necessary. treatment based on patients’ preferences and conditions
  • Strategies for conveying uncertainty associated with health and healthcare evidence that increase the likelihood that patients and caregivers will understand the information and be able to incorporate it into decision making and evaluate personal trade-offs.
  • CER dissemination strategies while evaluating the potential for implementation in real-world settings (e.g., hybrid effectiveness-implementation design trial).
  • The effectiveness of health literacy- and numeracy-sensitive health communication strategies that relay risks and benefits of health decisions so that individuals can make sound healthcare decisions.
  • Strategies and methods that optimize communication between the patient, family/caregiver, and the healthcare team.
  • Innovative approaches in the use of existing electronic clinical data and other electronic modalities (e.g., electronic health records) from the healthcare system or from a network of systems to enhance clinical decision making by patients and providers.
  • Compare and identify best practices of dissemination and translation techniques to facilitate shared decision making in everyday practice.
  • Identify the most-effective approaches to disseminating CER results to healthcare providers, with the goals of sustained changes in clinical practice and effective dissemination to patients of results that enable behavior changes (e.g., self-care).
/ All letters of intent have been screened to ensure programmatic alignment with the CDR program’s research interests.
Programmatic fit should NOT impact your scores or your written critiques.
Please notify your MRO if you have any questions or concern.
CDR must fund applications that compare at least 2 viable alternatives
  • NOTE: “Usual care” may be a comparator if this is a realistic choice faced by patients and other stakeholders.
  • If usual care is included, the application (1) must justify the choice to use usual care, and (2) should clearly explain how the term “usual care” is defined, and clearly describe the components of usual care that will be used or measured in the research.
/ Please notify your MRO as soon as possible if you believe any of your assigned applications do not include a comparator.Your MRO will discuss any identified issues with program staff and provide you with further instructions.
What is CDR NOT interested in?
PCORI cannot fund applications that measure cost effectiveness.
  • Research cannot conduct a formal cost-effectiveness analysis in the form of dollar-cost per quality-adjusted life year.
  • Research cannot measure the relative costs of care of two or more alternative approaches as the primary criterion for choosing the preferred alternative.
  • PCORI cannot fund applications that develop practice guidelines, coverage recommendations, payment or policy recommendations.
/ If an application meets any of these conditions, please notify your MRO as soon as possible. Your MRO will discuss all identified issues with program staff and provide you with further instructions.
CDR applications does not fund applications with a primary focus on tool/decision aid development, testing, and/or validation.
PCORI cannot fund research whose findings will include:
•Practice guidelines
•Coverage recommendations
•Payment or policy recommendations
•Creation of clinical practice guidelines or care pathways
•Establishing efficacy for a new clinical strategy
•Pharmacodynamics
•Study of the natural history of disease
•Fundamental science or study of biological mechanisms
CDR applications shouldnot focus on the following categories:
  • Instrument development
  • Developing, testing and validating new decision aids/tools or clinical prognostication tools
  • Pilot studies intended to inform larger efforts
  • Comparisons of patient characteristics rather than clinical strategy options
Studies comparing interventions for which the primary focus in the role of community health workers or patient navigators.

1