Summary Report
Needs Assessment Survey
January 2013
Guideline Implementability Research and Application Network
This work was supported by funding from the Canadian Institutes of Health Research
1
Needs Assessment Summary Report, January 2013
BACKGROUND
Guideline implementability
Implementability refers to the characteristics of guidelines that enhance their implementation by users. This approach capitalizes on the guideline as a vehicle for providing users with context-specific information or instructions (that we collectively label as “tools”) in guidelines or as adjunct products by which to identify barriers, implement guidelines, and evaluate guideline-specific performance. Following are practical examples of guideline implementability tools (GItools).
Domain/Definition / Examples of GItoolsResource Implications
Equipment or technology; type/number of health professionals; education, training of staff; anticipated changes in workflow during or after adoption; costs /
- Canadian Stroke Strategy Unit Care Guide
- Nova Scotia Human Resource Toolkit
Implementation
Identifying barriers; selecting and tailoring implementation strategies that address barriers; point-of-care tools that integrate guidelines /
- Opioid Manager (for EMR) for Canadian opioid use guidelines
- Canadian Stroke Strategy Implementation Toolkit
Evaluation
Instructions and tools on how to identify, develop, measure or interpret performance measures /
- Canadian Association of Radiologists Maximizing the Effectiveness of Clinical Audits: A step-by-step guide
- Canadian Stroke Strategy Performance Measurement Manual
Research
We launched the guideline implementability research program from 2008 to 2010. One, we reviewed the literature for information or instructions included in or with guidelines associated with guideline use (1). This generated an implementability framework that included 22 elements organized in eight domains: adaptability (versions for different users/purposes), usability (formatting), relevance (explicit purpose), validity (quantity/quality of evidence), applicability (tailoring to individual patients), communicability (information for patients), resource implications (costs and processes), implementation (instructions for adopting) and evaluation (instructions for monitoring).Two, we examined the content of 20 high-quality guidelines on various clinical indications (1). Most did not contain these elements, revealing many opportunities to improve implementation by including information on one or more of these elements in or with guidelines. Details on resource implications, implementation and evaluation were the least prevalent therefore our ongoing research focuses on these domains.Three, we explored the relevance of implementability by interviews with 30 guideline developers or implementers from government and professional societies in seven countries. They preferred implementability relative to traditional approaches for implementing guidelines because it was considered feasible and therefore more likely to have impact due to broad adoption, plus there was a demand for such information among users of their guidelines (2). Notably, they requested guidance for developing and incorporating this information in guidelines. Four, we examined the content of guideline development instructional manuals for advice on how to produce implementable guidelines but found most were lacking in this regard, confirming the need for guidance on how to develop implementability information (3). Five, we held a Canadian meeting of 40 guideline developers, implementers and researchers to prioritize implementability development and research. Participants recommended we establish methods and criteria/attributes by which to develop GItools; create an inventory of GItools; and develop new GItools for barrier analysis, performance evaluation, and multidisciplinary decision-making, and evaluate their impact. Thus our ultimate intent is to develop a searchable Internet directory, and a complementary process for reviewing and approving GItools submitted by others to the directory. This may lead to a sustainable effort where international guideline developers, implementers and researchers contribute to, and draw from this shared resource.
Collaboration
The Guideline Implementability Research and Application Network (GIRAnet), in partnership with the Guidelines International Network (G-I-N) was launched in November 2011. GIRAnet brings together international guideline developers, implementers, and researchers to identify, develop, evaluate and share GItools. In year one, we established a Steering Committee, developed a strategic plan, and assembled an Interest Group. We also began to search for, and describe exemplar GItools that could populate a GItool Directory and serve as templates by which to develop others. We examined the content and web sites of 100 international guidelines on various clinical topics identified in the Agency for Healthcare Research and Quality’s Guideline Clearinghouse. GItools were defined as information that helps users (clinicians, managers, policy-makers, others) to implement guidelines. They could be located within or adjunct to guidelines; formatted in a variety of ways; and generic and broadly applicable, or specific to a clinical indication or guideline. In association with these guidelines we identified 5 Resource Implications GItools, 13 Implementation GItools and 3 Evaluation GItools. To further describe GItools we analyzed their features according to criteria considered to be basic elements of transparency, reliability and validity that were also recommended by G-I-N members when surveyed in September 2012: purpose was stated, source(s) of content were cited, details about tool development were provided, and instructions for use were provided. Most of the 21 GItools we identified lacked one, and frequently more of these details.
OBJECTIVES
Our prior work suggests the need to develop new GItools, and criteria/attributes and methods for developing high-quality GItools. At the same time it is important to solicit the views of guideline developers, implementers and researchers about whether and how these, or potentially other resources would be most helpful. We undertook a needs assessment survey of G-I-N members to gauge their:
- Awareness of GIRAnet
- Understanding of the implementability concept
- Interest in GItools, and resources by which to assess or develop GItools, and
- Willingness to participate in GItool assessment or development.
The findings will be used to guide GIRAnet planning for years two and three of its funded period.
METHODS
To gather views and suggestions that would inform GIRAnet activities we surveyed G-I-N members using a cross-sectional, descriptive approach. The survey was not tested for psychometric propertiesbecause it was meant to describe common views or practices (descriptive survey approach) and not establish associations or causation (analytic survey approach). The survey was administered on the Internet using the SurveyMonkey application. G-I-N members were invited by email to complete the survey with reminders at two and four weeks. The survey included nine questions: awareness of GIRAnet, perceived impact of implementability, interest in learning more about GItools, identification of organizations that have developed GItools, perceived value of various GIRAnet products, interest in various GIRAnet services to support information sharing about GItools, research priorities, willingness to participate in various GIRAnet activities, and preferences for ongoing communication from GIRAnet. Response mechanisms included rating options on a 7-point scale ranging from strongly disagree to strongly agree, multiple choice or choose all that apply, and open-ended comments. The survey was accessible for three months, closing on January 25, 2013. Responses were analyzed for the entire group using summarystatistics (frequency, proportion). Open-ended comments and suggestions were analyzed and grouped thematically.
RESULTS
Respondents
There were a total of 204 respondents from 33 different countries, most frequently from Australia (16, 11.5%), Germany (15, 10.8%), Hungary (14, 10.1%), United States (14, 10.1%) and The Netherlands (11, 7.9%). Many respondents held multiple roles, for example 33 (23.1%) were guideline developers, implementers and researchers. Those who identified in a single role included 27(19%) developers, 8 (5.6%) implementers and 18 (12.6%) researchers.
Awareness of GIRAnet
Of 204 respondents, 116 (56.9%) were not aware of GIRAnet prior to the needs assessment survey. Those already aware of GIRAnet first learned about the network in a variety of ways. Most frequently mentioned means of communication included G-I-N web site (34, 38.6%), G-I-N newsletter (29, 33.0%), G-I-N annual meeting (25, 28.4%), from a colleague (17, 19.3%) and email from G-I-N (16, 18.2%).
Perceived impact of GItools
Ratings were mixed when respondents were asked whether GItools enabled guideline implementation: 42 (26.1%) and 96 (60.0%) agreed or strongly agreed that GItools on their own or combined with other strategies, respectively, enabled guideline implementation. Some thought that “Without GItools it would be exceedingly difficult to adequately implement a guideline…”. A variety of comments provide further insight on these findings. For example, several respondents said “I am not familiar with GItools”. Others believed that “…in most cases multiple tools and strategies are needed…” but the complementary strategies differed, including training, educational meetings, champions, guideline format, integration with electronic decision support systems, organizational factors, teamwork, and political will and financing. Others said that “More robust evidence is needed to show that these tools actually make a difference…”.
Relevance of GItools
Half of respondents agreed or strongly agreed that their organization was interested in GItools (77, 49.7%) while 42 (27.1%) agreed or strongly agreed that their organization had developed GItools. A total of 94 (60.6%) respondents said their organization was interested in learning more about GItools. For some respondents their organization may not be ready for GItools because no guidelines are developed in their country or they “are just starting in the guideline business”. Others said they “hope to adopt/adapt tools developed by other organizations”. Most comments were positive: “We have always been and remain interested in developing better ways to implement guideline recommendations”.
GIRAnet products of interest
Resources of interest included an implementability framework that defines and provides examples of different type of GItools (119, 82.6%); a GItool directory available on the Internet in which to search for and link to existing guideline-specific or generic GItools (104, 72.2%); a GItool development instructional manual describing the best practices and resources needed to developed and evaluate GItools (99, 68.8%); and a GItool assessment kit including definitions, example, instructions and application templates for criteria to assess the merits of GItools (82, 56.9%). Additional information of interested expressed in comments included “access to reports from organizations that have assessed the merits of GItools” and “adaptability of GItools to different settings”.
GIRAnet services of interest
Services to support information sharing about GItools that interested respondents included a web site featuring implementability information and products (127, 90.7%), workshop at the annual G-I-N meeting (89, 63.6%) and webinars on a variety of topics such as the implementability concept and framework, how to search the GItool directory, how to develop GItools, etc. (86, 61.4%). Two respondents urged that products and resources be integrated with the G-I-N web site.
GItool research priorities
Ongoing research recommended by respondents spanned a range of issues included evaluating the impact of GItools using quantitative means such as trials or time series (97, 69.8%) or qualitative approaches such as exploring how users interpret and apply GItools (89, 64.0%); developing and evaluating new GItools (78, 56.1%); developing and validating GItool assessment criteria (71, 51.1%); and exploring the methods and resources used by developers to create GItools (51, 36.7%).
Participation in GIRAnet
Respondents were interested in taking part in a variety of GIRAnet activities:
Activity / Description / Interested respondentsInterest Group / Receive periodic email updates / 99 (68.8%)
GItool Screening Panel / Review GItools using assessment criteria to determine eligibility for inclusion in online GItool Directory / 38 (26.4%)
Collaborator Group / Pilot-test GIRAnet products and GITools through interviews or survey; assist with research recruitment; participate in research by allowing existing or future guideline to be modified with prototype GItool / 34 (23.6%)
Partner Group / Disseminate GIRAnet products or communications within your organization; or as a developer of GItools participate in delivering educational services / 33 (22.9%)
Co-Investigator Group / For those with formal academic affiliations, assist with research related to the development of GItools or evaluation of their impact / 29 (20.1%)
Communication preferences
To learn about ongoing GIRAnet activities respondents recommended the following mechanisms: G-I-N newsletter (107, 75.9%), workshops at annual G-I-N meeting (80, 56.7%), GIRAnet newsletter (64, 45.4%), G-I-N web site (56, 39.7%), GIRAnet web site (52, 36.9%) and periodic webinar (40, 28.4%).
DISCUSSION
The findings of this needs assessment survey confirm the results of our previous research and the views of stakeholders with whom we consulted. GItools are a promising means of integrating implementation with development to improve guideline use. While further research is needed to definitively establish this, survey respondents recommended a number of resources be developed and shared via web site, G-I-N newsletter and annual meeting, and periodic webinars to enable the development and application of GItools. These include a GItool Directory, and instructions and criteria by which to develop high-quality GItools. The large number of respondents was encouraging, as was the number who newly joined the GIRAnet Interest Group now featuring 193 members, and expressing interest in active involvement in GIRAnet activities. Therefore during years two and three GIRAnet will focus on generated the recommended resources, and engaging international developers, implementers and researchers in distributed development and evaluation of GItools. We look forward to a continuing partnership with G-I-N, and to providing ongoing updates of GIRAnet activities and products.
REFERENCES
- Gagliardi AR, Brouwers MC, Palda VA, Lemieux-Charles L, Grimshaw JM. An exploration of how guideline developer capacity and guideline implementability influence implementation and adoption. Implement Sci 2009; 4: 36.
- Gagliardi AR, Brouwers MC, Palda VA, Lemieux-Charles L, Grimshaw JM. How can we improve guideline use? A conceptual framework of implementability. Implement Sci 2011; 6: 26.
- Gagliardi AR. “More bang for the buck”: exploring optimal approaches for guideline implementation through interviews with international developers. BMC Health Serv Res 2012; 12:404.
- Gagliardi AR, Brouwers MC. Integrating guideline development and implementation: Analysis of guideline development manual instructions for generating implementation advice. Implement Sci 2012; 7: 67.
1