Draft Submitted for Subcommittee Approval 7.2313

Draft submitted for subcommittee approval 7.2313

Quick Start Manual

Authors: Gloria Kuhn (subcommittee chair), Mark Clark, Sam Luber, Gillian Schmitz, Stephen Wolf (alphabetical order)

Contents:

Information and components of the Next Accreditation System (NAS):
Challenges and solutions to implementation of the milestones
Glossary of terms: Appendix A
Websites for more information: Appendix B

NAS

Next Accreditation System (NAS): A new system of accrediting residency training programs designed to foster innovation and improvement in the learning environment, increase the accreditation emphasis on educational outcomes, increase efficiency and reduce burden in accreditation, and improve communication and collaboration with key internal and external stakeholders. Additionally, the goals of the NAS are tofree good programs to innovate, to assist poor programs to improve, to reduce the burden of accreditation, and to provide accountability for outcomes to the Public.

Components of NAS: 1) Resident assessment through milestones that were envisioned to expand the description of the competencies into a progression of observable behaviors, that would allow assessment of whether residents had attained the knowledge, skills, attitudesand proficiency levels needed topractice their chosen field of medicine. This process will include 1) formation of a clinical competency committee to look at multiple sources of assessment data and report their findings on each resident semi-annually to the program director 2) periodic faculty surveys completed online regarding components of the training program, 3)periodic online resident surveys regarding components of their educational experience, 4) evaluation of the Clinical Learning Environment (CLE)to assess how the sponsoring institution fulfills its responsibility for the quality and safety of the environment for learning and patient careand the clinical learning environment review (CLER) that assesses the CLE.

Challenges and Solutions to Implementation of the Milestones

Lack of standardization in determining resident proficiency

Solution: Develop anchors, or specific behaviors, that define each proficiency level. Anchors provide benchmarks to define what would constitute advancement to the next level.
Solution: Create a standardized approach to daily shift evaluations that incorporate the milestones

Minimizing Grade Inflation

Solution: Consider an evaluation form with unique anchor language that clearly delineates behaviors needed to reach each level of competency. This can improve reliability between faculty members and improve feedback to residents.
Solution: Utilize multiple assessment techniques and evaluations from different faculty
Solution: Faculty development, feedback, and tracking may demonstrate to faculty how they grade in relation to their peers and may help reduce grade inflation.

Elimination Bias in Scoring Methods

Solution: Eliminate proficiency level identifiers. This identifier links the milestones to certain headers, such as “novice” or “postgraduate year (PGY)-1”. These introduce bias as evaluators may score to the level of expected, rather than actual, performance.
Solution: Consider binary scoring (ie yes/no) rather than traditional scaled scoring (i.e. a 5 point scale). Binary scoring may eliminate bias by simply assessing the presence or absence of a specific action. For example, “Did the resident place oxygen on the patient, Yes or No?” Binary scoring allows more accurate assessment of specific tasks but these behaviors may not be applicable for every patient interaction.
Solution: Consider using two different types of shift evaluation cards: one for novice and one for more experienced residents. This may simplify the evaluation by focusing on only the first two proficiency levels for less experienced residents. Once he or she consistently meets requirements of novice card, he or she is then evaluated with an advanced card, which focuses on proficiency levels 3-5.

Incorporating Milestones into Evaluation Forms

Solution: Some vendors are now linking evaluations with specific sub-competencies and milestones
Solution: Develop shift evaluation cards that measure a few specific sub-competencies on every shift

Obtaining Sufficient Data within each Sub-competency

Solution: Randomize the sub-competencies such that 4 or 5 are evaluated after each shift. Alternate shift cards weekly, monthly. Etc.

Solution: Create specific scenarios in simulation lab or standardized patients to allow assessments of sub-competencies that are not frequently encountered on a clinical shift
Solution: Empower residents to be responsible for distributing their own evaluations and ensuring they have adequate responses on each sub-competency
Solution: Creating a Performance Dashboard that highlights how many evaluations each resident has for each of the 23 sub-competencies.

Working with Residency Software Management Systems to provide feedback

on future product development that better meet needs of emergency medicine educators.. Encourage innovation, collaboration, and communication.

Solution: A Milestone “show and tell” will be highlighted at the Joint Milestone Task Force at Scientific Assembly for vendors to participate in a moderated panel discussion and demonstrate their products.

Development of Validity Evidence

More research is needed to support validity evidence of current performance evaluations and implementation of milestones.

Glossary of Terms

Appendix A

ABMS: American Board of Medical Specialties

ACGME: Accreditation Council for Graduate Medical Education

ADS: Accreditation Data System (also known as WebADS) is a Web-based system that contains critical accreditation data for all sponsoring institutions and programs. Serves as an ongoing communication tool with programs and sponsoring institutions and incorporates several ACGME applications and functions. Programs are required to enter program specific data (hospital ED census data, resident procedure averages, etc) into WebADS at a minimum annually.

Clinical Competency Committee (CCC): A committee created by a residency program designed to make consensus decisions on the progress of each resident; CCC members should include core faculty members who have the opportunity to observe and evaluate residents, but may include other members such as assessment specialists and non-MD medical educators. The main task of the CCC will be to determine, based on multiple types of assessment data, whether a resident has met the milestones and will make recommendations to the PD regarding progression.

Clinical Learning Environment Review Program (CLER): CLER focuses on six areas important to the safety and quality of care in teaching hospitals and the care residents will provide in a lifetime of

practice after completion of education: 1) engagement of residents in patient safety; 2) engagement of residents in quality improvement (including opportunities for reducing health disparities); 3) enhancing practice for care transitions; 4) promoting appropriate resident supervision; 5) duty hour oversight and fatigue management; and, 6) enhancing professionalism in the learning environment and in reporting to the ACGME

Common Program Requirements: Defined program requirements as set forth by the ACGME that programs must adhere to; these requirements are subcategorized into:

Core Requirements:Statements that define structure, resource, or process elements essential to every graduate medical educational program.

Detail Requirements:Statements that describe a specific structure, resource, or process, for achieving compliance with a Core Requirement. Programs in substantial compliance with the Outcome Requirements may utilize alternative or innovative approaches to meet Core Requirements.

Outcome Requirements:Statements that specify expected measurable or observable attributes (knowledge, abilities, skills, or attitudes) of residents or fellows at key stages of their graduate medical education.

Core Competency: One of the 6 competencies defined by the ACGME on which physicians in training should be assessed: patient care; medical knowledge; practice-based learning and improvement; interpersonal and communication skills; professionalism; and systems-based practice.

Educational Milestones: Observable developmental steps, organized under the six competency areas, that describe a trajectory of progress on the competencies from novice (entering resident) to proficient (graduating resident) and, ultimately, to expert/master (expert/master level usually achieved after graduation).

Faculty Survey:an annual survey administered by the ACGME in which core faculty are anonymously asked about the program in the areas of: faculty supervision and teaching, educational content, resources, patient safety, and teamwork. A 60% response rate is required by the ACGME.

Focused Site Visit:Assesses selected aspects of a program and may be used to address potential problems identified during review of annually submitted data,to diagnose factors underlying deterioration in a program’s performance, and to evaluate a complaint against a program

GME: graduate medical education

Next Accreditation System (NAS): A new system of accrediting residency training programs in designed to foster innovation and improvement in the learning environment, increase the accreditation emphasis on educational outcomes, increase efficiency and reduce burden in accreditation and improve communication and collaboration with key internal and external stakeholders. Major components of the NAS include milestone assessment, clinical competency committees and the Clinical Learning Environment Review Program (CLER). Additionally, the goals of the NAS are to begin the realization of the promise of Outcomes, to free good programs to innovate, to assist poor programs to improve, to reduce the burden of accreditation, and to provide accountability for outcomes to the Public.

Phase 1 Specialties: Specialties that will enter the NAS in July 2013: Diagnostic Radiology, Emergency Medicine, Internal Medicine, Neurological Surgery, Orthopaedic Surgery, Pediatrics, Urology

Program Information Form (PIF): The PIF was a standardized reporting tool for residencies to demonstrate compliance with ACGME Core and Specialty-specific program requirements. With the introduction of the NAS, PIFs will no longer be used for continued accreditation.

Resident Survey: an annual survey administered by the ACGME in which residents are anonymously asked about the program in the areas of: duty-hours, faculty, evaluation, educational content, resources, patient safety and teamwork. A 70% response rate is required by the ACGME.

RRC:Residency Review Committee (of the ACGME). RRCs are setup for each specialty and are tasked with reviewing/approving applications for new residency programs as well as approving continued accreditation of established programs

Self-Study Visit: A visit by the ACGME that will be scheduled every 10 years. It will be conducted by a team of visitors who will interview residents, faculty and leadership. The goal of the visit is to foster continuous improvement in the program, learn future goals of the program and verify compliance with core requirements. This visit will take the place of the site-surveys conducted in the past.

Subcompetency: Defined competencies under one of the 6 core competencies in which educational milestones have been developed.

WEBSITES:

Appendix B

JMTF

“The Wiki” is a website created by members of CORD who formed the Joint Milestone Task Force (JMTF). The purpose of the website is to help develop resources for residency programs around the country. It is open access to all who wish to obtain information. To actually join the group and place materials on the wiki access must be requested. It has a wealth of information including articles, answers to frequently asked questions, and checklists to evaluate procedural competency. The site is well organized to enable materials to be easily found.

ACGME

The ACGME has created multiple materials on their website relating to NAS. The site has webinars explaining NAS which at this time are accessible only to program directors, however the webinar slides are open access. The easiest way to gain information is to go to and search the site for information related to either NAS or milestones.

Link to EM Milestones:

ACGME/ NAS Website:

CORD

CORD website:

End of Shift Milestone Evaluation Forms:

You can create a “test” shift evaluation using the following administrative ID and password. Click on "For Residencies," and use 3530 as the Administrator ID, and 21528 as the administrative password. Then click on "Milestones" on the far right. You are taken to a screen where you can download the Word documents, enter in forms, or generate reports. The end of shift evaluation forms are easy to use.

WEBINARS

ACGME Phase I Webinar Series (note: the webinars are not accessible to all faculty at this time. Program directors can access by logging in.)

ACGME Phase I: Emergency Medicine

June 13, 2013

11:00-12:30 p.m. Central Time

Speakers: Louis Ling, MD—Senior Vice President, Hospital-based Accreditation, ACGME

Wallace Carter, MD—Chair, ACGME Review Committee for Emergency Medicine

Michael Beeson, MD—Vice-Chair, ACGME Review Committee for Emergency Medicine;

Chair, Milestone Working Group

This is a webinar to update the emergency medicine community about the NAS, which goes into effect July 1, 2013. Drs. Carter, Beeson, and Ling will discuss recent questions and clarifications about the NAS including: 1. using data as a screening tool for the Review Committee; and, 2. the Program Requirements for the Clinical Competency Committee (CCC) and the Program Evaluation Committee (PEC).

This session will assume that the audience has a basic background of the NAS.