House Officer Evaluation

Chapter 7

Faculty Development Series

Madigan Healthcare System

Tacoma, Washington 98431

House Officer Evaluation

Chapter Contents

Group Leader Checklist

Learner Needs and Resources Assessment (LNRA)

Attendance Sheet

Evaluation Form

Example Eight Steps

Handout

Summary of Supporting References and Resources

Checklist for the Group Leader

Before the Session....

___ 1.Review the suggested eight steps of planning for this presentation.

___ 2.Review the PowerPoint, handouts, and supporting references in this chapter.

___ 3.Duplicate and distribute the LNRA to faculty.

___ 4.Have faculty return the LNRA at least 5 days before the session.

___ 5.Review the faculty LNRA prior to the session.

___ 6.Modify the suggested eight steps and write your plan to fit your needs.

___ 7.Modify the PowerPoint and handout to fit your plan.

___ 8.Duplicate the appropriate number of copies of the attendance roster, faculty evaluation forms, and handouts.

During the Session....

___ 9.Have each participant signin using the attendance roster.

___ 10.Distribute the handout(s) to the participants.

___ 11.Conduct the session based on your eight steps of planning.

After the Session....

___ 12.Collect the evaluation forms from the faculty.

___ 13.Keep the attendance roster for the session in your department and provide the appropriate amount of CME to each participant.

___ 14.Reflect on the seminar How did it go? What was good about it? What could have been better? Is there a better approach to this topic? Were there needs identified during this session that would be the basis for future seminar(s) in your program?

___ 15.Where willyour program go from here based on this seminar?

Learner Needs and Resources Assessment

Please complete the following needs assessment for the upcoming seminar on House Officer Evaluation, as part of your faculty development program.

The seminar will consist of an introduction by your group leader, a short PowerPoint presentation, reviewingarticles with common standards, and a discussion period.

The purpose of this needs assessment is to determine yourlearning needs and interests, so that the seminar is most useful for you. This needs assessment should also stimulate you to think about evaluationsbefore the seminar begins. We need your enthusiastic participation now, and in the seminar. It will be fun, and at the end of it, we'll be asking for your feedback!

Please turn this in toyour group leader (______) no later than (______). Your group leader will return this form to you at the beginning of the session.

Have you had any formal training in evaluation? YES NO (Circle one)

In your opinion, what is the difference between a standard and a goal?

When planning an evaluation, what items do you think should be considered?

What are 4 main skills that you think should be evaluated?

1 3

2. 4.

5. Complete the following statement..."When I reflect on our department's evaluation system for

residents, I think it provides a/an ______assessment." (Circle one answer below)

Extremely Inaccurate Inaccurate Neutral Accurate Extremely Accurate

What are some types of errors that can be made in evaluations?

1. 3.

2. 4.

What tools are available for making evaluations in your department?

What 3 things do you most want to learn or discuss regarding evaluation?

Any other comments, concerns, or interests for this topic?

ATTENDANCE ROSTER – Page____of_____Pages

Department: ______Institution:______

Title of CME Activity: Faculty Development Workbook Series – House Officer Evaluations

Course Content: Didactic and Group Discussion

Instructor (Group Leader):______

Date:______Time: Began______Ended______Total ______

Name / Rank / Check One / Department or Mailing Address
Staff Physician / Resident Physician / Other Professional Discipline

Total Number of Learners Attending This Activity: ______

Faculty Development Session Evaluation Form

Date / Speaker / Topic

Please rate the speaker using the scale below:

Strongly Disagree / Disagree / Somewhat Agree / Agree / Strongly Agree
Content was relevant to my needs
The speaker conveyed the subject matter clearly
The speaker used active learning techniques to teach this subject
The speaker gave adequate time for questions
Audiovisual / handout material added to the presentation
Overall, the speaker was effective

List one thing that you learned from this presentation:

Please add your comments/suggestion for improving this session on the back—they are VERY helpful

Faculty Development Session Evaluation Form

Date / Speaker / Topic

Please rate the speaker using the scale below:

List one thing that you learned from this presentation:

Please add your comments/suggestion for improving this session on the back—they are VERY helpful.

House Officer Evaluations

Example Eight Steps

The eight steps presented below may be used as a guide for your planning.

Modify these steps to meet your specific needs.

Who:20 faculty learners from the Department of Family Medicine

Why:Enhance clinical teaching as part of a required faculty development curriculum

When:0730-0900 on a Wednesday morning, blocked schedule for faculty development

Where:Classroom, individual desks, accessible, AV supported, requires own computer

What:Driven by the LNRA and:

The importance of evaluations
Essentials of evaluations
Common skills assessed in resident evaluations
Tools of assessment that are valid and widespread

Barriers and difficulties of faculty evaluators

What For: By the end of this session, we will have:

Identified essentials of evaluations
Examined our definition of standards
Listed common skills to assess
Reviewed available tools to assist staff
Recognized potential pitfalls and biases
Practiced with cases

How:General: Active learning through small group activities and discussion, larger group discussion, minimal PowerPoint slides. Room contains individual desks. Will pre-configure desks into larger half-moon shape. Within larger half-moon configuration, will group four desks in smaller half-moon shape for five groups of four learners each. This will facilitate small group activities followed by larger group discussions. The session will take place at 0730, so will provide coffee and bagels.

Grabber: “You Be the Judge: 2011 X-Game BMX Big Air Competition”

Break the audience into 3 groups. Explain that each group will act as a judge for a competition. Explain that each team must give each competitor a score from 1-10 (lowest to highest). Do not offer any further assistance than these instructions. Show the crowd a series of short, 1-minute video clips of three separate competitors from the 2011 X-Games BMX Big Air Competition. Be sure to pause for 1 minute between each competitor to allow each team to assign a score.

If desired, the leader can tally the scoring and assign place rankings at the end. However, disregard the numerical values assigned, ask the group as a whole:

How did you come to your rating for each competitor?
What were some of the challenges that you discovered?
Did you notice any biases in your assessments?

Induction Task

Why is evaluation important? Encourage the audience to reflect on:

Why they think evaluation of our learners is important?
What problems have they encountered in the past with evaluations?

Input #1

Essentials of effective evaluations.

Activity #1

In pairs, have the audience discuss some of the things that they mentally take into account before writing a learner’s evaluation. Use the concept of including what standard to measure against to transition to the next input topic.

Input #2

Evaluations should be as objective as possible and aim to evaluate specific skills.

Offer specific examples from ACGME core competencies.

Activity #2

1)Have the audience break up into their 3 original groups from the Grabber exercise. Give them a deck of skill cards (included in this chapter). Ask the group to put the skills in a rank order from most important to least important. Have them write their rankings on butcher block or a white board to display.

2)Display the most common answers from most important skills on the LNRA on PowerPoint. Compare the LRNA answers to the group displays and comment on similarities and differences. Conclude that the group as a department likley has culturally weighted preferences of skills to evaluate but that all are important in different circumstances.

Input #3

Show audience LRNA results of their defintions of what a standard is. Ask the group if one of the examples speaks to them more than others.
Transition to generally accepted components of measurable standards.

Activity #3

In pairs, distribute copies of different standards that are employed at their departments (examples included in this chapter). Have them review the standards and ask the following questions relating to the key components of effective standards:

Are they written?
Are they shared with staff and residents at key time periods?
Are they understood?
Are they current and relevant? Have they been updated and improved recently?
Summarize findings and comment on possible refinement of some of these utilized standards.

Input and Task #4

Display a list of some common assessment tools that are available to faculty.

Ask the large group to discuss some of the pros and cons of a few of them.

Input #5

One of the main pitfalls in evaluation lies within evaluator bias.

Activity #5

Matching exercise of different bias types.
Ask the large group to comment on:
Which ones do they encounter more often
How do they avoid these biases?

Implementation Task

Have the audience get back into pairs. Distribute 2 blank evaluation sheets that their department uses for summative evaluations. Also distribute 2 cases for the pairs to read and discuss (examples included). Ask the groups to create an evaluation for each case.
Come together as a large group and ask the audience to comment on:
What things they took into account about the case prior to writing?
What skills were they evaluating?
What standards were they using to measure?
Where there potential biases involved?

Integration Tasks:

Display the results of the LRNA question regarding their feelings of how accurate their department’s evaluation system assesses its learners.
Ask the group (time permitting) to brainstorm to processes that could be used to improve evaluations. What personal changes will they commit to for upcoming evaluations?

So What:

Learning: Learners understand characteristics of effective evaluations. Introduced key components of evaluations and standards. Established that there are several types of evaluations that can target specific skills. Explored personal and organizational biases to avoid their impact on accuracy.

Transfer: Learners build upon this knowledge and improve their upcoming written evaluations. They will commit to improve their evaluations through increased planning pertaining to their student’s personal situation, specific skills, use of standards, and awareness of personal biases

Impact: Departmental evaluations are enhanced, department standards are reviewed, leading to improved learning and success for residents.

House Officer Evaluation

Handout

Opening Activity – Why Evaluate?

Learning Task 1 – With the End in Mind: Evaluation Essentials

Learning Task 2 – Ranking Our Learner’s Skills

Learning Task 3 – A Review of Our Standards

Learning Task 4 – Knowing Our Bias

___ Recent Incident Bias / A. Being overly critical
___ Central Tendency / B. Sitting on the fence of avoiding the extreme anchors on a scale for fear of being too strong
___ Extreme Response Bias / C. The opposite of central tendency bias. Respondents tend to mark extremes rather than those in between. It is difficult to know honest ratings from the halo effect
___ Affirmation/Yea-Saying Bias / D. A potential for negative bias against a trainee because of an isolated recent negative incident or statement, which does not necessarily reflect the usual work ethic of that person
___ Incompetence Bias / E. Being overly charitable
___ Leniency Bias / F. Occurs when evaluators assign high ratings because of lack of confidence or competence
___ Halo Effect / G. The tendency to give positive responses irrespective of their context, also known as inflation of ratings
___ Contrast Bias / H. A rater's overall impression of a person will affect his or her rating on each item
___ Stringency Bias / I. Rating against another person's performance rather than a standard

Examples of Additional Handouts Needed

Example of Skill Cards for Learning Task 2

Examples of Different Standards to Review in Learning Task 3

Minimum Program Requirements Language
Approved by the ACGME, September 28, 1999Educational Program
The residency program must require its residents to obtain competencies in the 6 areas below to the level expected of a new practitioner. Toward this end, programs must define the specific knowledge, skills, and attitudes required and provide educational experiences as needed in order for their residents to demonstrate:
Patient Care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health

Medical Knowledge about established and evolving biomedical, clinical, and cognate (e.g. epidemiological and social-behavioral) sciences and the application of this knowledge to patient care
Practice-Based Learning and Improvement that involves investigation and evaluation of their own patient care, appraisal and assimilation of scientific evidence, and improvements in patient care
Interpersonal and Communication Skills that result in effective information exchange and teaming with patients, their families, and other health professionals
Professionalism, as manifested through a commitment to carrying out professional responsibilities, adherence to ethical principles, and sensitivity to a diverse patient population
Systems-Based Practice, as manifested by actions that demonstrate an awareness of and responsiveness to the larger context and system of health care and the ability to effectively call on system resources to provide care that is of optimal value

Evaluation
Evaluation of Residents
The residency program must demonstrate that it has an effective plan for assessing resident performance throughout the program and for utilizing assessment results to improve resident performance. This plan should include:

Use of dependable measures to assess residents' competence in patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice
Mechanisms for providing regular and timely performance feedback to residents
A process involving use of assessment results to achieve progressive improvements in residents' competence and performance

Programs that do not have a set of measures in place must develop a plan for improving their evaluations and must demonstrate progress in implementing the plan.
Program Evaluation The residency program should use resident performance and outcome assessment results in their evaluation of the educational effectiveness of the residency program.

The residency program should have in place a process for using resident and performance assessment results together with other program evaluation results to improve the residency program.

Example Cases for Integration Task

Case 1

Dr. TB has an appropriate fund of knowledge and clinical skills for a second year resident. He recently received his ITE scores and was pleased with his 92nd percentile performance. However, during this past FM Clinic rotation he was frequently noted being late for work by the nursing staff. The FMIT attending mentioned in last week’s staff meeting that he left a significant amount of work for the incoming residents who were relieving him after an overnight call shift. His chart audits demonstrate adequate documentation; however, they were rarely closed out within the required 72 hour period. Despite these concerns, two staff attendings created voluntary precepting evals during this rotation containing very positive comments. One specifically noted his ability to organize his H&P during an oral presentation, offer a broad differential diagnosis, and devise an appropriate A/P. The second evaluation commended Dr. TB on his pleasant demeanor and effective negotiating skills with a patient threatening to call the patient advocacy office after she waited for 40 minutes in the waiting room.

Case 2

Dr. LB is a diligent resident who is half way done with her first year of Internship. She routinely arrives to work early and stays late. As her current attending physician, you have received several positive, unsolicited comments from the ward nurses about her professionalism with the nursing and nursing staff. You have personally been asked by one of the ward patients if she could change her PCM to Dr. LB because of her outstanding bedside manner. Yet, she has struggled on her current clinical rotation (FMIT), lagging behind her peers in her fund of knowledge and her ability to synthesize clinical laboratory, and radiographic data. Her daily progress notes reflect this with limited expansions of DDx and A/P. Her November ITE score placed her in the 50th percentile overall. Although still in the early stages of residency training, she is already involved in a research project with an attending staff. Upon discussion with LB, it was learned that she lacked basic time management skills and that she was having difficulty juggling her clinical training with her research interests and personal life.