Elements of Continuous Internship Program Evaluation

Jan Willer, Ph.D.

Director of Training

VA Chicago Health Care System, Chicago, IL

Presented at APPIC Annual Conference, New Orleans, LA, March 30-31, 2001.

Overview

An essential element of any psychology internship training program is an appropriate program evaluation plan. Evaluation of various elements of the training program are required by the accreditation criteria (see below). However, the operationalization and implementation is delegated to the training program. So, how are these evaluations best accomplished by the training program? This poster provides a format and tools for continuous program evaluation throughout the training year, which can be readily implemented and fulfills the requirements of the accreditation criteria. Site visitors from the Committee on Accreditation recently complemented this program evaluation format for its thoroughness and completeness.

Relevant Accreditation Criteria

A.5 “The program engages in actions that indicate respect for and understanding of cultural and individual diversity. This is reflected in the program’s policies for the recruitment, retention and development of staff and interns and in didactic and experiential training . . .”

B.1 “Training for practice is sequential, cumulative and graded in complexity.”

B.3(b) “. . . socialization into the profession of psychology . . . mentoring, didactic exposure, role-modeling and enactment, observational/vicarious learning, supervisory or consultative guidance.”

B3(d) “The content of internship training activities addresses the application of psychological concepts and current scientific knowledge, principles and theories to the professional delivery of psychological services to the consumer public; professional conduct and ethics; and the standards for providers of psychological services.”

B.4 “. . . the program requires that all interns demonstrate an intermediate to advanced level of professional psychological skills, abilities, proficiencies, competencies and knowledge in the areas of . . .”

B.5 “. . . The program . . . promotes the integration of practice and scholarly inquiry.”

C.1 “ . . . intern training supervisors who are of appropriate quality . . . serve as professional role models”

C.2(f) “. . . interns who have meaningful involvement in . . . decisions that serve to enhance internship training and education . . .”

D.1 “The program has made . . . efforts to attract and retain interns and staff from differing . . . backgrounds . . .”

F.1 “The program engages in regular ongoing self-studies that address . . . achievements . . . outcomes”

Key Areas to Assess and Document, Derived from Accreditation Criteria

  • Provide written feedback on intern competencies in key areas, at least semiannually
  • Assess program’s success at addressing issues of cultural diversity relevant to training
  • Assess quality of training, including supervision, didactics and role modeling
  • Evaluate outcomes of the internship—interpretation:

Document that interns have reached an appropriate competency level by the end of internship

Document positive outcomes for graduates’ job placements and career paths

  • Perform regular on-going self-studies of the internship program
  • Document improvements made to the program from intern input

Implementation

Psychology Intern Competency Assessment Form

Format: questionnaire filled out by every supervisor on each intern supervised every 4-6 months.

Frequency: every 4-6 months

Accreditation Goals Fulfilled: document that interns are adequately trained, document that interns achieve specified competency levels by graduation, provide for development of remedial plan, if needed,

provide written feedback on intern competencies in key areas.

Characteristics:

  • Checklist format, easy to use;
  • Addresses all key competency areas, especially including those mentioned in the accreditation criteria;
  • Is specific enough to discourage “grade inflation;”
  • Clearly specifies criteria for successful completion of training experience;
  • Informs trainees of the appropriate standards of competency and relevant behaviors;
  • Provides for development of remedial plan, if needed;
  • Has a highest rating level that reflects realistic standards, not idealized performances;
  • Is suitable for use in scientist-practitioner settings, addressing both clinical and research competencies.
  • Provides opportunities for written feedback on strengths and areas of future development; and
  • Addresses competencies in individual diversity.

Summary of areas assessed:

  • General Professional Competencies: application of ethical principles, seeking consultation as appropriate, professional interactions, taking on responsibility/self-direction, positive coping strategies, rapport, sensitivity to cultural and other individual differences and gathering interview data/confidentiality;
  • Competence in Psychological Assessment: administration, interpretation and report-writing, with recommendations;
  • Competence in Psychotherapeutic Interventions: case conceptualizations, treatment goals, appropriateness of interventions, and use of countertransference;
  • Competence in Group Psychotherapy Skills;
  • Competence in Professional Consultation; and
  • Scholarly and Research Competencies.

Supervisor Feedback Form

Format: questionnaire filled out by every intern on each supervisor every 4-6 months.

Frequency: every 4-6 months

Accreditation Goals Fulfilled: evaluate and maintain quality of training, assess program’s success at addressing cultural diversity issues, document intern input and feedback in key areas.

Characteristics:

  • Checklist format, easy to use;
  • Addresses quality of supervision in areas in accreditation criteria, including role modeling, observation, guidance and so on;
  • Is specific enough to discourage “grade inflation;”
  • Reminds supervisors of appropriate standards;
  • Has a highest rating level that reflects realistic standards, not idealized performances;
  • Provides opportunities for written feedback from interns;
  • Is suitable for use in scientist-practitioner settings, addressing both clinical and research supervisory skills; and
  • Addresses supervisory competencies in individual diversity.

Summary of areas assessed:

  • Supervisory Responsibilities: being on time, availability, documentation
  • Supervisory Content: ethical issues, diversity issues, risk management, case material
  • Supervisory Process: communication, trust, comfort, fostering growth
  • Assistance in Professional Development: role-modeling, flexibility, allowing autonomy, professional identity
  • Assistance in Development as a Scientist-Practitioner: knowledge of recent research, guidance to literature

Evaluation of Didactic Presentations

Format: filled out by interns to assess the quality of didactic presentations.

Frequency: every intern fills it out after every didactic presentation.

Accreditation Goals Fulfilled: document intern input and feedback, assess program’s success at addressing cultural diversity issues, evaluate and maintain quality of training

Characteristics:

  • Checklist format, easy to use;
  • Provides opportunities for written feedback from interns that presenters use to improve their didactics;
  • Evaluates various aspects of presentation, including whether relevant individual diversity issues were addressed.

Annual Self-Study Report

Format: report written by the director of training.

Frequency: yearly, towards the end of the training year, after internship selection process.

Accreditation Goals Fulfilled: document intern input and feedback, assess and document program’s success at addressing cultural diversity in didactics and recruitment, evaluate and maintain quality of training, document interns’ achievements and career outcomes.

Summary of areas assessed:

  • Internship Preparedness and Selection, including diversity recruitment. Yearly statistics are kept on the number of minority candidates interviewed.
  • Intern Training, including changes made to various aspects of the training program in the past training year, especially those made in response to intern feedback.
  • Communication and Feedback
  • Internship Graduate Outcome Study of former interns’ career paths since the last site visit.

Optional and Additional Elements

  • End of year meeting of training director with interns for verbal overview of entire didactic series

Synergy of group provides useful information regarding the series.

  • Monthly meetings between internship class and training director
  • Questionnaire regarding all aspects of internship administered at the end of the training year
  • All interns write a White Paper about the whole internship experience together at the end of the training year.
  • Administer a survey to internship applicants who did not choose or were not matched to the site to evaluate the interview process.

To Get Copies of Sample Instruments and Reports

To get copies, send me an e-mail at . I will send them to you as an attachment. Please specify exactly which instruments you are interested in getting copies of.