Seeing the Body Elsewise: Promoting Evaluation in the University of Michigan’s Program in Culture, Health and Medicine
Mary E. Piontek, Ph.D., Evaluation Researcher
The Center for Research on Learning and Teaching, University of Michigan
3300 School of Education Building, 610 East University
Ann Arbor MI 48109-1259
E-mail: website: www.crlt.umich.edu
Overview
Seeing the Body Elsewise: Connecting the Pre-Health Sciences and the Humanities grant project of the University of Michigan's Program in Culture, Health, and Medicine rethinks the ways cultural diversity is taught in pre-health education. The project includes an interdisciplinary model for teaching pre-health undergraduate students (pre-medicine, pre-nursing, pre-life sciences) about the intersections of race, gender, health, and ethnicity.
This paper presentation will focus on one of the project’s evaluation activities, the development and implementation of a survey instrument, that supported three central goals of the project’s evaluation design: (1) gathering formative quantitative and qualitative evaluation data to clarify further the overarching goals of the project and inform decisions about improving the quality of the project's course innovations, public programs, and other project activities; (2) improving the evaluation instruments including the Cultural Bases of Health Survey developed by the project; and (3) evaluating the impact on individuals' attitudes about the effects of culture on the issues of medicine and health. Dr. Mary Piontek of the University of Michigan’s Center for Research on Learning and Teaching (CRLT) joined the project as the internal evaluator in January 2002 (Year 2 of the grant). The internal evaluator’s major responsibilities are to design, implement, analyze, and report on all evaluation activities related to the project.
The project Seeing the Body Elsewise: Connecting the Pre-Health Sciences and Humanities is funded by the U.S. Department of Education’s Fund for the Improvement of Postsecondary Education (P116B001807). The project began in November 2000 and ends in October 2003. The project directors are Dr. Jonathan Metzl, Department of Psychiatry/Program in Women’s Studies () and Dr. John Carson, Department of History () at the University of Michigan, Ann Arbor, Michigan. Partners in the project include Michigan State University (MSU) and Wayne State University (WSU). The project acknowledges the collaboration of the following individuals: Marcia Inhorn (UM, Anthropology/Health Behavior & Health Education), Peggy McCracken (UM, Women’s Studies/Romance Languages), Martin Pernick (UM, History), Nancy Rose Hunt (UM, History/CAAS), Sidonie Smith (UM, Women's Studies/English), Tricia Tang (UM, Medical Education), Silke-Maria Weineck (UM, German/Comparative Literature), Judith Moldenhauer (WSU, Information Design/Graphic Art), and Steve Rachman (MSU, English/Center for Medical Ethics). The project’s website is http://www.umich.edu/~pchm
Project Background
Cultural competency has become an increasingly important component of medical education. However, current efforts to impart understandings of the cultural and cross-cultural aspects of medicine are rarely components of pre-health undergraduate education. Pre-nursing and pre-medical undergraduates learn about the organic, chemical, and biochemical aspects of disease, but an understanding of the cultural aspects of these illnesses often begins only in professional school, residency, or in post-graduate degree professional education. Undergraduate education provides an excellent opportunity to teach students about cultures at the same time that they begin to learn about diseases and bodies. Undergraduate education also allows for a unique emphasis on written and verbal communication, self-reflection, critical reading, and comparative thinking that will later become vital to their lives as practitioners of the medical sciences.
Housed jointly in the Institute for Research on Women and Gender and the Department of Psychiatry at the University of Michigan in Ann Arbor, Michigan, the Seeing the Body Elsewise project partners with humanities and medical faculty from the University of Michigan (UM), Michigan State University (MSU), and Wayne State University (WSU). The project consists of (a) a series of semester-long, interdisciplinary courses, (b) a public lecture series, and (c) dissemination activities to expose undergraduate students to issues of cultural competency at an early stage in their pre-health/pre-medical academic studies.
Three central goals of the project’s evaluation design were to gather formative quantitative and qualitative evaluation data to (1) clarify further the overarching goals of the project and inform decisions about improving the quality of the project’s course innovations, public programs, and other project activities; (2) improve the instruments and other methods, both qualitative and quantitative, that are used to assess changes in the target population’s response to issues of cultural competence in the health-care professions; and (3) assess the impact on the target population’s response to issues of cultural competence in health care professions and the extent of progress towards the project’s outcomes. The development and implementation of a survey (Cultural Bases of Health Survey) of students in the courses associated with the project was one of many evaluation activities that contributed to meeting these three evaluation goals.
Courses offered at the University of Michigan, Michigan State University, and Wayne State University associated with the project included such titles as Global Health: Anthropological Perspectives; Gender, Medicine and Culture in U.S. History; Health and Illness in African Worlds; Information Design and Visualizing the Human Body; Women, Autobiography, and the Medical Body; Literature, Medicine, and the Diagnostic Dialogue; Interdisciplinary Approaches to Men's Health; and History of the Human Sciences. Each course encouraged ongoing exchange between medical and humanities perspectives, and focused on illuminating understandings of health and illness within a variety of geographical, structural, and historical locations. Many of the courses were team-taught by humanities and health-sciences faculty, or involved lectures by practitioners from both of these disciplinary backgrounds. Each of these courses was open to undergraduate students; some courses also enrolled graduate-level students.
Evaluation Activities
Throughout the project, the project directors, project staff, and core faculty have used formative evaluation information and their professional expertise to continuously reflected on the current design of the project and developed future activities to improve and expand the project. Periodic meetings of the faculty advisory group, which also included the two project directors, project staff, and the internal evaluation consultant, were held to discuss the project’s overall goals and central activities. The focus of the meetings included discussion of the course offerings; evaluation data collected via the project-developed student survey (Cultural Bases of Health Survey) and focus groups from students in the courses associated with the Seeing the Body Elsewise project; strategies for institutionalizing all or parts of the project at the participating institutions of Higher Education; the design for public presentations, speaker series, and future conferences; and the further review and revision of the definition of cross-cultural competency as it relates to the health professions.
Development of the Cultural Bases of Health Survey
As part of the evaluation of the project, in Year 1 (2000-2001) the two project directors set out to devise a survey instrument that was at once broad enough to appreciate the project’s diverse topics and methodologies, and specific enough to determine the effects of our curricular interventions. The ultimate goal was to develop an assessment tool that would illuminate undergraduate student attitudes about the effects of culture on the issues of medicine and health, and that could be used to mark changes in those attitudes in response to a range of pedagogical interventions. The internal evaluator joined the project in Year 2 of its implementation and is providing the results of the data analysis of the Cultural Bases of Health Survey (CBHS). The qualitative and quantitative data analysis of the survey is proving valuable in understanding of the sensitivity and specificity of the instrument in assessing curricular outcomes. Refer to Appendix A for a copy of the Cultural Bases of Health Survey instrument.
The project directors and evaluator recognize that the basic identity structures of the target population of undergraduate students will not be significantly changed by the limited interventions available in such a project (one-semester undergraduate-level courses in multiple curricular areas), and thus the focus of the project’s efforts are on gaining deeper understanding of how to teach undergraduates the skills necessary to appreciate the complexities of identity and the relations of identity to health to produce a measurable change in attitudes. Thus far, the project has evaluated the cross-cultural skills, expectations, and beliefs of the students in the curriculum (~300 across Years 1-3) using quantitative and qualitative materials developed by the project directors and evaluator. The experience with the CBHS at the University of Michigan, Michigan State University, and Wayne State University suggests that the instrument is a unique addition to the assessment tools currently available to evaluate curricular changes designed to enhance pre-health students’ awareness of cultural diversity issues in healthcare.
As a first step in the instrument’s development, the project directors conducted a comprehensive review of the existing literature on cross-cultural competency in relation to medicine. They found that most evaluation tools focus on graduate-level or medical school personnel, or assess satisfaction of program participants in sociocultural medicine programs. Few instruments exist that assess knowledge acquisition or changes in students’ attitudes/awareness of cultural competency issues at the undergraduate level. Adapting the concepts related to cultural competency from the literature and existing evaluation tools including the Sociocultural Attitudes in Medicine Inventory: SAMI developed by researchers at the University of Michigan to assess student attitudes towards sociocultural issues in medicine and patient care in a medical school setting, the project directors designed the CBHS to be a health-specific instrument that would illuminate changes in student attitudes about the effects of culture on issues of medicine and health (Tang, TS, Fantone, JC, Bozynski, ME, Adams, BS. Implementation and evaluation of an undergraduate sociocultural medicine program. Academic Medicine. 2002. 77: 578-85). The instrument is still in the preliminary phase of development and validation, but is valuable means of providing objective assessments of changes in attitudes and skills related to the connections between culture and health.
The CBHS instrument consists of three close-ended and one open-ended demographic questions; 35 close-ended, Likert-scale cultural competency questions; and one open-ended clinical case vignette question. Section one of the CBHS is a general survey of undergraduate student attitudes and knowledge about culture and medicine, based on the Association of American Medical College’s (AAMC) definition of cultural competencies. Questions range from self-assessment of knowledge of and interest in sociocultural issues (My overall knowledge about sociocultural issues, beliefs, attitudes, and practices in health and medicine is adequate; I have a good understanding of the relationship between sociocultural background, health, and medicine; I am interested in reading/learning more about the socioculture aspects of health and medicine); to beliefs about race and racial categories (Genetic variations largely explain differences in the incidence and mortality rates of diseases in the U.S.; Racial and ethnic disparities in health arise largely from differences in sociocultural circumstances); to specific beliefs about cross-cultural medicine (Health care professionals are better judges of how a person’s sociocultural background can influence health and health care decisions than laypeople; Learning about a person’s sociocultural background is critical in treating that person; Understanding the sociocultural background of patients is as important as knowing the medical aspects of their condition). Also included in the CBHS instrument are a series of closed-ended questions that assess student beliefs about the importance of sociocultural background on health outcomes and healthcare utilization (Sociocultural background has a significant effect on treatment decisions; …health status; …treatment compliance).
Section two of the Cultural Bases of Health Survey is a mock scenario describing an interaction between Cuban, Cuban-American, Caucasian-American, medical, and HMO cultures in the setting of a medical examination room: The one open-ended and twelve closed-ended questions following the scenario are designed to detect skills appropriate to undergraduate students who, while interested in matters of health and health care, are not yet responsible for treating patients. The open-ended question (Please write a few sentences discussing the most significant features of this clinical encounter) gathers students’ written reactions about the encounter between a physician, a patient, and a member of the patient’s family in a medical context that suggests issues of cross-cultural interaction. This scenario provides specific details about the physician/patient interaction, but allows for multiple interpretations of the scenario. The open-ended format of the question allows the respondents to choose the features they find most significant about the clinical case and to describe those features using their own choice of vocabulary. This question, along with the twelve closed-ended ones, highlights such student interpretive skills as the ability to detect ambiguity, to read closely, and to conduct textual analysis that the pre-health courses in the curriculum are intended to impart.
Implementation and Analysis of the Cultural Bases of Health Survey
For each course, the survey data on each of the 35 questions of the CBHS was entered in an Excel spreadsheet file. Individual respondents were identified using a four digit numerical ID code. The course-level data sets were then combined into two additional cross-course data sets: one full set of all responses and one set of all paired respondents within each academic year. For each of the courses, a paired set of pre- and post-survey data set was developed from the total number of students completing the CBHS. Since the project is primarily interested in the degree to which there was a significant change in the responses after completion of the courses, the data analysis focused on a question-by-question analysis of the overall change in pre- and post-course responses. For each of the 35 questions a pre-test mean, a post-test mean, and a mean shift (post-test mean minus pre-test mean) were calculated. An independent sample, equal variance, two-tailed t-test was used to analyze the full set of all respondents using a p-value of .05. For the paired sets within each of the courses and the set of all paired respondents a paired-sample, two-tailed test with a p-value of .05 was used in the analysis. Refer to Tables 1 and 2 for example findings of 2001-2002 survey data.
In addition to the statistical analysis completed on the 35 close-ended, Likert-scale cultural competency questions of the CBHS, the written responses to the one open-ended clinical case vignette question were coded and analyzed by Dr. Piontek. The purpose of the clinical case vignette question is to gather students’ written reactions about an encounter between a physician, a patient, and a member of the patient’s family in a medical context that suggests issues of cross-cultural interaction. The vignette presents a scenario that provides specific details about the physician/patient interaction, but allows for multiple interpretations about the scenario. The open-ended format of the question allows the respondents to choose the features they find most significant about the clinical case and to describe those features using their own choice of vocabulary. The written responses were coded and analyzed using a qualitative software program, QSR N-Vivo.