General Education Assessment Report (2011-2017)

General Education Assessment Report (2011-2017)

University of Mary Washington

General Education Assessment Report (2011-2017)

Human Experience and Society (HES)

Debra Schleef, Institutional Analysis and Effectiveness

Don Lee, Economics; Caitie Finlayson, Geography; Eric Bonds, Sociology

Introduction

The Human Experience and Society requirement of General Education provides an opportunity for Mary Washington graduates to inhabit and negotiate a world that requires a sophisticated understanding of the nature and dynamics of human conduct, organization and change. Through this experience, students explore the complex forces shaping human activity, cultures, communities, interactions, intellectual systems, and political, economic, and social institutions. The study of human experience and society helps students better understand the role of cooperation and conflict in society and in social relationships. Understanding the forces that impel and interfere with human connections fosters appreciation for the complexities involved in social decision-making, encourages exploration of alternatives for addressing social barriers, confronts the ethical questions which stem from the choices involved in human engagement, and ultimately, prepares graduates to be effective decision-makers in their own communities.

The Human Experience and Society requirement prepares students to see how theoretical, quantitative, and/or qualitative modes of inquiry are applied to the study of human behavior, social interaction, and institutions. Students will learn to critique the collection, reporting, and analysis of evidence on the human and social activities being explored, and examine their broader cultural and historical context. Given the variety of approaches to studying human experience and society and the virtue inherent in studying the issues involved from multiple perspectives, students are required to complete two courses from two different disciplines. In this way they can explore the commonalities and differences of the human experience.

Methodology for Assessment and Data Analysis

The following learning outcomes are the Human Experience and Society (HES) learning outcomes that have been assessed in most years:

  1. Students will be able to explain human and social experiences and activities from multiple perspectives.
  2. Students will be able to draw appropriate conclusions based on evidence.
  3. Students will be able to apply knowledge and skills learned to a novel situation.

For each course selected, the instructors assessed students’ mastery of the learning outcomes using one of two assessment methods. The first assessment method involved the use of a committee-developed rubric that scored a writing assignment or an examination completed by all or a random sample of students on each of the three learning outcomes on a scale of 1 to 4. The second assessment method, used only in 2012, allowed instructors to create a novel scenario unique to each discipline for students to read, and then answer six multiple-choice questions with two questions assigned to each learning outcome.

2011-12 (Method 1) / HES_1 / HES_2 / HES_3
Freq. / % / Freq. / % / Freq. / %
Unacceptable (1 point) / 37 / 7.8% / 43 / 9.2% / 44 / 9.2%
Fair (2 points) / 140 / 29.4% / 115 / 24.5% / 127 / 26.5%
Good (3 points) / 208 / 43.6% / 192 / 40.9% / 198 / 41.3%
Excellent (4 points) / 92 / 19.3% / 119 / 25.4% / 110 / 23.0%
Total (without missing) / 477 / 469 / 479
Mean (Scale 1-4) / 2.74 / 2.83 / 2.78
SD / 0.86 / 0.91 / 0.90
% with "Fair" and above / 92.2% / 90.8% / 90.8%
2011-12 (Method 2) / HES_1 / HES_2 / HES_3
Freq. / % / Freq. / % / Freq. / %
None correct (0 point) / 59 / 15.8% / 27 / 7.2% / 17 / 4.5%
One correct (1 point) / 123 / 32.9% / 129 / 34.5% / 102 / 27.3%
Both correct (2 points) / 192 / 51.3% / 218 / 58.3% / 255 / 68.2%
Total / 374 / 374 / 374
Mean (Scale 1-4) / 1.36 / 1.51 / 1.64
SD / 0.74 / 0.63 / 0.57
% with at least one point / 84.2% / 92.8% / 95.5%
2012-2013 / HES_1 / HES_2 / HES_3
Freq. / % / Freq. / % / Freq. / %
Unacceptable (1 point) / 3 / 5.3% / 2 / 3.5% / 3 / 5.3%
Fair (2 points) / 10 / 17.5% / 10 / 17.5% / 14 / 24.6%
Good (3 points) / 18 / 31.6% / 15 / 26.3% / 14 / 24.6%
Excellent (4 points) / 26 / 45.6% / 30 / 52.6% / 26 / 45.6%
Total / 57 / 57 / 57
Mean (Scale 1-4) / 3.18 / 3.28 / 3.11
SD / 0.91 / 0.88 / 0.96
% with "Fair" and above / 94.7% / 96.5% / 94.7%
2013-2014 / HES_1 / HES_2 / HES_3
Freq. / % / Freq. / % / Freq. / %
Unacceptable (1 point) / 11 / 3.3% / 15 / 4.5% / 13 / 9.1%
Fair (2 points) / 63 / 18.7% / 73 / 21.7% / 79 / 26.4%
Good (3 points) / 130 / 38.6% / 138 / 40.9% / 136 / 41.2%
Excellent (4 points) / 133 / 39.5% / 111 / 32.9% / 109 / 22.9%
Total / 337 / 337 / 337
Mean (Scale 1-4) / 3.14 / 3.02 / 3.01
SD / 0.83 / 0.85 / 0.84
% with "Fair" and above / 96.7% / 95.5% / 90.9%
2015-2016 / HES_1 / HES_2 / HES_3
Freq. / % / Freq. / % / Freq. / %
Unacceptable (1 point) / 51 / 26.6% / 26 / 13.0% / 13 / 6.7%
Fair (2 points) / 59 / 30.7% / 68 / 34.0% / 50 / 25.6%
Good (3 points) / 53 / 27.6% / 67 / 33.5% / 85 / 43.6%
Excellent (4 points) / 29 / 15.1% / 39 / 19.5% / 47 / 24.1%
Total / 192 / 200 / 195
Mean (Scale 1-4) / 2.31 / 2.60 / 2.85
SD / 1.03 / .95 / 0.86
% with "Fair" and above / 73.4% / 87.0% / 93.3%
2016-2017 / HES_1 / HES_2 / HES_3
Freq. / % / Freq. / % / Freq. / %
Unacceptable (1 point) / 15 / 4.4% / 27 / 8.4% / 32 / 9.4%
Fair (2 points) / 62 / 18.3% / 86 / 26.6% / 82 / 24.0%
Good (3 points) / 152 / 44.8% / 100 / 31.0% / 106 / 31.0%
Excellent (4 points) / 110 / 32.4% / 110 / 34.1% / 122 / 35.7%
Total / 339 / 323 / 342
Mean (Scale 1-4) / 3.05 / 2.91 / 2.93
SD / 0.83 / 0.97 / 0.98
% with "Fair" and above / 95.6% / 91.6% / 90.6%

Overall Results by Year (2011-2017)

Average scores / Mean HES-1 (Scale 1-4) / Mean HES-2 (Scale 1-4) / Mean HES-3 (Scale 1-4)
2011-12 / 2.74 / 2.83 / 2.78
2012-13 / 3.18 / 3.28 / 3.11
2013-14 / 3.14 / 3.02 / 3.01
2015-16 / 2.31 / 2.60 / 2.85
2016-17 / 3.05 / 2.91 / 2.93
% Meeting expectations and above / HES-1 / HES-2 / HES-3
2011-12 / 92.2% / 90.8% / 90.8%
2012-13 / 94.7% / 96.5% / 94.7%
2013-14 / 96.7% / 95.5% / 90.9%
2015-16 / 73.4% / 87.0% / 93.3%
2016-17 / 95.6% / 91.6% / 90.6%
% Meeting expectations and above (if standard changed) / HES-1 / HES-2 / HES-3
2011-12 / 62.9% / 66.3% / 64.3%
2012-13 / 77.2% / 78.9% / 70.2%
2013-14 / 78.1% / 73.8% / 64.1%
2015-16 / 42.7% / 53.0% / 67.7%
2016-17 / 77.2% / 65.1% / 66.7%

Discussion of Results

The working group carefully reviewed the language of the Student Learning Outcomes, the rubrics for assessment, and the relevant data. First, we find that there has not been consistent or careful collection of data over the years. Courses that were scheduled to be assessed in any given year were sometimes never assessed. There are courses that fulfill the HES requirement that have been assessed several times, but there are also courses that have never been assessed or assessed only once. Disciplines assessed only once include Art History, Geography, Historic Preservation, Psychology, Religion, and Women’s Studies.

When we broke out results by discipline and by year (results not shown), some disciplines are consistent within a learning outcome across years, which suggests reliability of measurement over time. Learning outcomes do differ significantly across disciplines, suggesting departments measure these differently, which is not necessarily problematic. Math and Philosophy, assessed more than once, are consistent internally, and also consistent with what other departments are doing across the years. Sociology, however, is inconsistent with when compared to itself over time, and also across the years with other disciplines. History varies considerably internally even in the same years, depending on the course assessed, as well as across the years internally, but it relatively consistent on average with what other disciplines were doing. Linguistics varies considerably with other reporting as well. Psychology is one very clear outlier – measured only once, in 2016, the ratings are quite a bit lower than the other disciplines (lowest category has 32% of assessments for HES_1, 16% for HES_2, and 8% for HES_ 3). As long as measures are consistent within a discipline, they don’t need to be consistent across disciplines. However, because not every discipline is measured each year, the choices can still results in changes across each year for each learning outcome that may not be associated with actual changes in student learning.

Because of this clear need for more consistent data, we were hesitant from the very beginning to even assess the SLO trends across the years (and indeed, there are no clear patterns). We did have a lengthy discussion about how faculty and students are interpreting the questions asked in this assessment, and we felt immediately, from our own experiences, those of our colleagues, and those of our students, that student learning goals as written are not capturing what we want them to capture.

Our Recommendations

1. Since data collections appears to be our most significant issue, we propose that data on HES be collected every year (at least until we are more assured we’re getting reliable and valid data), and that more efforts be taken to ensure that the courses that are selected to be assessed are in fact assessed each year.

2. Second, consistent data collection can be enhanced with an assessment timeline, posted to the website and communicated to relevant professors each semester, along with the already communicated data assessment plan. Similarly, professors teaching HES courses should share rubrics for the assessment they are doing. We would suggest that if at all possible, professors teaching courses meet briefly at the beginning of the fall to share syllabi, discuss how they will address the learning outcomes during their courses, and share rubrics or develop rubrics together (either within or across disciplines; it is not necessary that all professors have the same rubrics, but a discussion of these ideas may prove fruitful). Instructors could also touch base again at the end of the semester just before the assessments are done.

3. We recommend qualitative feedback as well as quantitative analysis. A reflective statement written by each instructor at the end of each semester, on how each accomplished the SLOs in their courses, would be very valuable.

4. We do feel that the criteria have been set too low. Since we are consistently hitting the target of 80% meeting expectations or above (categories 3-5), even in the absence of completely sound data, we think we can do better. We propose keeping the criteria at 80% of the students, but change the categories to “meeting expectations significantly” or “exceeding expectations” (categories 4 and 5). Right now, we hit ranges of 60-75% for these two categories, so this gives a realistic goal of improvement to shoot for. We also feel these criteria will give us a better sense of where our teaching efforts should be directed in the future.

5. Finally, we are recommending two changes to the description of the SLOs themselves. Currently, HES_1 reads: “Students will be able to explain human and social experiences and activities from multiple perspectives.” Instructors using this rubric felt it was unclear what this meant and how to assess it. Did we mean multiple disciplinary perspectives (e.g., sociology versus psychology)? Multiple theoretical perspectives, either within or across disciplines? (e.g., Marxism versus behaviorism)? Differing perspectives from various cultures and/or societies? We determined that the goal for students to understand competing disciplinary perspectives would already be met by the requirement for this general education goal that students take two courses in different disciplines. Therefore, we determined that this SLO would be clearer if we addressed the latter two, but left it up to the instructor as to which of the two would be a part of the rubric, allowing some flexibility but also acknowledging that understanding the human experience does require a look at competing points of view from different groups within or across cultures. We propose a new HES_1 to read:

  1. Students will be able to explain human and social experiences and activities from multiple cultural and/or theoretical perspectives.

The second learning outcomes reads: “Students will be able to draw appropriate conclusions based on evidence.” Here, we felt that as written the requirement left out some important forms of evidence that were nevertheless more common in the human behavior-associated humanities courses that are also a part of this requirement (and sometimes relevant to social science examinations of evidence as well): texts including novels, poems, or lyrics, maps, and other types of visual or auditory materials. We do not list them all, but we wanted to broaden the notion for instructors and students as to what was meant by evidence. We recommend the new HES_2 will read:

  1. Students will be able to draw appropriate conclusions based on evidence, texts, or other materials.

The third learning outcomes reads: “Students will be able to apply knowledge and skills learned to a novel situation.” We felt it was unclear what was meant by “novel situation” and that the goal made no mention at all of what is begin discussed in this goal. It could be written for any goal, and indeed is similar to some of the others in wording. We wanted to make clear that the focus was the skills and application in the social sciences and humanities. We propose a new HES_3 to read:

  1. Students will be able to apply knowledge and skills related to human behavior, ideas, and social structures to better understand actual social life.

Appendix A: Courses Assessed by Year

2011-12Number

GEOG 102 / 65
GEOG 122 / 47
HIST 121 / 31
HIST 122 / 61
HIST 131 / 81
HIST 141 / 31
MATH 207 / 35
SOCG 105 / 101
SOCG 155 / 29

2012-13Number

ARTH 460 / 12
HIST 325 / 24
HIST 334 / 21
Scheduled, not assessed / Multiple

2013-14Number

LING 101 / 92
LING 302 / 24
PHIL 101 / 107
PHIL 160 / 35
RELG 101 / 15
RELG 103 / 48
RELG 205 / 16
Scheduled, not assessed / Multiple

2015-16Number

HIST 122 / 19
MATH 207 / 17
PSYC 100 / 166
Scheduled, not assessed / Multiple

2016-17Number

HISP 101 / 60
LING 301 / 23
PHIL 101 / 14
PSCI 101 / 25
SOCG 105 / 58
WGST 102 / 16
Scheduled, not assessed / ARTH 460, PHIL 225

Courses with two years or more of assessment:

HIST 122, MATH 207, PHIL 101, SOCG 105

Not yet been assessed:

CLAS 110, ECON 201, 202, GEOG 331, PHIL 201, 225, PSCI 201, RELG 206

1