Transcript Review 2009

Office of Institutional Research and Assessment

The Evergreen State College

October 2010

NSSE 2010 Benchmarks Report.doc - Office of Institutional Research and Assessment – August 2010 4

Introduction 2

General Education Assessment 2

Transcript Review 3

The 2009 Transcript Assessment Workshop Overview 4

Transcript Characteristics 6

Transcripts by Credit Concentration, non mutually-exclusive categories 6

Transcripts by original admission status 6

Transcripts by degree obtained 6

Inter-rater Reliability 7

Transcript Analysis Results 8

Expectation 1: Articulate and Assume Responsibility for Your Own Work 8

Expectation 2.1: Participate Collaboratively and Responsibly 9

Expectation 2.2: Participate in Our Diverse Society 10

Expectation 3: Communicate Creatively and Effectively 11

Expectation 4: Demonstrate Integrative, Independent, and Critical Thinking 12

Expectation 5.1: Demonstrated Ability to Use Qualitative Modes of Inquiry 13

Expectation 5.2: Demonstrated Ability to Use Quantitative Modes of Inquiry 14

Expectation 5.3: Demonstrated Ability to Use Creative Modes of Inquiry 15

Expectation 5.4: Appropriately Apply Modes of Inquiry to Theoretical and Practical Problems 16

Expectation 5.5: Appropriately Apply Modes of Inquiry across Disciplines 17

Expectation 6.1: Depth of Learning 18

Expectation 6.2: Breadth of Learning 19

Expectation 6.3: Synthesis of Learning 20

Expectation 6.4: Ability to Reflect on Personal and Social Significance of Learning 21

Expectation 6.5: Evidence of a Culminating Experience 22

Earned Credits 23

Discussion: Evergreen’s Teaching and Learning Practices 25

Appendix A 28

Appendix B 29

Appendix C 30

Appendix D 40

Office of Institutional Research and Assessment 1

October 2010

Introduction

General Education Assessment

There are several external and internal reasons for the assessment of general education learning outcomes at Evergreen. Evergreen's accrediting body, the Northwest Commission on Colleges and Universities (NWCCU), requires general education instruction with identifiable learning outcomes and required competencies in the areas of written and oral communication, quantitative reasoning, critical analysis and logical thinking, and literacy in the discourse or technology appropriate to the program of study; and a general education program that includes offerings in the humanities and fine arts, the natural sciences, mathematics, and the social sciences, as well as interdisciplinary courses that focus on the interrelationships between these fields of study.

In addition, the Higher Education Coordinating Board established a series of Academics strategic goals including that "all institutions develop learning outcomes for all academic programs [as well as] learning outcomes measures and competencies in writing, quantitative reasoning, and technological literacy" and to "develop measures of student learning to document student achievement in ways that have credibility with external audiences and usefulness to faculty for the improvement of teaching [and] more effectively use Planning Unit assessment data for curricular and pedagogical improvement." (note: able to find this in the Green Report, but scoured HECB website and our website and couldn’t find a mention of this – should I still include this?)

A cornerstone of Evergreen’s pedagogy is student autonomy. Students are responsible for their own educational path and as a part of this, graduation requirements are minimal; in particular, there are no specific distribution requirements for graduation. The external demand for students to meet general education requirements seemingly runs counter to the educational autonomy that is a central part of an Evergreen education. To address this external requirement, rather than institute a checklist of items required of each graduate, in 2001 the Evergreen faculty approved the Six Expectations of an Evergreen Graduate. These expectations are based on principles that are woven into the fabric of the curriculum and provide a framework to guide students in their academic progress. By approving expectations of graduates rather than requirements to graduate, the faculty reiterated the central principle that the student is responsible for shaping his or her educational journey: The Expectations communicates to the student the intellectual and academic development that is expected of him or her, but the nature and content of the path that is taken is the choice and responsibility of the student.

As a result of Evergreen’s unique structure and learning environment, students’ educational achievements are assessed much more directly than at a conventional institution with distribution requirements and grades. The transcript is the primary source for assessing student learning and the Six Expectations are broad standards by which that learning is assessed.

Transcript Review

Every three years a team of faculty, students, and staff assemble over the course of two weeks to assess a representative sample of Evergreen transcripts from the previous year’s graduating class. One goal of this evaluation is to determine the proportion of students who have taken programs or courses where they have received credit in one or more of five general education categories: art, humanities, social sciences, natural sciences, and quantitative reasoning. A second goal of this evaluation is to determine whether graduates have met the Six Expectations and to what degree they have been met. Finally, the evaluation prompts discussions of how teaching and learning can be modified and improved so students and faculty alike become more keenly aware of institutional expectations and students may better meet these expectations.

The key tools used to assess student transcripts are course and program descriptions, faculty narrative evaluations, student self-evaluations, summative evaluations, and transfer transcripts. The class descriptions detail the nature and content of the class, class activities, and materials studied. Narrative evaluations inform readers about what a student has learned in a course or program from the perspectives of both student and faculty member. A unique contribution of the student summative and self-evaluations is that they can communicate academic development and change beyond the program level.

In August 2002 the first transcript review was conducted on a random sample of 152 transcripts from the 1,026 seniors who graduated in 2000-01. A team of faculty, staff, and one student designed and applied a key for the analysis of Evergreen transcripts, to determine to what extent seniors graduating in 2000-2001 met the Six Expectations. This was considered a baseline sample since it was composed of students graduating from Evergreen before the Expectations were implemented.

Divisional credit areas were classified for a subset of 71 of these transcripts. This analysis revealed 42% of classes lacking self-evaluations. Fewer than ten percent of students completed a summative evaluation and the course designed to help them through the process. The analysis also revealed a notable percentage of transcripts that lacked evidence of quantitative reasoning, the sciences, or the arts.

(what was done between 2003 and 2005 that could account for improvements?) expand this section

In 2005 there were marked improvements in the percent of transcripts exhibiting evidence of art, natural sciences, and quantitative reasoning. (what about self evaluations and summative evaluations?

Between 2005 and 2008 faculty, administrators and staff set into motion several strategies to reduce the percentage of transcripts that lacked quantitative reasoning, art, or science. These efforts included workshops and seminars encouraging faculty to add quantitative reasoning and science components to their program. In addition, Evergreen took steps to help guide students into programs that included quantitative reasoning, art, or science. For a summary of progress on divisional credit representation, refer to the Earned Credits section of this report

The primary goals of the workshop and transcript review were to update the institutional assessment of student learning as evidenced through student transcripts, as well as to streamline, increase inter-rater reliability of and increase repeatability of the assessment process for this and future transcript reviews. The assessment of some domains exhibited much more variability between coders than others, such as “appropriately apply creative modes of inquiry.” To reduce this variability and increase consistency both within one Transcript Review and between reviews in subsequent years, one of the goals for Transcript Review 2009 was to establish defined levels of achievement for each domain of the Six Expectations. Specific anchor points were to be established that would help distinguish transcripts that met the domains of the Six Expectations exceptionally, to Evergreen’s standards, to accreditation standards, or not at all. In addition, well-defined anchor points would create a more streamlined and systematic method of collecting these assessment data as well as to ensure that results between years were comparable.

For the 2009 Transcript Review, a team of faculty, staff, and students evaluated 143 transcripts from a random sample of recent graduates for each of the Six Expectations of an Evergreen Graduate using the coding key developed in the workshop. Additionally, Institutional Research staff assigned divisional credits to each Evergreen and transfer transcript.

The results from the 2009 transcript review and credit analysis are presented in detail in this report. In addition, the assessment process, improvements to the process, the resultant Expectation Key and inter-rater reliability are detailed.

The 2009 Transcript Assessment Workshop Overview

A random sample of 155 transcripts was selected from the 1105 transcripts of students awarded degrees in 2007-08 (14% of the class). The random sample was determined to be representative of the entire pool of transcripts (Appendix A). Of the transcripts sampled, 12 were excluded prior to analysis because their transfer data was no longer available to complete the record. The earliest admission term was identified so that the original admission status of Returning Greeners was represented.

The 26 regular participants in the nine-day Transcript Assessment Workshop included instructors from all planning units, staff members, and students (Appendix B). In addition, academic deans, center heads, and advising staff visited the workshop.

On the first day of the workshop a quick read exercise was performed to familiarize participants with the structure and content of an Evergreen transcript as well as to simulate the impact of an Evergreen transcript to an outside audience such as a recruiter or interviewer. (How many transcripts were used for the quick read?) Based on the quick read, transcript reviewers had to assess the depth and breadth of the student’s work; the student’s area of concentration; and the student’s thinking, communication skills, and ability to work collaboratively.

Over the next two days, participants divided into smaller groups to discuss separate domains of the Evergreen Six Expectations. Within these groups, they evaluated how sample transcripts demonstrated elements of these domains, and developed anchor points for each of these domains to be incorporated into a larger coding key for the Six Expectations. After developing measures for all of the components of the Six Expectations separately, participants reconvened to discuss the anchor points chosen, the appropriateness of selected criteria, what it meant to meet or surpass Evergreen’s expectations or to meet Accreditation standards, and how reproducible the anchor points were from transcript to transcript and from evaluator to evaluator.

These portions of the coding key were then refined over the course of several days by the entire group using sample transcripts to help fine tune existing criteria and identify any additional assessment criteria. Afterwards, two groups of approximately ten workshop participants evaluated a total of four transcripts using a draft of the coding key. This exercise helped further identify any weaknesses and ambiguities in the coding key for improvement. Criteria within the coding key were re-worked until all participants reached consensus.

During the remainder of the workshop, participants divided into groups of two or three and assessed student transcripts using the final version of the Expectation Key as developed in the first part of the workshop (Appendix C). One of the areas with the largest amount of disagreement between coders (was this qualitative modes of inquiry?) was re-worked one additional time during this part of the process.

Transcripts were categorized as meeting each expectation minimally, convincingly, with distinction; or not meeting the expectation at all. Students who met an expectation at least minimally were described as meeting the expectation to accreditation standards. Students who met an expectation either convincingly or with distinction were classified as meeting the expectation to institutional standards.

After the conclusion of the workshop, Institutional Research staff coded earned credits, both transfer and Evergreen, by division: Math, Science, Social Science, Art, Humanities, and Other. A convention was developed for ambiguous or hard to classify subject areas to ensure consistency between transcripts (Appendix D).

The reading of transcripts also sparked a discussion that covered topics including faculty and student evaluations, summative self-evaluations, the evidence of breadth and depth in transcripts, and the alignment of desired educational outcomes with Evergreen’s teaching practices. A brief synopsis is included after the transcript analysis results.

Transcript Characteristics

Transcript review data were disaggregated by campus, student admission status, the type of degree earned, and divisional credit distribution. To analyze transcripts by credit distribution, transcripts from students with 33% or more of their credits in Art, Humanities, Math, Natural Science, or Social Science were analyzed separately from one another. In addition, there were a number of students who took over one-third of their credits in Art and over one-third of their credits in Humanities as well as students who took over one-third of their credits in Humanities and over one-third in Social Science. These students, while included in the separate divisions, are also analyzed in the dual concentration groups. With the exception of the comparison of Olympia campus to off-site campus data, all disaggregated data in this report includes only Olympia campus data. Off-site campus data were excluded because only BA degrees were awarded to off-site campus students, all off-site campus students were transfer students, and areas of credit concentration were limited to Humanities, Social Science, or both Humanities and Social Science.

Transcripts by Credit Concentration, non mutually-exclusive categories

Campus / Art / Humanities / Math / Natural
Science / Social
Science / Art/
Humanities / Humanities/
Social Sci.
Olympia / 27 / 54 / 8 / 19 / 41 / 9 / 14
Off-Site / – / 5 / – / – / 14 / – / 4
Tacoma / – / 2 / – / – / 11 / – / 2
Tribal / – / 3 / – / – / 3 / – / 2
Total / 27 / 59 / 8 / 19 / 55 / 9 / 18

Of the Olympia campus transcripts, one had credit concentrations in Art and Social Science, one had concentrations in Humanities and Math, one had concentrations in Humanities and Science, one had concentrations in Math and Social Science. These four transcripts were included in the analyses for both of the divisions in which they had concentrations. Six transcripts showed no credit concentrations in any division and these were excluded from the analysis.