INTRODUCTION
One of SCC’s Core Themes is Program Excellence, which describes how the college implements program improvement to full its mission of meeting the educational and workforce needs of its diverse community. One objective under this core them is that “Students perceive the college prepared them adequately,” with indicators derived from an annual graduate survey.
The College’s General Education Outcomes are strongly related to this objective; they were “designed to provide our students with the skills, knowledge and awareness they will need to make informed decisions, lead healthy and productive lives, and contribute to the global community as lifelong learners.” If students gain the skills described in the outcomes, they will “perceive the college has prepared them adequately” for almost any possible post-college pathway.
CONTEXT:
In 2012, as part of a campus internationalization initiative, a group of faculty, led by one faculty serving as “Global Awareness Assessor”, formed the Global Awareness Assessment Working Group (GAAWG). This groupengaged in ongoing conversations to discuss these outcomes and how they could be assessed. how best to assess whether Shoreline students were gaining the skills described in the Global Awareness general education outcome outlined above. These initial conversations focused on in-depth review of the outcomes were performed to articulate what they are and what they mean in terms of understanding student learning, and construction a rubric for assessing these outcomes in terms of demonstrating key student learning. In parallel to these GAAWG group conversations, in-depth interviews and surveys of faculty about how these outcomes are being measured and assessed in courses took place. A 2012 report summarized these efforts, with four broad themes surfacing as essential in the assessment of global awareness:basic knowledge,contextual understanding,ability to shift perspectives, andglobal awareness as a developmental process. The 2012 report provided key recommendations on how to promote global awareness at the College and assess it effectively, including institutional support, outcomes review, assessment methodology and process, and changes to administrative procedures. These recommendations are detailed in the box below:
Work of the GAAWG continued through the Spring 2014, culminating in the coordination of substantial data gathering and subsequent analyses, including a full review of the Master Course Outline data. The findings from that work were summarized in a Spring 2014 report.
ASSESSMENT INSTITUTE 2015
The July 2015 Assessment Institute was the next step in the process, providing an opportunity to review work to date andanalyze student reflectionsabout the impact of courses on their own global awareness. The goals of the session were as follows:
- Revitalize work related to assessment of the Global Awareness General Education Outcome, and “close the assessment loop” by sharing data analysis and making recommendation for action
- Gain insight into how lessons learned from the GAAWG experience can inform a long-term plan for ongoing assessment work at Shoreline Community College
Participants:
Bayta Maring, Director, Institutional Assessment & Data Management (facilitator)
Stephanie Diemel, Physics (presenter)
Davis Oldham (Assessor), English
Chip Dodd, Geography and International Studies
Tim Payne, Economics and International Studies
Larry Fuell, Political Science, Director, Global Affairs Center
Terry Taylor, History, Political Science, International Studies
Juliet Scarpa, Assistant Director, Institutional Assessment & Data Management (note taker)
Discussion:
Bayta Maring kicked off the meeting by providing a brief introduction of the Global Awareness Assessment process to date and its relationship to the larger Mid-Cycle Accreditation Report and ongoing, thriving student learning assessment work at Shoreline. Specifically, she flagged this work as a prime example of assessing student learning at the institutional level, with a focus on understanding outcomes, methods for outcome assessment, and the development of a framework to support a continuous and ongoing process of reflection and potential revision. Such a framework is under construction, to be reviewed later in the meeting.
Student Reflection Analyses
In Spring of 2014, a sample of student reflections were gathered about the impact of various courses on their global awareness.Institute participants were asked to review a sample of these reflections (30 in total) from the original eight courses that participated, using a very basic rating system. This rating system, (See Appendix A), served as a rubric for comparison between student’s perceptions of global awareness compare to faculty perceptions based on the student response in terms of:
- Does the reflection contain a specific reference to the course topic?
- Whether or not reference to the Global Awareness outcomes (if any) are present in the Student Reflections (asked to indicate by outcome):
- Recognize the value and significance of artistic and religious expressions in various world cultures
- Articulate the values and beliefs that influence humans in seeking identity and meaning within their culture
- Describe the impact of global interdependence on local cultures including those within the United States
- Identify the origin of events that have led to contemporary global conflict, competition, and cooperation
- Demonstrate awareness and knowledge of the economic forces that have led to the interdependence of national economies and the imbalance of distribution of wealth
- Demonstrate knowledge of the impact of global interdependence on the natural world
The group voted on whether each student reflection revealed each of the six Global Awareness outcomes. Rough counts were taken, detailed in Table 1, to illustrate disparities if any between the expected outcomes, based on the information in the MCO system, and Assessment Institute review of student responses.
ABE 033 – Math Skills for Like 3 (2 student reflections)
CMST 247 – Journalistic Writing (4 student reflections)
GEOG 204 – Weather/Climate and Ecosystems (4 student reflections)
HIIM 154 - Medical Reimbursement Systems (4 student reflections)
HIST 148 – US History III (4 student reflections)
MUSIC 163 – Class Voice Instruction Advanced (4 student reflections)
MUSTC 274 - Music Tech Portfolio (4 student reflections)
PHYS 110– Concepts Physical World/Physics of Current Issue (4 student reflections)
This process was not necessarily designed to yield conclusions about whether students had gained global awareness, but to prompt discussion about the validity of the links between course outcomes and General Education Outcomes.
Analysis Summary:
The student reflection analysis exercise as proof of concept for future outcomes assessment revealed a variety of concerns and questions from Institute participants.
Alignment
Thestudent reflection exercise revealed that there was only moderate alignment between students’ reflections about what they had gained and the Global Awareness outcomes listed in the MCO’s.
Assessment of Particular versus General Learning
Concerns were raised as to whether the reflection prompt may steer individuals to respond towards a particular topic or lesson, as opposed to a general learningoutcome from a particular course. Reframing the prompt may allow for more appropriate responses. Further, it was suggested that asking students to address the GA outcome directly may be a more precise way of garnering student learning towards these specific outcomes.
The analysis also raised questions as to whether students are accessing prior knowledge as opposed to writing to what they have learned in the particular course. Several group members suggested that whether or not students bring external experiences is not necessarily the concern, but rather that the rubric by which we measure the learning experience needs to be intentional, and therefore the outcomes themselves need to be clear.
Understanding Outcomes
It also raised the question, when faculty are indicating in the MCO system whether a particular course outcome is linked to a general education outcomes, whether the faculty understands the intent and meaning of the general education outcome.
The argument was put forth that additional clarity around the meaning of both course and GA outcomes, as well as accountability towards those outcomes (through data or assessment), is essential and needed. The inverse is also true – examining data may reveal that a particular course may in fact be addressing particular outcomes. In cases when data show that students are moving in a different direction than the stated course outcomes, the faculty should then question whether to change the MCO or change their approach with students who may be off track.
Methodology
In general, it was agreed that if the student reflection process, in its current role as proof of concept, be the chosen method for garnering evidence for student learningmoving forward, that the balance between participation and rigor needs to be addressed. Specifically, the means by which we garner information from students and faculty to assess these outcomes need to not be so burdensome that they thwart participation. In addition, it is a model that can be used to assess all students across all outcomes, but still maintains enough rigor to provide meaningful information for assessment.
Support
Broadly, the reflection analysis process underlined the need to provide extensive professional development and institutional support for the ongoing conversations about the MCO process – who is part of those conversations, how it is informed through ongoing collaboration.
Singular Exercise
Finally, some concern was expressed about the notion of putting weight on to a single focused question to assess entirety of course in its meeting Global Awareness outcomes.
Table 1. Summary of Student Reflection Analyses, July 2015
ABE 033(N=2) / CMST 247
(N=4) / GEOG 204(N=4) / HIIM 154(N=4) / HIST 148(N=4) / MUSC 163(N=4) / MUSTC 274(N=4) / PHYS 110(N=4)
# / MCO / # / MCO / # / MCO / # / MCO / # / MCO / # / MCO / # / MCO / # / MCO
- Recognize the value and significance of artistic and religion expressions in various world cultures
- Articulate the values and beliefs that influence humans in seeking identify and meaning within their culture
- Describe the impact of global independence on local cultures including those within the United States.
- Identify the origin of events that have led to contemporary global conflict, competition, and cooperation
- Demonstrate awareness and knowledge of the economic forces that gave led to the interdependence of national economies and the imbalance of distribution of wealth
- Demonstrate knowledge of the impact of global interdependence on the natural world
Review of 2014 Report
Following the Student Reflection Analyses process, conversation turned to how this information sits in context with the findings from the 2014 Report.
Concerns were expressed in terms of the unequal representation of outcomes across the course offerings. This leads one to question the expectations of these learning outcomes and whether it is the intent that all students exhibit all six Global Awareness outcomes by the end of their learning experience, or some combination. The expectation may be different based on the type of student (i.e. transfer versus professional/technical students) and the type of GE outcome (i.e. some GE outcomes are also core outcomes or requirements for graduation, so much more specifically defined in the course of a student’s learning trajectory.) The lack of clarity in the expectations of the GE outcomes as well as their alignment with core distribution requirements leads these questions to surface.
There was some concern around the ability to use general questions to assess student learning on general education outcomes as it makes comparison across disciplines or individual instructors difficult. Yet, while in-class work would be more logistically manageable, ultimately GE outcomes are trying to gauge learning across courses.
Overall, the strongest theme in the conversation was the need for tighter alignment between course-level outcomes with general education outcomes. In current practice, an instructor can link each course outcome with any number of general education outcomes, which makes targeted assessment of those general education outcomes particularly challenging. The next step in that process is the crux of the proposed work, which is putting in place a system that articulates how these conversations at the various levels – course, department, and institution – begin to take place in a routine and meaningful way, supported by data, so that assessment of student learning outcomes becomes a routine part of instructional work at Shoreline.
Best Practices
Professor Diemel shared with the group examples of best practices in student learning assessment from other colleges in Washington State, where such cultures of assessment are currently being implemented.
The most notable examples are as follows:
- Whatcom Community College who have developed indirect and direct measures of course-level, program-level, college-level, and community-level assessment.
- Clark College, which has realized full participation in student learning outcomes assessment, meaning that 100% of their programs are engaged in outcomes assessment. Clark has also developed a faculty assessment handbook, which the group reviewed briefly.
Framework - General Assessment Cycle
Diemel and Maring then put the entirety of the work around Global Awareness Outcome Assessment in context of a proposed General Assessment Cycle, a mechanism by which assessment can become an ongoing process at the course, program and general education level. Institute participants, as part of reviewing this proposed cycle, stressed that such a framework needs to be equitable, transparent and accessible to everyone that is relevant to the process. At the core of such a model is the ongoing conversations with course and department level staff about clarification of the outcomes themselves as well as alignment between outcomes at program and institutional levels.
The nature of this cycle also implies that assessment, as it should be, is an ongoing part of the lifecycle of student learning. Therefore, all outcomes should continually be in a stage of the cycle to ensure they remain clear, aligned, and relevant. A practical application may be to stagger the review of the outcomes in such a way that, once the system is up and running, everyone is in a stage of assessment, recognizing also that not all outcomes are treated equally, and therefore the timescale for some may be longer than others. Most of all, it is essential that all levels of this process are supported by administration/leadership.