Sarah Skripsky,
9 Sept. 2011, revised 2-3 Oct. 2011
Recommendations for assessment of WSI curriculum and outcomes
Attn: General Education and Program Review Committees
Note: As an earlier draft, this recommendation letter was sent to Rick Pointer (Acting Provost) and Tatiana Nazarenko (Dean of Curriculum and Educational Effectiveness).
Dear colleagues:
Thank you for taking the time to review this recommendation and offer feedback based on your own expertise and experience.
As you may know, Rick Pointer and Tatiana Nazarenko helped to finance and support my participation in the Dartmouth Seminar for Composition Research in July-August 2011. The seminar was excellent and offered training in empirical research methods relevant to writing program assessment; I exited the seminar with ongoing research questions about our Writing/Speech-Intensive (WSI) curriculum but with some preliminary data analysis completed.
Based on my work this summer, I have some preliminary data analysis and helpful resources to share with faculty and administrators. My overall goal is to help us improve and sustain our GE's emphasis on WSI curriculum by contributing to both formative and summative assessment. I offered a brief introduction to these data and resources at the Faculty Forum on Sept. 15 with an invitation to participate in table talk in response to some existing data. I also sent a follow-up faculty survey on Writing Across the Curriculum that will close on Monday, Oct. 3rd. Once the survey closes, I will be able to begin statistical analysis of the survey data, including comparative analysis of student and faculty responses. Though I am reasonably confident in such analysis, I would appreciate the assistance of those experienced in statistical analysis in relation to assessment, especially mathematicians such as Ray Rosentrater and/or Patti Hunter (should they be willing). I have also developed a Writing/Speech-Intensive Faculty site for Eureka to encourage ongoing resource sharing and forum discussion: https://eureka.westmont.edu/course/view.php?id=11036 (use the enrollment key "wacfac.key"). Faculty buy-in is crucial to the sustainability of program development, and we should not be content to "do assessment" in ways that may satisfy accreditation agencies but may not satisfy our own institutional goals and mission. To that end, I want to invite ongoing faculty dialogue surrounding our WSI curriculum and support faculty development in pedagogy and assessment.
Proposal for summative assessment of GE outcomes for written and spoken communication (WSI curriculum):
My summative assessment proposal is three-fold. I propose that we: (1) institute senior writing portfolio assessment in the Phase 2 model described below; (2) join the Consortium for the Study of Writing in College and administer corresponding CSWC surveys to faculty and students on a regular basis; (3) design an assessment rubric (or similar tool) for our spoken communication outcome in collaboration with faculty who regularly teach our speech-intensive courses (currently, only two SI courses are approved for GE credit). For the purposes of this document, I will elaborate most on the first and most significant part of this three-fold summative assessment with brief attention to the additional recommendations.
Summary of Phase 2 Portfolio Assessment of Student Writing
Based on my analysis of our writing program, existing studies by Ed White et al, and recommendations from Les Perelman and other experts in writing assessment who offered specialized consultations at the Dartmouth Seminar, I have made a formal recommendation for Westmont to institute senior writing portfolios with reflective cover letters for the purpose of summative assessment of our WSI outcomes this year. Such portfolios follow White's "Phase 2 portfolios" model of writing assessment, will offer meaningful samples of student writing that will allow us to measure our WSI outcomes, will allow students (in the cover letter) to self-assess their competence in Westmont's written communication outcomes. This model amplifies metacognition for students and offers a valuable sample of materials for those scoring the portfolios.
Benefits of Phase 2 Portfolio Assessment for Students and Faculty (excerpts from White's 2005 study):
· Benefits for students: “[I]n over three decades of experience with assessments of writing, . . . portfolios are the only assessments I have known that students genuinely find interesting, useful, and worth doing. Students rarely care enough about tests to retrieve them after they receive grades. But . . . portfolios are valuable to students, who normally not only retrieve them after they are graded, but carry them to job interviews and preserve them as a record of their college years . . .." (White 591).
· Benefits for faculty/programs: “The faculty assessment [of these portfolios] can be done relatively quickly and responsibly, yielding reliable grades [i.e., scores], at reasonable cost in time and effort . . .. [T]he entire experience is valuable in its own right. It supports student learning by requiring self-assessment and responsibility, provides direct information to faculty on the outcomes of their programs, and uses existing documents in a new way that is demonstrably direct and valid.” (594)
Logistics/Planning for Phase 2 Portfolio Assessment at Westmont
Dr. Les Perelman, Director of Writing Across the Curriculum at MIT, offered individual consultations at the Dartmouth seminar; he recommends that Westmont faculty rate Phase 2 portfolios using both analytic and holistic rubrics to achieve the best inter-rater reliability. We will need two raters for each portfolio, and we need only assess 25% of senior portfolios in which writing samples from at least two WSI courses are included (25% of approx. 420-440 graduating seniors=approx. 105-110 senior portfolios to rate as a representative sample). Perelman estimates that we can do table training plus portfolio rating during a faculty workshop of approximately 4 days, for which we should offer financial honorariums. Ideally, the raters should be faculty who teach WSI courses regularly (for the workshop to foster professional development as well as to achieve summative assessment). Faculty raters should represent all three academic divisions at Westmont, though it would be appropriate to have 3 raters from Humanities (in which WSI courses are taught most frequently) and 1-2 raters each from our other divisions for a total of 6-7 raters. These raters may be drawn primarily from a Writing Across the Curriculum faculty group (in effect, a new learning community) forming for Spring 2012, but portfolio raters need not be limited to this group. Development of the portfolio rubric(s) may draw on existing models but should involve some input from our own faculty, especially those involved in the WAC faculty group who will develop relevant expertise through reading and discussion.
We need to develop a procedure for current seniors to submit portfolios in the spring and then publicize these guidelines well in advance of the deadline. I would appreciate help from the General Education Committee and/or Program Review Committee in reviewing available e-portfolio models/platforms that will be feasible for Westmont. E-portfolios are already in use by the Education Department, and Jane Wilson and others may be willing to assist in this review process. Above all, the e-portfolio method we choose should be sustainable from year to year in order to provide ongoing motivation for students to write and reflect on their writing as well as a stable archive of student writing for institutional research.
We should also discuss best practices for recruiting faculty raters for a 4-day summative assessment workshop, as described above. Though the majority of the budget for WSI assessment should be spent on faculty rater honorariums, we have reserved some monies for at least two faculty development workshops open to all faculty and in which a meal or snack will be served. Likely workshop topics include WSI response/evaluation practices (Fall 2011) and assignment design (Spring 2012). Cheri Larsen Hoeckley has agreed to collaborate with me in leading these professional development workshops. Rick Pointer and Tatiana Nazarenko have agreed to fund each workshop at a rate of $500 each (or $1000 total), allowing for meals for at least 20-25 faculty per workshop.
I look forward to ongoing collaboration on the WSI assessment initiative with your committee, with Cheri Larsen Hoeckley, and with all participating faculty.
Part 2: The Consortium for the Study of Writing in College
My second recommendation is that Westmont join the Consortium for the Study of Writing in College (http://comppile.org/wpa+nsse/index.htm) and adopt regular use of the National Survey of Student Engagement's (NSSE's) optional CSWC questions. Since we already use NSSE at Westmont with some regularity, the cost of using the additional 27 CSWC questions should be only $200. Preliminary/voluntary CSWC surveys of Westmont students and faculty in 2011 offer useful benchmarks, but annual use (with cohort data and statistical analysis) would be even more beneficial, providing a sustainable line of evidence for our WSI assessment. I also believe that these 27 questions reflect best practices in writing pedagogy, as supported by their endorsement by the Council of Writing Program Administrators. Additional benefits of CSWC membership and surveys are detailed online: http://comppile.org/wpa+nsse/benefits.htm .
Part 3: Lingering questions/challenges for assessing the GE spoken communication outcome
Currently, the few Speech-Intensive classes approved as WSI courses within our GE program offer a very narrow range of options for students to improve in spoken communication (currently, only two SI courses are approved for GE credit). The problem here runs much deeper than having limited data for assessment. If spoken communication is supposed to improve as part of students' GE experience, we need to expand our range of course offerings, possibly by designating new courses as Speech-Intensive or (perhaps more attractively) by inviting some Writing-Intensive course faculty to consider translating existing Writing-Intensive courses into Speech-Intensive courses. Likely "translation course" candidates include courses in Economics and Business and Political Science. With that said, Speech-Intensive assessment should be conducted in the existing Speech-Intensive courses for the 2011-2012 academic year. We should design an assessment rubric (or similar tool) for our spoken communication outcome in collaboration with faculty who regularly teach our speech-intensive courses. Likely collaborators include but are not limited to Communication Studies, Theatre Arts, and History faculty.
I look forward to your feedback on these recommendations and to working together to achieve our goals as a college.
Sincerely,
Sarah Skripsky
Works Cited
White, Edward M. "The Scoring of Writing Portfolios: Phase 2." College Composition and Communication 56.4 (2005): 581-600. JSTOR. Web. Sept. 2011.