1

Lisa Dominguez Abraham

“Writing Sample Assessment: Pilot Test”

Spring 2011 Type B Sabbatical

Submitted: October 24, 2011

Abstract

The CRC English department needs to decide how to move forward with our assessment instruments and with test portability between our sister campuses. Currently, students assess into CRC English classes via a multiple-choice test (the CTEP). Unfortunately, many of these students are under-prepared. Therefore, during the Spring 2011 school year, I used a Type B sabbatical to research the desirability of adding a writing sample component to our assessment process.

The key to this study was to compare students’ suggested course placements generated by a writing sample test to their suggested placements generated by the CTEP. In the interests of potential district test-portability, a validated prompt from the SCC writing sample test was administered to CRC composition students during the first week of classes, before teachers reviewed any concepts or gave any instruction in composition. Then, thanks to aCASSL grant, CRC composition course-level coordinators participated in an all-day norming and scoring session on January 28, 2011. Details of the process are described on pages 7-8.

Halfway through the semester, composition teachers filled out surveys that included midterm grades, as well as the teachers’ perspectives on how whether students were prepared for course work at the level in which they were enrolled. In August, final grades wereadded to the data bank, andthey were compared to suggested course placements generated by both the CTEP and the writing sample.

After reviewing the results of this comparison, the English department recommends discontinuing the use of the CTEP as an assessment tool. We would also like to further explore the writing sample test as a potential assessment instrument. A series of additional recommendations is described on pages 18-20.

Writing Sample Collection

Because SCC already uses a validated writing sample test, and because the department is interested in potential district test-portability, the writing sample test was conducted using materials from the SCC test bank.

On January 12, I sent out a request that teachers conduct the writing sample test during the first week of classes, before any instruction in composition. (Please see Appendix A, page 22). My request had the potential for disaster. It meant asking English teachers (especially our many adjuncts who teach at more than one college) to significantly revise their lesson plans during a particularly chaotic week in order to administer a test that had no immediate application to their curricula. However, despite my fears of dismissive laughter and outright refusal, there was tremendous support for the pilot test from full-time and adjunct English faculty alike.

Here are the particulars:

  • Each teacher was given a class set of lined paper pre-printed with his/ her course section number. There was also a space on which students could write their Los Rios ID numbers. No names were written on the papers. (Please see Appendix B, page 24).
  • In addition to receiving pre-printed paper, students were given a yellow half-sheet printed with the prompt. This half-sheet had spaces for the professor’s name and the student’s name. (Please see Appendix C, page 25). The yellow half sheet was then stapled to the front of the essay.
  • Students were given 35 minutes to write an essay in response the prompt.
  • Teachers then put their class sets of essays into marked collection boxes in the SOC mail room.

On the main campus of CRC, students take the multiple-choice test to assess into one of 54 sections of composition. The majority of these courses are taught by adjuncts. By the beginning of Week 2, teachers had turned in full class sets of writing samples for 46 of those sections. That is a return of 85%, far higher than the researchers and I had expected. The following chart lists the proportion of courses represented in the study:

Section / Total / Represented
41 / 6 / 5
51 / 7 / 6
101 / 14 / 14
300 / 25 (on campus) / 21

Another measure of support from English faculty came in the way they handled their students’ papers. Nearly every teacher took the time to go through his or her class sets in order to double-check the student ID numbers against their rosters to make sure that all the data was accurate. Although I had asked people to add that extra task to their busy schedules (see Appendix A), I thought only a few full-timers woulddo so. However, not only did the majority of the teachers donate the extra time, an overwhelming number emailed or wrote quick thank-you’s to me for conducting the study.

I now had a pool ofapproximately 1410 papers (47 sections x 30 students per section). The next step involved pulling the papers of students who had most recently taken the CTEP.

Finding Appropriate Papers

In order to do this, I used a roster compiled by Brad Brazil to identify in each class the papers of currently-enrolled students who had taken the multiple-choice test between March 2010 and January 2011. We went back as far as March 2010 because we had not anticipated that we would get so much participation from the English department and had been afraid we wouldn’t collect enough papers to be statistically valid. Given the total number of students taking composition courses during the Spring 2011 semester, Brad said we needed a minimum of 256 papers. The following process was used to identify appropriate papers:

  • Brad organized a roster by course section number, student ID number and student name. The roster also had columns where we could eventually enter writing sample scores, midterm grades, teachers’ perceptions of student progress and final grades. It was an invaluable tool throughout this process.
  • Thanks to a CASSL grant that also made the scoring session possible, I was able to employ a wonderful student aide, Stephanie Young, to put each class set of essays into numerical order according to student ID numbers.
  • Then, I arrangedall 47 sets of papers in numerical order according to course section numbers.
  • Next, for each set, Ipulled the papers of students who appeared on Brad’s roster. Using the roster, I then double-checked course and student ID numbers to make sure they matched the master list.
  • When I was sure they matched, I removed the half-sheets with names, ending up with a stack of about 400 papers identified only with numbers.
  • The 400 papers were then shuffled and reshuffled so that they represented a mix of course levels and student abilities.

Scoring Session

On January 28, the course-level coordinators and I met for an all day scoring session of the papers. However, before we began, we realized that some of the papers in the stack of 400 were of students who had assessed into one class but were now taking the next class. In other words, we could tell from Brad’s list that a student assessed in August 2010 into ENGWR 51 but was now taking ENGWR 101. That meant that the student must have completed ENGWR 51 during the fall semester. The whole point of the study was to compare a student’s skills as measured by the two kinds of tests without the student having received more instruction between taking the two tests.

To the credit of my colleagues, they were willing to take the time to sort the papers further. We made a giant list of ID numbers that indicated the students who had passed the previous class. The list was written onto a white board so that once we started our session, we could remove those papers from the stack. However, even once those 90+ papers were pulled, we still had over 300 papers written by students who had directly assessed via the CTEP into the course in which they were now enrolled. We ended up reading 299 of those papers.

We used the norming packets sent over by SCC in order to get ourselves to read consistently with one another. Then, we followed this process:

  • Each reader put his/her score and initials on a paper, then covered the score with a removable dot.
  • The second reader scored the paper “blind.” Only after the second reader had written a score did he/ she remove the dot to see whether the two scores matched.
  • Because each score represented a particular course level, if the two scores were the same, the paper was considered done.
  • If the scores were a point apart (for instance, a 3 and a 4), the paper went to a third reader. Sometimes at this point, the readers also got together to once again discuss the standards.

At SCC, if the readers for the English department feel that a paper has significant ESL markers, that student is advised to take the ESL writing sample test. Rhonda Farley and Marlo McClurg-McKinnon read the 13 papers that were identified as having significant ESL markers. They suggested an appropriate ESL course for 12 of these papers. One paper out of the 13 was identified by them as more likely being the writing of someone with a learning disability.

Data Entry

After the January 28 session, scores for each of the 299 papers were recorded on a spreadsheet. Brad set up the spread sheet so that we could enter up to three scores for each writer. Stephanie Young, the student aide mentioned above, helped with data entry. Because the scores indicated suggested course placements as generated by the writing sample, Brad, Jeanne and I could get an early sense of whether they differed from course placements suggested by the CTEP. As an aside, upon analyzing the sets of scores,Jeanne Edman was impressed by how few third-reads were necessary. She says it suggests that holistic scoring is consistent.

Midterm Survey

In addition to analyzing writing sample scores, we also wanted to add to our understanding by surveying teachers regarding whether they felt these students were adequately prepared for the courses in which they were enrolled, in other words, whether the CTEP, which helped place them there, had been effective. (Please see Appendix D, page 26). We used the same questions asked in previous CRC midterm surveys. However, we didn’t want to overburden teachers by asking about everyone on their roster. Fortunately, Brenda Limón, the administrative assistant for CASSL, volunteered to make me individual spread sheets for each teacher that contained only the names of those students in their classes whose papers had been read during the January 28 scoring session. Brenda graciously not only made the rosters, she also created an alphabetized list of the teachers and sent me both paper and electronic copies of everything. Teachers were asked to record the following information:

Midterm Grade / Placement Perception (1= Should be at lower-level, 2=Placed appropriately, 3= Should be at higher-level) / Has the student Dropped? (Y/N)

Because most teachers only had a few students whose papers were part of the study, the survey took little time; the response was therefore timely and thorough.

I gave the data from the above survey to Jeanne Edman, and she analyzed it along with the results of the writing sample scoring session. What follows is an excerpt from a letter I sent to English faculty on May 10, 2011:

Overall

When looking at the entire pool of papers in the study, we found that writing sample scores correlate with midterm grades in a statistically significant way. However, if one breaks down the results by course level, it predicts

  • 101 placements very well
  • 41, 51, 300 placements—poorly

Here are my initial theories as to why this may be true:

In terms of 101

The prompt we used for the study is similar to the types of prompts used for the ENGWR 51department final, essentially a single sentence that asks students to communicate their own experiences. For this study it was “Describe an important life lesson you have learned.” Students’ ability to construct an organized, clearly written response to a prompt of this type indicates whether they are ready to tackle writing at the ENGWR 101 level.

It’s possibly also true that ENGWR 101 midterm grades are strongly linked to the students’ essay grades, rather than their attendance, homework, in-class work, quizzes, etc. Therefore, it makes sense that their ability as measured by the writing sample reflects their ability do well in class.

In terms of 41 and 51

Because both of these courses work on preparing students to write essays, by the middle of the semester, students may have been working on sentence and paragraph-level skills. Therefore, midterm grades might more strongly reflect their attendance, homework, in-class work, quizzes, etc.

In terms of 300

This kind of writing sample test might not identify the range of skills necessary to succeed in ENGWR 300. Santa Rosa Junior College and Monterrey Peninsula Community College—both which have used a writing sample assessment test since the 1980s—have moved to using an assessment test that’s a simplified version of our current ENGWR 101 final exam. Students read a ¾ to one-page article that states a position, then are asked to briefly summarize it and explain whether they agree or disagree with the author’s perspective.

Representatives at both colleges cannot say enough good things about this change. While they fought hard over the years to keep any writing sample test, even the simpler one, they feel that this newer version more accurately assesses the reading comprehension necessary to succeed in writing classes, as well as the ability the write academically (as opposed to personally). It therefore assesses students more accurately since composition classes at all levels are designed to help students write academically.

Midterm grades and teacher perceptions

Students assessed into our composition classes via a multiple-choice test (the CTEP). Therefore, we asked teachers to rate, on a 3-point scale, whether students who had taken the CTEP were accurately placed in their classes.

Course levelShould be lowerAppropriateShould be higher

41 3% 83%6%

5114%72%14%

10120%72%7%

30017%81%

As I recall, the minimum standard for assessment tests is that they should accurately place students 75% of the time. At the time, I thought the CTEP was responsible for the higher numbers for 41 and 300. However, as is made clear in the final analysis on page 14, the CTEP might not be a factor in any of these results. In any case, I’m concerned about the number of students who seem to be misplaced in 51 and the number of students in 101 and 300 who might be better served in other classes.

Also, while the midterm grade/ teacher perception survey didn’t ask about ESL students, some teachers spontaneously offered their opinion that particular students would be more appropriately placed in ESL classes.

ESL Survey

As mentioned above, several teachers took the initiative to write comments about the number of students they felt would be better served by an ESL class. I wasn’t sure how to record these comments, so I turned to Jeanne Edman. She suggested doing a follow-up survey to address this particular concern. Unfortunately, the survey didn’t go out until the end of the semester. What follows are the numbers generated by the survey. Appendix E describes the suggested method for tallying ESL students and also contains the narratives that accompany these numbers (pages 27-31).

ESL Survey Results, Spring 2011

Teachers’ perceptions: Percent of students originally enrolled in their reading and composition classes who might have been better served by taking an ESL class instead. They took into account students who dropped, sometimes early in the semester.

InstructorCoursePercent

AbrahamENGWR 10122%

AhearnENGWR 10123%

CasarenoENGRD 110 8%

ENGRD 31021%

FeindertENGWR 101 3%

ENGWR 300 7%

GieddENGWR 110 7%

HooperENGWR 41 7%

ENGWR 5111%

HughesENGWR 4122%

ENGWR 10124%

HutchesonENGWR 3008.6%

MacDonaldENGWR 300 10%

PeralesENGWR 300 5%

(2 sections)

SeamonsENGRD 11010%

TittleENGWR 300 0%

WashingtonENGWR 10120%

WeinshilboumENGWR 5114%

While I’m not sure how to interpret these results, it is clear that it is the best interests of our students that the English, Reading and ESL departments continue working closely together to insure that students are placed into the appropriate classes as quickly as possible.

Final Analysis

Over the summer, Jeanne Edman added final course grades to the information we had collected earlier and analyzed the data. We met on August 22, and she summarized the results as follows:

  • There is a relationship between the writing sample and course GPA for ENGWR 101.
  • There is no relationship between the writing sample and course GPA for ENGWR 41, 51 and 300.
  • There is no relationship between the CTEP and course GPA for ENGWR 41, 51, 101 and 300.

The results for the writing sample mirror our initial discoveries when we added the midterm survey information to the writing sample scores. My analysis of those results is the same as stated on pages 10-11.