Estrella Mountain Community College
Student Academic Assessment Committee
Composition / Writing Assessment Results
Spring 2013
Page 1 of 14
Background:
Compositional writing is one of seven Estrella Mountain Community College (EMCC) General Educational abilities. Starting in the fall 2010, EMCC faculty adopted a curriculum-integrated assessment approach to assess student abilities instead of standardized, national instruments. The spring 2013 semesters represents the first time theCompositional Writing abilityhas been assessed using the integrated assessment approach. Prior to the spring 2013 semester,the College Basic Academic Subjects Examination (CBASE) was used to assess student’s writing ability during the spring semesters of 2008, 2009, and 2010. The results on CBASE were less than desired. EMCC faculty collaborated to develop a writing-across-the-curriculum rubric that could be used by all faculty. A version of this rubric was employed for the spring 2013 assessment curriculum integrated assessment.
EMCC's Definition of Composition/Writing
The Composition/Writing assessment was developed todemonstrate student’s ability to understand the subject matter by creating a focused purpose statement followed by the appropriate support to sustain the document’s focus.Students are challenged to demonstrate clarity in the organization of thoughts and use of language through the proper utilization of mechanics and the ability to cite sources.
Purpose and Methodology:
The primary objectives of the assessment are to:
- Establish initial data for EMCC’s writing outcome using curriculum-integrated assessment practices
- Identify gaps in performance between the freshman and sophomore cohorts
- Inform the faculty of our student body’s strengths and areas of needed improvement in writing, in order to help focus faculty members’ pedagogical improvements
- Develop best practices for SAAC and the administration for future composition/writing assessment
During spring 2013, SAAC updated the existing writing rubric in preparation for the 2013 writing assessment, no longer using the CBASE assessment instrument. Thespring 2013 assessment had faculty participants collect artifacts of students' writing ability and provide them to EMCC's Writing Center. Trained writing tutors used the EMCC Writing rubric (see Appendix A) to assess students' work on the basis of content, clarity and editing. The writing tutors critiqued content for subject matter, focus, and support issues. Clarity was evaluated on organization and language use issues. The writing tutors evaluated editing on the basis of format, citation, and mechanics issues.Instructor grades on the submitted documents were independent to the writing tutor scoring of the same documents.
Instructors were initially invited (see Appendix B)during fall 2012 to participate in the spring 2013 writing assessment. Guidelines for participation were as follows:
•Faculty choose/create a writing assignment that meets the criteria
•Students submit their assignment to their instructors electronically so that the assignments can be forwarded to the Online Writing Center(OWC)
•Instructors shall grade/assess the papers for the purposes of their course, but keep an unmarked digital copy of the essay for submission to the OWC
•Instructors create an electronic folder to store the digital essays
•Faculty submit the folder electronically to the OWC
Primary participation criteria consider a writing assignment or prompt of sufficient breadth and depth and assessable using the Writing Rubric. The length of the assignment is sufficient to allow assessment of each component. Additional requirements for assessment standardization include:
•12 point Times New Roman font
•Double Space with1” margins
•Student’s MEID in title of document
Student papers were evaluated on both an ordinal scale and qualitative comments. The ordinal scale indicated the degree a writing sample addressed the necessary issues.Qualitative comments indicate why a score may have been assigned. The ordinal scale evaluated student writing for specific issues relating to content, clarity and editing. The ordinal scalerating:
4 = "The reader is easily able to understand and/or appreciate"
3 = "The reader is generally able to understand and/or appreciate"
2 = "The reader is often challenged to understand and/or appreciate"
1 = "The reader is seriously challenged to understand and/or appreciate"
The spring 2013 assessment was conducted across 10 classes (AJS123, ECN211, ECN212, EDU222, ENG081, ENG091, ENG101, ENH254, MAT217, SOC212), 15 sections, 8 instructors, and included 214 unique students. There were a total of 235 documents submitted for consideration in the composition / writing assessment. Some students had multiple submissions.
Results included within this report compare overall results of new freshman (0-7 cumulative earned hours at beginning of term) to sophomore students(30+ cumulative earned hours at beginning of term).
KEY FINDINGS:
Overall Findings:The EMCC Writing rubric was designed to assess students' work on the basis of content, clarity and editing. Clarity was the strongest of the three major categories with a mean score of 3.52, followed with a content mean score of 3.49. Editing was the weakest of the three major assessment categories, with a mean score of 3.21. Overall, sophomoresoutperformed freshmen withthe exceptions of organization and formatting under the clarity component of the rubric. However, these differences were statistically significant for Editing. Both Citation and Mechanics resulted in statistically high mean score for sophomores vs. freshmen at p<.05. Table 1 reports on the mean scores.
Table 1: / Overall / New Freshmen / SophomoresContent: (composite score) / 3.49 / 3.46 / 3.56
Subject Matter / 3.61 / 3.58 / 3.69
Focus / 3.53 / 3.53 / 3.59
Support / 3.33 / 3.28 / 3.42
Clarity: (composite score) / 3.52 / 3.49 / 3.51
Organization / 3.69 / 3.72 / 3.69
Language / 3.34 / 3.27 / 3.40
Editing: (composite score) / 3.21 / 2.93 / 3.18
Format / 3.48 / 3.49 / 3.45
Citation * / 2.99 / 2.83 / 3.12
Mechanics * / 3.15 / 3.03 / 3.23
* Statistically Significant @ P<.05 (Citation at 0.033, Mechanics at 0.04)
Key Findings Regarding Content:
The overall mean score of (3.49 out 4.0) suggests a high level of student performance in the category of Content. Content was assessed in terms of subject matter issues, focus issues, and support issues. Studentperformance was highest for subject matter (3.61) and focus (3.53). The one area of content that suggest a need for improvement is support for the documents focus (3.33). Each of the content areas are explored in more detail.
Subject Matter Issues:The subject matter of the document should be appropriate for the assignment, adequate to complete the assignment, and suitable for the level of the assignment.Overall, the mean respondent score was 3.61; the new freshmenaverage was lower than sophomores (3.58 vs. 3.69) but the difference not statistically significant at p<.05. Table 2 also includes a frequency breakdown for the subject matter using the rubric categories. Of note is that more than 2 out of 3 (69%) sophomore students received the highest rubric rating.
Table 2 indicates the writing evaluator scoring regarding subject matter issues comparing new freshmen to sophomore responses, in terms of the count and percentage of total responses for the question. Sophomores scored higher in being easily or generally able to understand and/or appreciate the subject matter.
Table 2: Subject Matter / Student LevelNew Freshmen / Sophomores
The reader is easily able to understand and/or appreciate / Count / 50 / 58
% within Student Level / 67.6% / 69.9%
The reader is generally able to understand and/or appreciate / Count / 19 / 24
% within Student Level / 25.7% / 28.9%
The reader is often challenged to understand and/or appreciate / Count / 3 / 1
% within Student Level / 4.1% / 1.2%
The reader is seriously challenged to understand and/or appreciate / Count / 2 / 0
% within Student Level / 2.7% / .0%
Totals: / Count / 74 / 83
% within Student Level / 100.0% / 100.0%
Additional Subject Matter Issues Feedback from the Writing Evaluators: The evaluators noted that the subject matter was NOT adequate to complete the assignment for 24.7% of all respondents (n=58). Sophomores recorded a marginally better percentage (20.5%, n=17) than new freshmen (23.0%, n=17).
Focus Issues:The focus (or main idea or thesis) should be apparent to the reader; presented so as to interest the reader, and targeted to achieve the purpose of the assignment. Overall, the average score for all respondents was 3.53; the new freshmen average was lower than sophomores (3.53 vs. 3.59) with the difference shy of statistical significance.Table 3 also includes a frequency breakdown for the subject matter using the rubric categories. Nearly 2 out of 3 (64%) sophomore students received the highest rubric rating.
Table 3 indicates the writing evaluator scoring regarding focus issues comparing new freshmen to sophomore responses, in terms of the count and percentage of total responses for the question. Sophomores scored higher in being easily or generally able to understand and/or appreciate the focus.
Table 3: Focus / Student LevelNew Freshmen / Sophomores
The reader is easily able to understand and/or appreciate / Count / 44 / 53
% within Student Level / 59.5% / 63.9%
The reader is generally able to understand and/or appreciate / Count / 25 / 26
% within Student Level / 33.8% / 31.3%
The reader is often challenged to understand and/or appreciate / Count / 5 / 4
% within Student Level / 6.8% / 4.8%
Totals: / Count / 74 / 83
% within Student Level / 100.0% / 100.0%
Additional Focus Issues Feedback from the Writing Evaluators: According to writing tutors observances the focus (main idea or thesis) was NOT considered targetedenough to achieve the purpose of the assignment for 33.6% of all respondents(n=79). [s1] A similar percentage of sophomoresstudents received the same comment (31.1%, n=26) marginally better percentage results (31.3%,n=23) compared to new freshmen.
Support Issues:The support should be appropriate to sustain the document’s focus, carefully chosen to meet the needs of the assignment and of sufficient quantity to carry out the development of the document. Overall, the average score for all respondents was 3.33; the new freshmen average was lower than sophomores
(3.28 vs. 3.42) but the difference is not statistically significant. Table 4 also includes a frequency breakdown for the subject matter using the rubric categories. Nearly 2 out of 3 (64%) sophomore students received the highest rubric rating.
Table 4 indicates the writing evaluator scoring regarding support issues comparing new freshmen to sophomore responses, in terms of the count and percentage of total responses for the question. Sophomores scored higher in being easily or generally able to understand and/or appreciate the support.
Table 4: Support / Student LevelNew Freshmen / Sophomores
The reader is easily able to understand and/or appreciate / Count / 32 / 40
% within Student Level / 43.2% / 48.2%
The reader is generally able to understand and/or appreciate / Count / 33 / 38
% within Student Level / 44.6% / 45.8%
The reader is often challenged to understand and/or appreciate / Count / 7 / 5
% within Student Level / 9.5% / 6.0%
The reader is seriously challenged to understand and/or appreciate / Count / 2 / 0
% within Student Level / 2.7% / .0%
Totals: / Count / 74 / 83
% within Student Level / 100.0% / 100.0%
Additional Support Issues Feedback from the Writing Evaluators: The writing tutors found that the support is NOT of sufficient quantity to carry out the development of the document for 27.7% (n=65) and was NOT carefully chosen to meet the needs of the assignment for 23.4% of all respondents (n=55). Sophomores had better percentage scores (26.5%, n=22 and 21.7%, n=18) compared to new freshmen (32.4%, n=24 and 20.3%, n=15).
Key Findings Regarding Clarity:
Student content clarity was assessed in terms of document organization and language use. Student responses averaged between “generally” and “easily” able to understand or appreciate the clarity of the assessment.Students were more able to address organizational issues compared to language issues.
Organization Issues:The document should be developed in a way that shows how all material relates to the focus, shows how all development relates to other parts of the document, and provides adequate transitions to guide the reader and illustrate these relationships. The average score for all respondents was 3.69; withnew freshmen averaging slightly higher than sophomores (3.72 vs. 3.69),howeverthis is not statistically significant.
Table 5 indicates the writing evaluator scoring regarding clarity issues comparing new freshmen to sophomore responses, in terms of the count and percentage of total responses for the question. Sophomores scored marginally lower in being easily or generally able to understand and/or appreciate the clarity.
Table 5: Clarity / Student LevelNew Freshmen / Sophomores
The reader is easily able to understand and/or appreciate / Count / 54 / 59
% within Student Level / 73.0% / 71.1%
The reader is generally able to understand and/or appreciate / Count / 19 / 22
% within Student Level / 25.7% / 26.5%
The reader is often challenged to understand and/or appreciate / Count / 1 / 2
% within Student Level / 1.4% / 2.4%
Totals: / Count / 74 / 83
% within Student Level / 100.0% / 100.0%
Additional Organizational Issues Feedback from the Writing Evaluators: Writing tutors indicatedthe document is NOT currently developed in a way that provides adequate transitions to guide the reader and illustrate these relationships for 14.0%of all respondents (n=33). Sophomores had marginally better percentage scores (10.8%, n=9) compared to new freshmen (16.2%, n=12).
Language Use Issues: The language of the document should aid the reader’s comprehension and appreciation by using organized paragraphing, clear and correct sentences, and appropriate word choice.The overall average score for respondents was 3.34; with new freshmen average slightly lower than sophomores (3.27 vs. 3.40).The difference is not considered statistically significant.
Table 6 indicates the writing evaluator scoring regarding language use issues comparing new freshmen to sophomore responses, in terms of the count and percentage of total responses for the question. Sophomores scored marginally higher in being easily or generally able to understand and/or appreciate the language use.
Table 6: Language Use / Student LevelNew Freshmen / Sophomores
The reader is easily able to understand and/or appreciate / Count / 27 / 40
% within Student Level / 36.5% / 48.2%
The reader is generally able to understand and/or appreciate / Count / 40 / 36
% within Student Level / 54.1% / 43.4%
The reader is often challenged to understand and/or appreciate / Count / 7 / 7
% within Student Level / 9.5% / 8.4%
Totals: / Count / 74 / 83
% within Student Level / 100.0% / 100.0%
Additional Language Use Issues Feedback from the Writing Evaluators: The writing tutors noted that the language of the document DOES NOT aid the reader’s comprehension and appreciation by using appropriate word choice for 43.8%of all respondents (n=103). Sophomores had better percentage result (33.7%, n=28) compared to new freshmen (45.9%, n=34).
Key Findings Regarding Editing:
Student format and citation usage content was also assessed. Student responses averaged between “generally” and “easily” able to understand or appreciate the editing of the assessment. Students performed well with formatting issues but were less successful with mechanics and citation issues.
Format Issues:The format of the document should aid comprehension by being clear, consistent, and appropriate for the assignment. Theaverageoverall score for all respondents was 3.48; withnew freshmen averaging slightly higher than sophomores (3.49 vs. 3.45); this isnot statistically significant.
Table 7 indicates the writing evaluator scoring regarding format issues comparing new freshmen to sophomore responses, in terms of the count and percentage of total responses for the question. Sophomores scored higher in being easily or generally able to understand and/or appreciate the format.
Table 7: Format / Student LevelNew Freshmen / Sophomores
The reader is easily able to understand and/or appreciate / Count / 43 / 41
% within Student Level / 58.9% / 50.0%
The reader is generally able to understand and/or appreciate / Count / 24 / 37
% within Student Level / 32.9% / 45.1%
The reader is often challenged to understand and/or appreciate / Count / 5 / 4
% within Student Level / 6.8% / 4.9%
The reader is seriously challenged to understand and/or appreciate / Count / 1 / 0
% within Student Level / 1.4% / .0%
Totals: / Count / 73 / 82
% within Student Level / 100.0% / 100.0%
Additional Format Feedback from the Writing Evaluators: Writing tutors found that the document format is NOT assignment-appropriate for 37.0%of all respondents (n=87). New freshmen scored a higher percentage (43.4%, n=36) compared to sophomores (31.1%, n=23).
Citation Issues:The citation in the document should be complete, correct, and appropriate for the assignment.The average overall score for respondents was 2.99; the new freshmen average score was lower than sophomores (2.83 vs. 3.12),although not statistically significant. Note, however, that 25 freshman papers and 13 sophomore papers did not have a score for editing citation. This may be a result of the assignment submitted did not have a citation requirement.
Table 8 indicates the writing evaluator scoring regarding citation issues comparing new freshmen to sophomore responses, in terms of the count and percentage of total responses for the question. Sophomores scored higher in being easily or generally able to understand and/or appreciate the citation.
Table 8: Citation / Student LevelNew Freshmen / Sophomores
The reader is easily able to understand and/or appreciate / Count / 7 / 20
% within Student Level / 14.6% / 29.0%
The reader is generally able to understand and/or appreciate / Count / 27 / 38
% within Student Level / 56.3% / 55.1%
The reader is often challenged to understand and/or appreciate / Count / 13 / 10
% within Student Level / 27.1% / 14.5%
The reader is seriously challenged to understand and/or appreciate / Count / 1 / 1
% within Student Level / 2.1% / 1.4%
Totals: / Count / 48 / 69
% within Student Level / 100.0%
Additional Citation Feedback from the Writing Evaluators: Writing tutors indicatedcitation in the document is neither complete for 32.3%(n=76)nor correct for 15.7%of all respondents (n=37). Sophomores performed better (32.5%, n=27) than new freshmen (36.5%, n=27)in terms of not being appropriate for the assignment. Sophomores scored poorer on the citation component (18.1%, n=15) compared to new freshmen (14.9%, n=11).
Mechanics Issues:The document should be comprehensively edited to ensure that the spelling is correct, that punctuation is properly used, and that grammar is correct.The overall average score for all respondents was 3.14; withnew freshmenscoring lower than sophomores (3.03 vs. 3.23), this is not statistically significant.
Table 4 indicates the writing evaluator scoring regarding mechanics issues comparing new freshmen to sophomore responses, in terms of the count and percentage of total responses for the question. Sophomores scored higher in being easily or generally able to understand and/or appreciate the mechanics.
Table 9: Mechanics / Student LevelNew Freshmen / Sophomores
The reader is easily able to understand and/or appreciate / Count / 15 / 27
% within Student Level / 20.3% / 32.9%
The reader is generally able to understand and/or appreciate / Count / 46 / 47
% within Student Level / 62.2% / 57.3%
The reader is often challenged to understand and/or appreciate / Count / 13 / 8
% within Student Level / 17.6% / 9.8%
Totals: / Count / 74 / 82
% within Student Level / 100.0% / 100.0%
Additional Mechanics Feedback from the Writing Evaluators: Thedocument is NOT considered comprehensively edited to ensure that that grammar is correct for 62.2%of all respondents (n=148), according to the writing tutors.