2016-2017 Annual Program Assessment Report

Please submit report to your department chair or program coordinator, the Associate Dean of your College, and to , director of assessment and program review, by September 30, 2017. You may, but are not required to, submit a separate report for each program, including graduate degree programs, which conducted assessment activities, or you may combine programs in a single report. Please identify your department/program in the file name for your report.

College: Science and Mathematics

Department: Chemistry and Biochemistry

Program: BA/BS/MS

Assessment liaison: Thomas Minehan

  1. Please check off whichever is applicable:

A. ___√____ Measured student work.

B. ___√____ Analyzed results of measurement.

C. ______Applied results of analysis to program review/curriculum/review/revision.

  1. Overview of Annual Assessment Project(s). On a separate sheet,provide a brief overview of this year’s assessment activities, including:
  • an explanation for why your department chose the assessment activities (measurement, analysis, and/or application) that it enacted
  • if your department implemented assessment option A, identify which program SLOs were assessed (please identify the SLOs in full), in which classes and/or contexts, what assessment instruments were used and the methodology employed, the resulting scores, and the relation between this year’s measure of student work and that of past years: (include as an appendix any and all relevant materials that you wish to include)
  • if your department implemented assessment option B, identify what conclusions were drawn from the analysis of measured results, what changes to the program were planned in response, and the relation between this year’s analyses and past and future assessment activities
  • if your department implemented option C, identify the program modifications that were adopted, and the relation between program modifications and past and future assessment activities
  • in what way(s) your assessment activities may reflect the university’s commitment to diversity in all its dimensions but especially with respect to underrepresented groups
  • any other assessment-related information you wish to include, including SLO revision (especially to ensure continuing alignment between program course offerings and both program and university student learning outcomes),and/or the creation and modification of new assessment instruments

3. Preview of planned assessment activities for next year. Include a brief description and explanation of how next year’s assessment will contribute to a

continuous program of ongoing assessment.

2. Overview of Annual Assessment Project(s). Provide a brief overview of this year’s assessment activities.

The following assessment activities took place this year:

•Measure Student Work

  1. Assess basic knowledge in general chemistry, organic chemistry, and biochemistry (SLO1) using standardized exam questions in course finals.
  2. Assess students ability to keep a laboratory notebook in chem 334L using our departmental notebook rubric
  3. Administered a signature assignment for longitudinal assessment of knowledge (SLO1) in gateway (Chem 333, Chem 321) and capstone (Chem 401) courses.
  4. Assess graduate students’ scientific oral communication abilities in literature and thesis seminars, relevant to SLO2m:Organize and communicate scientific information clearly and concisely, both verbally and in writing

•Analyze Results of Measurement

  1. An analysis of student performance trends in general chemistry, organic chemistry, and biochemistry was undertaken.
  2. Current student lab notebook performance was compared to previous assessments from several years ago
  3. The results of the signature assignment were reviewed and suggestions for an alternative exam format/source were made.
  4. Review evidence pertaining to SLO2m:Organize and communicate scientific information clearly and concisely, both verbally and in writing.

The department chose these activities so as to 1.) encourage faculty to continue doing assessment in their courses each year so as to identify trends over multiple semesters and weaknesses in student comprehension that need to be addressed at the individual course level and in the program as a whole, and 2.) move forward with our longitudinal assessment program, identifying possible sources of the lack of significant change in student performance at the gateway and capstone courses that were identified in previous assessments.

A: Measure Student Work

SLO’s addressed: SLO 1: Assess basic knowledge in the following areas of chemistry: general chemistry,organic chemistry, and biochemistry, both at the individual course level and for the program as a whole.

  1. 1. Assess basic knowledge in general chemistry (SLO1) using standardized exam questions. Alignment with core competencies: critical thinking, quantitative literacy.

General Chemistry Assessment: Our General Chemistry courses Chem 101 and Chem 102 service majors, minors and a large number of non-majors. Students taking Chem 101 in fall 2016 and spring 2017 were assessed using 12-15 questions from the ACS standardized exam in General Chemistry.A previously established benchmark for success on these assessments is 60% or more of the questions answered correctly. For one section of 78 students, in fall 2016 the average score for was 6.6 correct out of 12 (55%, questions administered as part of course final exam). 42 students (54% of the class) achieved the benchmark score of 7 (or more) correct out of 12 questions. For spring 2017, identical pre- and post-tests were administered (12 questions) to 78 students with the pre-test administered on the first day of class and the post-test administered as part of the final exam. The average for the pre-test was 3.8 correct out of 12 (32%), and only 7 students (9% of the class) achieved the benchmark score of 7 or more correct. The average for the post-test was 7.3 questions correct out of 12 (61%), and 50 students (64% of the class)achieved the benchmark score of 7 or more correct. In another section of Chem 101 (34 students), identical pre- and post-tests (15 questions) were administered on the first day of class and on the final exam. The average for the pre-test was 5.7 correct out of 15 (38%), and the average for the post-test was 9.3 correct out of 15 (62%). Interestingly, for the pre-test, only 5 out of 34 students (14.7%) achieved the benchmark of 60% (9 or more questions correct out 15); however, for the post-test, 21 out of 34 students (62%) achieved the benchmark score or higher.

e. Analysis of the Results of Measurement

Our Chem 101 general chemistry courses are doing a good job in improving student comprehension of the foundational concepts of chemistry. Students are clearly improving their understanding of the material as they proceed through the course, with significant gains evident in both the class average on the assessments and in the number of students achieving the benchmark level of performance.This upcoming year we will be expanding our assessment to other GE courses including Chem 100,Chem 102, Chem 103, Chem 104, and Chem 110, and although it is not desirable to “teach to the exam”, it may be helpful for general chemistry instructors to review the ACS exam questions and identify general topics with which the students seem to have more difficulty. This activity may result in changing the emphasis on some topics covered in the corresponding lecture course.

One Chem 101 instructor administered a survey at the end of the course asking students to reflect on what was the most valuable resource for their learning provided in the course: homework, lectures, exams, quizzes, textbook or discussion activities. There was an almost equal distribution of responses indicating homework, lectures, textbook and exams. However, only 5% of the class found discussion activities to be a valuable learning resource! Perhaps this indicates a prevailingstudent attitude toward discussion sections as “less valuable” than the lecture portion of the course. This attitude may provide a stumbling block for some students to take full advantage of group problem solving activities to improve their performance in the course, especially in more advanced chemistry courses.

  1. 2. Assess basic knowledge in organicchemistry (SLO1) using standardized exam questions. Alignment with core competencies: critical thinking, quantitative literacy.

The organic chemistry courses Chem 333 and Chem 334 are taken in sequence by all Chemistry BS, Chemistry BA and Biochemistry BS majors. In addition, a large number of non-majors (especially from Biology) take this course as required in their program. Ten questions from an ACS Standardized exam in organic chemistry were incorporated in the course final exams for Chem 333 (Organic Chemistry I, Fall 2016) and Chem 334 (Organic Chemistry 2, Spring 2017). In Chem 333, of the 66 students who took the final exam, 36 students (54%) achieved the benchmark score of 6 or more questions out of 10 answered correctly. In Chem 334, of the 59 students who took the final exam, 22/50 (37%) got 6 or more questions correct.

In another section of Chem 333 (Spring 2017) the instructor administered the entire 70-question ACS Exam in Organic Chemistry (First Semester) on the last day of class. Of the 63 students who took the exam, only 8 students (13% of the class) achieved the benchmark score of 40 (or more) correct out of 70 (50th percentile according to the national norms); the average class score on the exam was 27 correct out of 70, or 16th percentile. In Fall 2016 the entire ACS Exam in Organic Chemistry was administered to a section of Chem 334. Of the 64 students who took the exam, 19 students (30%) achieved the benchmark level of 39 (or more) correct out of 70 questions (50th percentile); the average class score was 34 correct out of 70, or 35th percentile according to the national norms. Finally, in Spring 2017 the entire ACS Exam in Organic Chemistry was administered to another section of Chem 334. Of the 66 students who took the exam, 12 students (18% of the class) achieved the benchmark score of 39 (or more) correct out of 70 questions (50th percentile); the average class score was 31 correct out of 70, or 26th percentile according to the national norms.

e. Analysis of the Results of Measurement

The results show that when the instructor selects the standardized questions for incorporation on the final (clearly focusing on material that was directly covered in the course), the students do better. When the entire ACS exam, which clearly contains some topics that may not have been emphasized as much by the instructor in the course, is administered, students perform more poorly. This clearly indicates that the majority of our students have problems applying what they have learned in the course to examples with which they are not already familiar.Since critical thinking skills are of paramount importance in science regardless of major, these findings, which are consistent in assessments year after year, are of great concern. It is interesting to note that even with mandatory discussion sections for each course (which emphasize problem solving activities) less than 30% of a typical class (and usually less than 20%) achieves the benchmark score of 50th percentile on the ACS standardized exam.The qualitative data available from Chem 101 on student attitudes toward the value of discussion sessions is especially informative in this case. Perhaps instructors need to emphasize the value/importance of discussion activities (for getting better grades!) on the first day of class.

  1. 3. Assess basic knowledge in biochemistry (SLO1) using standardized exam questions.Alignment with core competencies: critical thinking, quantitative literacy.

Chem 464 is a one-semester course in biochemistry taken primarily by non-majors; Chem 462 is the second-semester of a two-semester biochemistry course (Chem 461 and Chem 462) taken by biochemistry majors. In Spring 2017, 42 students in Chem 464 were administered 10 questions taken from the ACS standardized exam in biochemistry in a pre-test on the first day of class and in a post test on their course final exam. The benchmark for success in this assessment was 6 or more questions correct out of 10. For the pre-assessment, only 4 students (9.5% of the class) achieved the benchmark level or higher; however, for the post assessment, 24 students (57% of the class) achieved the benchmark level or higher. Interestingly, the percentage of students who improved their score (by >1) between pre-and post-assessment was 79%. Furthermore, the percentage of students who improved their score significantly (by >3) between pre- and post-assessment was 38%.

In Chem 462 (spring semester 2017, 22 students), 10 questions from an ACS standardized exam in biochemistry were given as a pre-test at the start of the semester and as a post-test on the course final exam. For the pre-assessment, 8 students (38%) achieved the benchmark of 6 or more questions answered correctly out of 10. The average score for the pre-test was 4.2/10. For the post-assessment, 18 students (82%) answered 6 or more questions correctly out of 10. Furthermore, 16 students (72% of the class) improved their score by 1 or more between pre- and post assessment; 10 students (45%) improved their score by 3 or more between pre- and post- assessment (see bar graphs below). The average score for the post-assessment was 6.6/10.

Improved student performance on the final assessment has been observed over the past several years (2015-2017), with post-assessments now showing >80% of the class achieving the benchmark level of performance or higher (see graph below). It should be noted that as a result of course assessments between 2012 and 2014, beginning in 2015 the questions selected for this assessment were changed so that poorly worded ones were replaced with questions perhaps geared more toward content emphasized in the course. While this may partially explain the jump in student performance between 2014 and 2015, it is also clear that students have made real gains in their understanding and knowledge of biochemistry throughout the years.

e. Analysis of the Results of Measurement

In Chem 462, it is evident that students are significantly improving in their knowledge of biochemistry from the pre-test to the post-test: the class average on the pre-assessment was 4.2/10 (below benchmark), while the class average on the post assessment was 6.6/10 (above benchmark). The gains in student performance from pre- to post are very impressive, with nearly half the class increasing their score by 3 or more. One suggestion to allow comparison of our students’ performance with national averages is to administer the entire ACS standardized exam in biochemistry on the last day of class, time permitting. This has been a useful comparison for general chemistry instructors and organic chemistry instructors, allowing them to gauge where our students stand with respect to others assessed across the nation in the same subject area.

A. Measure Student Work

SLO’s assessed: SLO 4: Work effectively and safely in a laboratory environment, including the ability to follow experimental chemical procedures and maintain a proper lab notebook.

  1. Assess students’ ability to keep a laboratory notebook in Chem 334L using our departmental notebook rubric. Alignment with core competencies: written communication.

Chem 334L, Organic Chemistry II laboratory, is taken roughly midway through the BS/BA Chemistry and Biochemistry majors, and students in this lab have already taken three required laboratory courses as prerequisites: Chem 101L, Chem 102L, and Chem 333L. With this preparation, it is expected that students in this course should be able to properly maintain their laboratory notebook, an important skill for all practicing scientists. 29 Chem 334L lab notebooks from Chemistry and Biochemistry majors were assessed using the departmental lab notebook rubric (see Appendix B) by our lab TA’s. Out of a possible score of 20 points, the average score was 16.3/20 (81.5%). Although this appears to be a good indication of our students ability to properly keep their lab notebooks, the highest scores were obtained for categories in which students prepared their notebook outside of the lab: abstract, TOC entry, page #s, completed table of amounts and physical properties of all reactants and solvents, procedure flowchart, etc.Poorer performance was observed in the categories “record of in-lab observations”, “notation of changes to experimental procedures”, and “conclusions and comparison of results to literature values”. For these three categories (assigned 4 points each), the average scores were 8.8/12, or 73%.

f. Comparison to data from previous years

In the 2013-2014 academic year laboratory notebooks were assessed in Chem 411, Chem 422, and Chem 433. Although these courses are taken after Chem 334L by our majors, the average scoresfor the notebook assessment was in the range of 15.1-18.1/20, which tells us little about whether our students are improving their notebook-keeping skills. Indeed, instructors also noted poorer performance in the same categories mentioned above(“record of in-lab observations”, “notation of changes to experimental procedures”, and “conclusions and comparison of results to literature values”). Since the importance of in-lab notebook record-keepingfor practicing scientists cannot be overstated, it is suggested that our rubric be amended to place a greater emphasis on in-lab observations, deviations from protocol, conclusions, and post lab-reflection of the results obtained.

A. Measure Student Work

c. Administered a signature assignment for longitudinal assessment of knowledge (SLO1) in gateway (Chem 333, Chem 321) and capstone (Chem 401) courses.

Signature Assignment Administration, year 3: previously, twenty multiple-choice questions from all subdisciplines of chemistry (general, organic, inorganic, analytical, physical, and biochemistry) were assembled to create an assignment with the input of the department faculty (Appendix A). For the past two years this assessment has been implemented into Moodle, and given as an extra-credit assignment to students in Chem 321 (Analytical Chemistry I, the gateway course for majors and minors) and Chem 401 (Inorganic Chemistry, the capstone course for majorsand “unclassified” graduate students who would like to demonstrate proficiency in inorganic chemistry). Since the previous two assessments showed no significant difference in performance on the assignment in the gateway and capstone courses, it was decided to administer the assessment in Chem 333 (Organic Chemistry I, a required course for majors and non-majors often taken before Chem 321)as well as in Chem 321 in the 2016-2017 academic year.