Spring 2015 and Fall 2015

Student Learning Outcomes Assessment Plan and Report

College: College of Education

Department: Middle, Secondary and K12 Education

Name of Degree/Certificate Program/Stand Alone Minor/Online Distance Education Program: Doctor of Philosophy in Curriculum and Instruction

Reflection on the Continuous Improvement of Student Learning
1. List the changes and improvements your program planned to implement as a result of last year’s student learning outcomes assessment data.
2. Were all of the changes implemented? If not, please explain.
3. What impact did the changes have on student learning?
1. As was indicated in the 2014 Annual Report, students in the PhD in Curriculum and Instruction met expectations for each of the three SLOs. As a result no changes were necessary. However, per consultation with program faculty, the PhD in Curriculum and Instruction decided to make the urban core comprehensive exam a take-home exam. This change was implemented to better assess students’ ability to cogently synthesize theories in urban education. Students for the 2014-2015 academic year who take the urban comps were enrolled in a Moodle projects page. In late March, questions for the urban exam were posted on the project page. Students had one week to complete the exam.
2. The change was successfully implemented.
3. Urban core faculty members noted that students’ responses to comp questions were well-developed and more thoughtful compared to previous years when the program used a timed exam.
Student Learning Outcome 1
(knowledge, skill or ability to be assessed)
SLO 1 (revised): Ph.D. students in the Curriculum & Instruction Doctoral Program will demonstrate in-depth knowledge of core urban education.
Changes to the Student Learning Outcomes Assessment Plan: If any changes were made to the assessment plan (which includes the Student Learning Outcome, Effectiveness Measure, Methodology and Performance Outcome) for this student learning outcome since your last report was submitted, briefly summarize the changes made and the rationale for the changes.
In 2013, the College of Education accrediting body, the Council for the Accreditation of Educator Preparation (CAEP), released new standards for educator preparation programs. To better align with these standards, the College of Education faculty have collaboratively worked this year to revise our Student Learning Outcomes (SLOs). In addition, the UNC Charlotte Office of Assessment recommends that programs revisit SLOs every 3-5 years to ensure that SLOs accurately assess student learning. As a result, SLO 1 has been changed. The SLO 1 reported in the 2014 report has been broken into two separate Student Learning Outcomes (SLO 1 and SLO 2). As indicated above, the revised SLO 1 focuses solely on knowledge of core urban education, and no longer also on knowledge of the concentration specialization. Knowledge of concentration specialization is now assessed via the revised SLO 2.
Effectiveness Measure: Identify the data collection instrument, e.g., exam, project, paper, etc. that will be used to gauge acquisition of this student learning outcome and explain how it assesses the desired knowledge, skill or ability. A copy of the data collection instrument and any scoring rubrics associated with this student learning outcome are to be submitted electronically to the designated folder on the designated shared drive.
The Urban Education Comprehensive Exam: The comprehensive examination is taken after a student has completed all the core courses. The Urban Core portion of the examination covers substantive course content and readings from the required urban core courses; thereby aligning with SLO 1.
Methodology: Describe when, where and how the assessment of this student learning outcome will be administered and evaluated. Describe the process the department will use to collect, analyze and disseminate the assessment data to program faculty and to decide the changes/improvements to make on the basis of the assessment data.
The Comprehensive Exam:
The questions will be prepared and graded by faculty members affiliated with the urban concentration and who have provided relevant instruction to the students in the urban core coursework. This exam is a take-home exam. Students choose one of five questions and craft responses up to fifteen pages, double-spaced. It is expected that the students complete the exam within one full week (seven days), cite relevant research and reference in APA format. Submissions are uploaded to the current academic platform used at UNC Charlotte.
The comprehensive examination is given once per year in the spring semester. For those students who receive a Revise and Resubmit or a failing grade, retakes are held in the late summer to early fall semester. The urban concentration faculty members review and rate the examination responses independently and then meet to review their scores. Using the consensus scoring method typical of federal grant panels, committee members can change their scores subsequent to this discussion. Consensus is then reached on the final scores by the committee members. All portions of the examination are evaluated by professors using the following scale:
* A grade denoting Pass indicates that a student has sufficiently answered the questions for that section of the examination, demonstrating mastery of the content and covering relevant content.
* A grade denoting Revise and Resubmit indicates that a student has answered most of the questions correctly from a particular section of the examination, but the evaluations were not sufficient for a grade of Pass. Therefore, the evaluators recommend that the student Revise and Resubmit his or her responses based on the evaluators’ comments.
* A grade denoting Fail indicates that a student has not answered the questions sufficiently on the examination. If a Fail is issued on any section(s) of the examination, students will only be required to retake section(s) that were failed. Students will be allowed to retake the failed portion only one time during the semester immediately following the spring examination (which will allow the student time to study or perhaps take a summer or fall course as needed.) If this second attempt fails, then the student will be denied Admission to Candidacy. The student will then be expected to develop a plan for strengthening his or her knowledge base prior to retaking the examination. This plan must receive approval from concentration’s faculty and the doctoral director before it is approved. No student is permitted to defend a dissertation proposal or do official dissertation work until all sections of the comprehensive examination are passed.
* Simple descriptive statistics are used to report the scores. Disaggregated findings are disseminated to faculty, reported at the program and College level, and discussed at Doctoral Advisory Committee meetings. The data are discussed during MDSK Department faculty meetings at least once per semester. In these meetings, next steps are determined to address any needs identified. All strategies determined during this closing the loop discussion are implemented during the next academic year. These meetings are documented by program directors and department chairs and revisited at each subsequent meeting to monitor implementation progress. All data reports created by the College of Education are housed on a secure website which is accessible to all faculty members within the College of Education.
Performance Outcome: Identify the percentage of students assessed that should be able to demonstrate proficiency in this student learning outcome and the level of proficiency expected. Example: 80% of the students assessed will achieve a score of “acceptable” or higher on the Oral Presentation Scoring Rubric. (Note: a copy of the scoring rubric, complete with cell descriptors for each level of performance, is to be submitted electronically to the designated folder on the designated shared drive.)
At least 80% of the students assessed will achieve a Pass score on the urban comprehensive exam after their first attempt or after revising and resubmitting their responses.
Spring 2015-Fall 2015 Assessment Data / Spring 2014-Fall 2014 Assessment Data
Semester: Spring 2015
Count: 14
Comprehensive Exam: Pass on first attempt – 71%
Comprehensive Exam: Pass on revise & resubmit –29%
Comprehensive Exam: Fail – 0%
Total pass: 100%
* Comps are given only in the spring semester / Semester: Spring 2014
Count: 11
Comprehensive Exam: Pass on first attempt – 82%
Comprehensive Exam: Pass on revise & resubmit – 18%
Comprehensive Exam: Fail – 0%
Total pass: 100%
* Comps are given only in the spring semester
Changes to be implemented Fall 2016: Based upon the 2015 assessment data included in this annual report, what changes/improvements will the program implement during the next academic year to improve performance on this student learning outcome?
Data indicated that candidates in the PhD program met the targeted performance outcomes for revised SLO 1. However, the College of Education is focused on continuous improvement based on data-based decision-making. Based on the data presented here, faculty will continue to monitor the percentage of candidates who are unable to pass on their first attempt to determine if trends emerge across multiple semesters. We have decided to create a rubric that will allow us to better assess students’ performance on the urban comprehensive exam. From the data generated from the rubric additional changes to the types of questions and overall format of the exam will be considered.
In addition, to facilitate more in depth data collection and analysis, data for SLO 1will be collected using the College’s data management system, Taskstream. These changes will be implemented for the 2016-year.
Student Learning Outcome 2
(knowledge, skill or ability to be assessed)
SLO 2 (Revised): Ph.D. students in the Curriculum & Instruction Doctoral Program will demonstrate in-depth knowledge of content concentration (urban education, literacy, mathematics, and elementary education.)
Changes to the Student Learning Outcomes Assessment Plan: If any changes were made to the assessment plan (which includes the Student Learning Outcome, Effectiveness Measure, Methodology and Performance Outcome) for this student learning outcome since your last report was submitted, briefly summarize the changes made and the rationale for the changes.
In 2013, the College of Education accrediting body, the Council for the Accreditation of Educator Preparation (CAEP), released new standards for educator preparation programs. To better align with these standards, the College of Education faculty have collaboratively worked this year to revise our Student Learning Outcomes (SLOs). In addition, the UNC Charlotte Office of Assessment recommends that programs revisit SLOs every 3-5 years to ensure that SLOs accurately assess student learning. As a result, SLO 2 has been modified. The original SLO 2 (as reported in the 2014 Report) is now reported as SLO 4. The revised SLO, as indicated above focuses on candidate content knowledge specific to the concentration specialization (Literacy, Urban Education, Elementary Education, and Mathematics).
To assess the revised SLO 2, an existing data source was identified:1) The Concentration Comprehensive Exam.
This is a new SLO that has been developed for the 2015-2016 year. For this reason, there are no 2014 data to report. Further, because this SLO was developed in Fall of 2015 and concentration exams are only given in the Spring semester, we do not have data for the 2015-year either. Data will be collected for Spring of 2016 and reported back in subsequent reports.
Effectiveness Measure: Identify the data collection instrument, e.g., exam, project, paper, etc. that will be used to gauge acquisition of this student learning outcome and explain how it assesses the desired knowledge, skill or ability. A copy of the data collection instrument and any scoring rubrics associated with this student learning outcome are to be submitted electronically to the designated folder on the designated shared drive.
The Comprehensive Exam: The specialization comprehensive examination is taken after a student has completed all the core courses, a minimum of 12 hours of required research courses, and at least 18 hours of specialization courses. Unlike the urban comprehensive exam, the concentration exam assess students’ knowledge of theory and research related to their designated specialization area (literacy, urban, mathematics, or elementary); therefore aligning with SLO 2.
Methodology: Describe when, where and how the assessment of this student learning outcome will be administered and evaluated. Describe the process the department will use to collect, analyze and disseminate the assessment data to program faculty and to decide the changes/improvements to make on the basis of the assessment data.
The Concentration Comprehensive Exam:
The Specialization portion of the exam is decided upon by each concentration, based on readings and research methodologies relevant to each student’s course of study. Concentrations may, for example, require additional examination questions on campus in a closed and monitored setting or they may ask students to complete a project with a final presentation to a committee of evaluators. The concentration portion of the comprehensive exam will be prepared and graded by faculty members who have provided relevant instruction to the students in the program.
The concentration comprehensive examination is given once per year in the spring semester. For those students who receive a Revise and Resubmit or a failing grade, retakes are held in the Fall semester. The committee members from each of the concentration (urban, elementary, mathematics, and literacy) review and rate the examination responses independently and then meet to review their scores. Using the consensus scoring method typical of federal grant panels, committee members can change their scores subsequent to this discussion. Consensus is then reached on the final scores by the committee members. All portions of the examination are evaluated by the committee members of each concentration professors using the following scale:
* A grade denoting Pass indicates that a student has sufficiently answered the questions for that section of the examination, demonstrating mastery of the content and covering relevant content.
* A grade denoting Revise and Resubmit indicates that a student has answered most of the questions correctly from a particular section of the examination, but the evaluations were not sufficient for a grade of Pass. Therefore, the evaluators recommend that the student Revise and Resubmit his or her responses based on the evaluators’ comments.
* A grade denoting Fail indicates that a student has not answered the questions sufficiently on the examination. If a Fail is issued on any section(s) of the examination, students will only be required to retake section(s) that were failed. Students will be allowed to retake the failed portion only one time during the semester immediately following the spring examination (which will allow the student time to study or perhaps take a summer or fall course as needed.) If this second attempt fails, then the student will be denied Admission to Candidacy. The student will then be expected to develop a plan for strengthening his or her knowledge base prior to retaking the examination. This plan must receive approval from strand’s faculty and the doctoral coordinator before it is approved. No student is permitted to defend a dissertation proposal or do official dissertation work until all sections of the comprehensive examination are passed