Precondition #5
The unit regularly monitors and evaluates its operations, the quality of its offerings, performance of candidates, and effectiveness of its graduates.
5.1 The unit's policies for systematic evaluation of its program (including the evaluation of its assessment systems and of its candidates’ performance).
The unit uses several regular systematic program evaluations.
• The National Association of Schools of Music (NASM) and the North Central Association of Colleges and Schools (NCA) administer reviews of the Conservatory of Music every ten years.
• The unit conducts an internal review of the Music Education Division every ten years.
• The unit distributes alumni surveys every five years.
• The unit meets monthly and reviews the progress of the teacher candidates and the effectiveness of the curriculum.
• The unit meets semi-annually to assess and redesign curriculum and assessment tools. At this time, the unit faculty discusses national educational trends and the feasibility and value of incorporating those trends into the curriculum.
• Cooperating teachers and unit faculty evaluate the teacher candidates' field experiences (Evaluation of Field Experiences form).
• Teacher candidates evaluate cooperating teacher, field experience coordinator, and unit faculty supervision of field experiences.
• The unit evaluates and analyzes its assessment tools annually in order to determine validity and reliability.
• The unit distributes student course evaluations at the completion of each semester.
Because of the unique nature and the size of the Music Education unit, the Division faculty is able to undertake formal and informal conversations with cooperating teachers and other faculty members on a frequent basis. Oberlin College currently retains a Director for the Center of Teaching Excellence. This person is a member of the Education Council, and she frequently serves in an advisory capacity to the unit. Recent discussions among unit faculty and the Oberlin College Education Council have resulted in the following initiatives:
• Revision of the Midterm and Final Student Teaching Evaluation forms (Doerksen, 1990; MENC, 2000; NTASC assessment criteria; ETS Pathwise Assessment Criteria, 2002)
• Formalization of assessment tool data collection and analysis within the unit
• Orientation session and dinner with cooperating teachers, student teachers, and college supervising teachers
5.2 A brief description of the development and refinement of the unit's assessment system
Over the years, the unit has developed and refined its assessment plan according to standards set by NCATE, NASM, ODE, and the institution. As each agency changed and refined its standards, the unit revised its curriculum and assessment plans to accommodate those revisions. What once began as anecdotal and written evidence of what the unit faculty taught in music education course soon evolved into written documentation of teacher candidates’ performance using benchmark assessment tools, in field experiences and in music education courses. In fact, course individual assessments and unit benchmark assessments continue to evolve to reflect the NCATE and ODE standards, the INTASC and ETS Pathwise assessment criteria and the unit values and dispositions. The unit designed the assessment plan in consultation with the Oberlin College Education Council.
Of course, with the collection of course assessments, assessments of individual teacher candidates’ performance in music education courses, cooperating teacher written and verbal feedback, alumni written and verbal feedback, the unit’s curricular content and assessment tools have changed. Each monthly unit meeting and semi-annual retreat includes an agenda of proposals and ideas for improving systematic and course assessment procedures. Documentation for each benchmark assessment is maintained in teacher candidates’ files in the unit office. Samples of individual assessments in courses offered by the unit are also maintained in students’ individual files.
The unit is assessed internally every ten years by the institution; the last institutional evaluation of the unit occurred in Spring 1996. A copy of that evaluation is available in the unit office in the Conservatory of Music. The last NASM approval of the institution and, thus, the unit occurred in 1998. The institution (and unit) were also assessed by the North Central Association of College and Schools (NCA) in 1998. Its next review will be in 2008. Upon site evaluations of the institution and the unit, each agency granted re-accreditation.
Please see the following Oberlin College Music Education Division (Unit) Assessment Plan for assessment procedures, forms, aggregated benchmark assessment data displays, and discussions of any data trends.
UNIT ASSESSMENT PLAN
The unit assessment system began its reconstruction in 2001. At that time, the unit faculty met by itself and in consultation with the Oberlin College Education Council. It was at that time that the unit sought to align its conceptual framework and dispositions with it benchmark and program assessments. The unit identified qualities in teacher candidates that it sought to enroll in the Music Education program—potential as a musician, a teacher, and a scholar. Based on the assessments, internal and external, the unit meets and discusses ways to improve the program it offers its teacher candidates. Forms of assessment include: NASM, NCA, ODE, Oberlin College, Oberlin Conservatory of Music, unit faculty, teacher candidates, music education alumni, cooperating teachers, Praxis II scores, field experience evaluations, course evaluations and teacher candidate performance in coursework and field experiences. Table 1 shows the unit’s regular systematic assessment plan.
Meeting minutes from 12/18/01 (see Appendix 17) indicate the unit’s creation of rating scales and the meaning of each rating for student teaching mid-term and final evaluation forms. At that time, the unit also created time-lines and forms for cooperating teacher feedback. The current unit director discussed her use of portfolio assessment in MUED 300 and how the procedure might be adapted for teacher candidates’ program exit professional portfolio in MUED 400. The unit also revisited and stipulated its student teaching, Comprehensive Musicianship Examination, field experience, and entrance/recruitment procedures. On 12/18/01, all music education (unit) faculty were present.
During the 2002-2003 academic year, the unit faculty began to understand the necessity of aligning its assessments (benchmark and course) and course content with teacher proficiencies outlined in the NTASC and Pathwise observational and assessment criteria. The unit continues to revise its syllabi so that the alignment is clearly indicated.
During 2003-present, the unit director continues to update the teacher candidate and program completer databases. The unit recognized the importance of tracking individual ratings in musicianship, pedagogy, and scholarship (in terms of recording actual numbers) at prospective students’ Conservatory of Music auditions and Music Education Division interviews.
At this point of the transition process in the assessment system, the unit believes that it has compiled, analyzed, and interpreted important data from benchmark assessment points, alumni survey, teacher candidate exit surveys, and Praxis II examinations. However, the unit recognizes that it has not received information on Praxis III scores. Nor has it been prompt in collecting employer data from its graduates. These two pieces of program-completer data are important in determining the success of its graduates in the music education profession. To date, the unit has relied on verbal anecdotal evidence from its alumni regarding their professional teaching success and preparation for teaching. In light of the survey data that it did collect, the unit recognizes that the survey needs to be more specific to the preparation that the alumni received and their ability to demonstrate skills in musicianship, pedagogy, and scholarship. The current alumni survey is too general in the type of information that it elicits. Another area that the unit intends to track more systematically is data from MUED 102: Practicum Experiences in School Music. Observational data from unit faculty and cooperating teachers might be predictors of future success (or failure) in the music education program, even prior to the administration of the Comprehensive Musicianship Examination.
Table 1: Music Education (Unit) Assessment Plan).
Points of Assessment / Internal Unit Data / Internal (Oberlin College) Data / External DataTeacher Candidates / Benchmark Assessment Points (entry, continuation, entry to student teaching, exit, completion of teacher licensure; program dispositions): interviews, CME performance, teacher candidate self-evaluations, student teaching placement, informal transcript reviews, college supervisor and cooperating teacher evaluations, professional portfolio, course grades, assignment rubrics / Music performance audition, application process; official transcript reviews, / Praxis II and III* data; GPA, SAT & ACT scores; high school recommendations; Pathwise & INTASC guidelines for observation and assessment;
PRAXIS II/Licensure / Maintain Praxis III scores in teacher candidate files; use of unit program assessment / ODE is the licensure-granting agency; ETS passing scores needed for teacher licensure
Unit program evaluation / Unit collects, analyzes, interprets and maintains teacher candidate data from courses (presentations, musical, written, videotaped, portfolios, notebooks, journals, exams, papers) and benchmark assessment points; and cooperating teacher written commentary; alumni surveys; teacher candidate exit surveys; unit monthly meeting agendas; yearly unit retreats; budget, technological support/facilities, admission / Oberlin College President and Conservatory Dean approve of program evaluation documents (submitted to internal and external evaluation agencies) / Title II reports, ODE preconditions, program approval, institution reports; NASM/NCA/ODE on-site visitation reports; Praxis II scores; Praxis III scores*; employer assessments of unit program completers*; Pathwise and INTASC observation and assessment guidelines
Unit faculty evaluation / Unit faculty evaluations for tenure and promotion; course evaluations, alumni surveys; teacher candidate exit surveys / Conservatory Faculty Council, Dean, President, Trustees review of materials for tenure, promotion, and merit pay / NASM, NCA, ODE assess faculty credentials and scholarship, teaching, service; external reviewers for tenure and promotion materials
Institution evaluation / Self-study reports / Self-study reports / On-site reviews and reports by NASM, NCA
*Program Completers’ Praxis III scores and Employer Assessments.
The unit recognizes the importance of tracking its program completers’ Praxis III scores and employer assessments, once the program completers are teaching. This benchmark assessment, although required by ODE, has not yet been a part of the unit assessment plan. Many of our graduates attend graduate school or pursue a music performance career immediately after graduation from Oberlin College. It is typical for our students to enter the teaching profession several years after college graduation. Furthermore, most of our students do not take the Praxis III exam, so that data would not be representative of all our graduates’ performance or our program efficacy.
Therefore, there are no comparative data to analyze, interpret, and respond to by the unit. The unit will begin recording scores as of Spring 2005. Therefore, the unit asks the Ohio Department of Education to understand the evolutionary process of the unit assessment plan, collection of raw data, and analysis and response to aggregate data and trends in this area.
PROGRAM AND TEACHER CANDIDATE ASSESSMENT: AGGREGATE DATA, ANALYSIS, AND INTERPRETATION
FOR UNIT BENCHMARK ASSESSMENTS
Teacher candidates must demonstrate the following musical, pedagogical, and scholarly competencies and expectations prior to completing the Bachelor of Music in Music Education degree at Oberlin College and Ohio teacher licensure (Table 2).
Table 2. Sequence of benchmark assessment administration.
Benchmark Assessment / Time of administrationEntry into B.M. Music Education program / Senior year of high school; first or second year at institution (late admission/transfer students)
Continuation in the B.M. Music Education program / Fourth semester (B.M. only) or fifth semester (Double major/double degree students); third semester in the program for late admission/transfer students
Entry to student teaching / Senior year (fourth or fifth year)
Exit from B.M. Music Education program / Completion of fourth (B.M. only) or fifth year (double major/double degree students)
Completion of Requirements for teacher licensure / Senior year or after graduation with the B.M. in Music Education
Aggregated Data and Interpretations for Benchmark Assessments
Raw data for all portions of the unit’s aggregated data are found in Appenidices 7-11.
BENCHMARK I: ENTRY AND EXIT DATA
Benchmark Description/Notes
· The data presented here are gathered from candidates entering the program from Fall 1997 through Fall 2004.
· Oberlin College Conservatory (OCC) standards for acceptance to the college and conservatory are:
SAT = 1200
ACT = 26
· In addition, the music education division expects a minimum high school GPA of 3.0.
Benchmark Data Reports
1. SAT ranges: 1490-920
Highest– 1490 (H Smeltz, A Nordquist, M Sansoni)
Lowest 920 (S Fortner-dropped major), 1020 (N Huyett),
1060 (A Helfer)
Average 1275
2. ACT ranges: 32-21
Highest– 32 (G Evans), 31 (J Stanley), 29 (S Klauer)
Lowest 21 (M Roest-dropped major), 22 (T Wedge), 22 (N Huyett)
Average 26
3. GPA ranges (high school): 4.3-3
Highest 4.3 (S Fortner-dropped major), 4.28 (B Peachey),
4.12 (Nashira Thomas), 4.1 (J Marquardt-dropped the major),
4.05 (B Brennan), 4 (J Stanley, L Talley, A James,
G Bier-not accepted into program,)
Lowest 2.7 (M Schneider), 3 (J Thomas, M Halberstadt-transferred),
3.1 (N Fisher)
Average 3.8
4. GPA ranges (graduation): 4.03-3.09
Highest 4.03 (J Stanley, 3.86 (S Clink, H Smeltz)
Lowest 3.09 (B Lange), 3.2 (S Robles, S Mello)
Average 3.58
Benchmark Interpretation and Trends
1. Although one candidate was admitted with a high school GPA of lower than 3 (2.7), all candidates maintained GPAs of 3.0 or higher at throughout the program.
2. Fall ‘97 to Fall ‘04, 65 music education majors were given Conservatory Admission Performance Ratings on a 5-point scale; 5 is Superior, 1 is Poor. Distribution was:
5 = 23% (15)
4 = 61% (40)
3 = 15% (10)
3. Of those 15 who rated 5, 100% were in agreement with the music education interview admit ratings. Music education admit ratings are made on a 5-point scale (5 is superior, 1 is poor) for 5 areas of assessment during the interview: Academic Strength, Teaching Potential, Functional Skills (keyboard, sight-singing, sight-reading, performance), Theory Knowledge (keys, scales, modes, intervals, chords), Aural Ability (intonation, intervals, chords, aural memory)
4. 2 of the 15 candidates admitted with a 5 rating did not complete the program (G. Bier, J-H. Koo). Of the 13 admits rated 5, 2 had difficulties with the CME (Brennan & Lange), the remaining 11 have no standout difficulties in the program (Sitomer, Smeltz, Stanley, Talley, Irvin, Klotz, Peachey, Nordquist, Sansoni, Newman, Roach).
5. The majority of music education majors enter with a Performance Rating of 4, and fewest (only 15%) enter with a rating of 3.