1.4.D Key Assessments and Scoring Guides for Programs Not Already Available in AIMS

Exhibit 1.4.d1

1.4.d Key assessments and scoring guides for programs not already available in AIMS

Two major assessments and scoring guides are not available in aims. These are the Candidate Dispositions Progress Report and the Mentor Rating of Candidates’ Use of Technology.

Dispositions

Précis:

• The Candidate Dispositions Progress Report was piloted in 2002-03 and has been in continuous use since that time.
• A study conducted 2006 provided evidence of strong inter-rater reliability for the assessment.
• Mean ratings on the Candidate Dispositions Progress Report document positive dispositions of both initial and advanced candidates. There are strong mean ratings of dispositions targeted by our Transformation Initiative Proposal.
• Weekly monitoring of the Candidate Dispositions Progress Report and Brief Reports provides programs with reports of exemplary dispositions and concerns. Concerns are addressed at the discretion of the program coordinator, and may involve the development of an action plan.
• Action plans are monitored and may result in these decisions: (a) Concern has been resolved/goals have been met; (b) Progress toward goal(s) noted – continue with revised Action Plan; (c) Advised to research other career options; (d) Dismissed from Licensure Program; candidates may develop a rebuttal for the report and decision.
• Due to unit-wide interest in going beyond the summative nature of the Candidate Dispositions Progress Report, a more detailed instrument is being piloted with various observers, including candidate peers, mentors, instructors, and university supervisors.
• Scholarly efforts related to our documentation of dispositions has resulted in a publication: Laine, C., Bauer, A. , Johnson, H. Kroeger, S., Troup, K., & Meyer, H. (2010). Moving from reaction to reflection. In P. Murrell, M. Diez, S. Feiman-Nemser, & D. Schussler, (Eds.), Teaching as a moral practice: Defining, developing, and assessing dispositions in teacher education, (pp. 73-94). Cambridge, MA: Harvard Education Press

______

Our unit dispositions were collaboratively identified by faculty members representing teacher education, art and music education, school psychology, educational foundations, and arts and sciences reviewed and provided input into these dispositions, as did elementary, secondary, middle school, and special education public school teachers and other professional school personnel. Consensus was reached on candidate dispositions for both initial and advanced programs in May, 2002, with pilots of our assessment during the 2002-2003 academic year. During the 2003-2004, following analysis of pilot data and review of the assessment, a revised assessment was implemented. Changes were, for the most part, positioning of the items and the rubric, and data are provided from 2002 to 2011.

As we entered into our Transformation Initiative, it became apparent that we all needed a stronger mutual understanding of our dispositions. Did we really want to begin by asking our candidates to be present, on time, and appropriately dressed, or were we more interested in our candidates demonstrating agency in working in complex environments? Feiman-Nemser (1991) describes the “critical orientation” as the educators’ obligations to students and society. This orientation challenges us to help educators develop educational opportunities that reflect democratic principles and teach them to questing assumptions about teaching, learning, knowledge, and students.

Variations among our programs reflect Feiman-Nemser’s “practical orientation,” which focuses on the aspects of teaching and learning to each licensure area. The practical orientation supports the notion that our candidates must learn to be flexible, and, at times, invent responses in the face of unsolvable problems. Schon (1983) describes this engaging in on-the-spot reflection and experimentation, drawing on a range of images, theories, and actions to construct an appropriate response. Feiman-Nemser contends that this practical knowledge is gained through a combination of clinical experiences, interaction with peers and mentors, and general exposure into the community of practitioners and practice.

Dispositions: As educators transforming the conversation of learning, and creating a third space for instruction we prepare candidates who:

• demonstrate initiative on behalf of all learners.
• demonstrate responsibility to promote effort and excellence in all learners
• demonstrate rapport with students, peers, and others.
• demonstrate a commitment to reflection, assessment, and learning as an ongoing process grounded in inquiry,
• are willing to work collaboratively with other professionals to improve the overall learning environment for students.
• appreciate that “knowledge” includes multiple perspectives and that development of knowledge is influenced by the perspective of the knower.
• are dedicated to teaching the subject matter and in keeping informed and are competent in the discipline and its pedagogy.
• appreciate both the content of the subject are and the diverse needs, assets, and interests of the students and value both short and long term planning.
• are committed to fairness and the expression and use of democratic values in the classroom.
• take responsibility for making the classroom and the school a “safe harbor” for learning, in other words, a place that is protected, predictable, and has a positive climate.
• recognize the fundamental need of students to develop and maintain a sense of self-worth, and that student misbehavior may be attempts to protect self-esteem.
• believe that all children can learn and persist in helping every student achieve success.
• value all students for their potential and people and help them value each other.
• are committed to high ethical and professional standards.

The assessment that has been used with slight modifications since 2003 is available at https://survey.education.uc.edu/websurvey/rws3.pl?FORM=Cand_Dispositions_Progress_Report.

In addition to this assessment,completed by all mentors a minimum of one time during a candidate’s field experience., we have used an “incident” reporting system since 2002. This “brief report” can be used by members of the professional community, faculty members, mentors, or university supervisors to document a concern or an exemplary disposition. This form is available at https://survey.education.uc.edu/websurvey/rws3.pl?FORM=Dispositions_Brief_Report. Instructors, concerned that assessment in the field by mentors and this brief report did not reflect the classroom environment, initiated an instructor disposition assessment in 2009. This assessment was implemented to monitor the professional behavior of candidates on campus and is available at https://survey.education.uc.edu/websurvey/rws3.pl?FORM=Candidate_Classroom_Dispositions_Report_Form . This form also provides an opportunity for faculty members to observe dispositions of advanced candidates.

Candidate Dispositions Progress Report Since its inception in 2002, the Candidate Dispositions Progress Report has followed essentially the same process for monitoring candidate performance. In addition to the mentor’s completion of the report, A university supervisor may complete additional disposition forms for a candidate if they see a need to document exceptional behavior, positive or negative. Completed disposition forms are be printed and shared with the candidate. The candidate must be aware of the report. A candidate may file a rebuttal to a disposition. Since 2006, an electronic version of the completed dispositions form is submitted to the Office of Assessment and Continuous Improvement. The Office of Assessment and Continuous Improvement monitors the forms weekly and provides any evidence of potential issues to the Program Chair. The chair then acts on the information depending on the level of concern. If the concern is significant, and may have an impact on the student continuing in the placement or entering a subsequent placement, an action plan is written. This action plan is an agreement between the candidate and program faculty, describing objectives for improvement, the ways in which the plan will be monitored, and a review date for the plan. Failure to successfully complete an action plan may result in termination of the placement, a change in placement, or leaving the program without licensure recommendation. The action plan format is provided at the end of this document.

The Survey and Aggregated Data Data are presented for the unit, initial programs, and advanced programs. In addition, ratings of dispositions demonstrated by advanced candidates in the university classroom are presented for the two years of implementation. Disposition data are presented as aligned with dispositions, institutional standards, and general education outcomes.

2002-2011 Unit Wide Candidate Dispositions Progress Report
D: Unit Disposition IS: Unit Institutional Standard GE: General Education Outcome
D Candidates are committed to high ethical and professional standards. / 02/03P / 03/04P / 04/05 / 05/06 / 06/07 / 07/08 / 08/09 / 09/10 / 10/11
The Candidate demonstrates professional behavior in attendance. / 3.53 / 3.53 / 3.54 / 3.53 / 3.32 / 3.59 / 3.54 / 3.53 / 3.62
The Candidate demonstrates professional behavior by being punctual. / 3.86 / 3.72 / 3.68 / 3.76 / 3.59 / 3.82 / 3.54 / 3.72 / 3.76
The Candidate is committed to high, ethical and professional standards. / a) Maintains high ethical and professional standards (e.g. respecting confidentiality) / 100% / 100% / 100% / 94% / 98.38% / 100% / 99.14% / 99.2% / 99.7%
b) The Candidate cites program policies and professional practices and responds appropriately / 100% / 100% / 100% / 92.5% / 96.98% / 100% / 99.1% / 99.3% / 99.52%
c) The Candidate maintains a professional appearance / 84% / 89% / 92% / 100% / 98.50% / 99.04% / 99.79% / 99.4% / 99.12%
D Candidates demonstrate a commitment to reflection, assessment, and learning as an ongoing process grounded in inquiry. / 02/03P / 03/04P / 04/05 / 05/06 / 06/07 / 07/08 / 08/09 / 09/10 / 10/11
The Candidate is responsive to constructive feedback/supervision. / 3.11 / 2.95 / 3.83 / 3.88 / 3.81 / 3.95 / 3.30 / 3.85 / 3.88
The Candidate demonstrates a commitment to reflection, assessment, and learning as an ongoing process. / 3.53 / 4.82 / 3.58 / 3.85 / 3.68 / 3.85 / 3.69 / 3.75 / 3.76
D Candidates demonstrate initiative on behalf of all learners / 02/03P / 03/04P / 04/05 / 05/06 / 06/07 / 07/08 / 08/09 / 09/10 / 10/11
The Candidate demonstrates initiative. / 3.85 / 3.48 / 3.50 / 3.35 / 3.42 / 3.53 / 3.30 / 3.38 / 3.45
D Candidates demonstrate responsibility to promote effort and excellence in all learners / 02/03P / 03/04P / 04/05 / 05/06 / 06/07 / 07/08 / 08/09 / 09/10 / 10/11
The Candidate is responsible. / 3.86 / 3.88 / 3.79 / 3.81 / 3.73 / 3.84 / 3.30 / 3.75 / 3.75
D Candidates demonstrate rapport with students, peers, and others / 02/03P / 03/04P / 04/05 / 05/06 / 06/07 / 07/08 / 08/09 / 09/10 / 10/11
The Candidate has rapport with students/peers/others. / 3.07 / 2.93 / 3.77 / 3.81 / 3.72 / 3.92 / 3.74 / 3.76 / 3.74
IS Candidates have foundation knowledge; content knowledge, able to articulate the concepts of their discipline / 02/03P / 03/04P / 04/05 / 05/06 / 06/07 / 07/08 / 08/09 / 09/10 / 10/11
GE Communication
The Candidate demonstrates effective communication in oral, visual, and language arts. / Items added Autumn 2008 to provide additional input into assessing general education outcomes / 3.85 / 3.85 / 3.86
IS Candidates demonstrate the moral imperative to teach all students; address issues of diversity with equity; engage in collaboration and positive systems change / 02/03P / 03/04P / 04/05 / 05/06 / 06/07 / 07/08 / 08/09 / 09/10 / 10/11
GE Social Justice
The Candidate applies knowledge and skills of social responsibility for the advancement of a diverse society. / Items added Autumn 2008 to provide additional input into assessing general education outcomes / 3.77 / 3.78 / 3.23
IS Candidates collaborate, demonstrating leadership, and engaging in positive systems change / 02/03P / 03/04P / 04/05 / 05/06 / 06/07 / 07/08 / 08/09 / 09/10 / 10/11
The Candidate is willing to work with other professionals to improve the overall learning environment for students. / 3.36 / 3.80 / 3.36 / 3.96 / 3.78 / 3.91 / 3.69 / 3.83 / 3.83
IS Candidateshaving content knowledge, able to articulate the central concepts, tools of inquiry structures of discipline. / 02/03P / 03/04P / 04/05 / 05/06 / 06/07 / 07/08 / 08/09 / 09/10 / 10/11
D Candidates appreciate both the content of the subject area and the diverse needs, assets and interests of the students
The Candidate is dedicated to teaching the subject matter and to keeping informed and is competent in the discipline and its pedagogy. / 3.32 / 3.28 / 3.36 / 3.67 / 3.66 / 3.89 / 3.70 / 3.71 / 3.74
In designing curriculum, the Candidate appreciates both the particular content of the subject area and the diverse needs, assets, and interests of the students and values both short and long-term planning. / 3.87 / 3.87 / 3.94 / 3.70 / 3.64 / 3.86 / 3.69 / 3.60 / 3.63
The Candidate is committed to the expression and use of democratic values in the classroom. / 3.85 / 3.78 / 3.32 / 3.56 / 3.76 / 3.94 / 3.69 / 3.83 / 3.79
IS Candidates demonstrate the moral imperative to teach all students and address this responsibility with tenacity; able to address issues of diversity with equity and CRE skills / 02/03P / 03/04P / 04/05 / 05/06 / 06/07 / 07/08 / 08/09 / 09/10 / 10/11
D Candidates appreciate that knowledge includes multiple perspectives; believe that all children can learn and persist in helping every student achieve success; value all students; recognize the fundamental need of students to develop and maintain self-worth; take responsibility for protected, predictable, and positive climate
The Candidate takes responsibility for making the classroom and the school a "safe harbor" for learning, in other words, a place that is protected, fair, predictable, and has a positive climate. / 3.64 / 3.74 / 3.33 / 3.51 / 3.78 / 3.95 / 3.69 / 3.81 / 3.84
The Candidate recognizes the fundamental need of students to develop and maintain a sense of self-worth and that student misbehavior may be attempts to protect self-esteem. / 3.40 / 3.63 / 3.31 / 3.51 / 3.74 / 3.92 / 3.69 / 3.78 / 3.76
The Candidate appreciates that "knowledge" includes multiple perspectives and that development of knowledge is influenced by the perspective of the "knower". / 3.09 / 3.09 / 3.66 / 3.64 / 3.73 / 3.89 / 3.69 / 3.73 / 3.74
The Candidate believes that all children can learn and persists in helping every student achieve success. / 3.67 / 3.87 / 3.31 / 3.56 / 3.82 / 3.91 / 3.69 / 3.85 / 3.83
The Candidate values all students for their potential as people and helps them learn to value each other. / 3.86 / 3.95 / 3.92 / 3.54 / 3.84 / 3.85 / 3.69 / 3.86 / 3.83
Number of responses / 252 / 138 / 864 / 635 / 310 / 216 / 491 / 674 / 686
P - pilots; some scores were re-calculated due to changes in scales
Percent of "Yes" responses after removal of "Unable to Judge"
2009-2011 Advanced Candidate Classroom Dispositions Report
D Candidates are committed to high ethical and professional standards. / 09/10 / 10/11
Attendance / 3.00 / 2.50
Punctuality and Commitment / 3.00 / 2.58
D Candidates demonstrate a commitment to reflection, assessment, and learning as an ongoing process grounded in inquiry. / 09/10 / 10/11
Preparation for Class / 3.57 / 3.33
D Candidates demonstrate rapport with students, peers, and others / 09/10 / 10/11
Classroom Participation / 3.29 / 3.50
D Candidates demonstrate initiative on behalf of all learners / 09/10 / 10/11
Critical Thinking / 3.29 / 3.33
IS Candidates demonstrate the moral imperative to teach all students; address issues of diversity with equity; engage in collaboration and positive systems change / 09/10 / 10/11
Social Responsibility / 3.00 / 3.42
D Candidates appreciate that knowledge includes multiple perspectives; believe that all children can learn and persist in helping every student achieve success; value all students; recognize the fundamental need of students to develop and maintain self-worth; take responsibility for protected, predictable, and positive climate / 09/10 / 10/11
Communication Oral / 3.00 / 3.50
Communication Written / 3.29 / 3.42
N / 14 / 12
1-2 not met; 3 - met; 4-5 exemplary

In 2006, 5% of teaching associates were randomly selected as participants in a reliability study. Ratings by university supervisors were compared to those of mentors, generating the following results:

Inter-Rater Reliability - Candidate Dispositions Progress Report
Item / Reliability
1. The Candidate demonstrates professional behavior in attendance. / 89.25%
2. The Candidate demonstrates professional behavior by being punctual. / 100.00%
3. The Candidate demonstrates initiative. / 100.00%
4. The Candidate is responsible. / 100.00%
5. The Candidate is responsive to constructive feedback/supervision. / 100.00%
6. The Candidate has rapport with students/peers/others. / 96.50%
8. The Candidate demonstrates a commitment to reflection, assessment, and learning as an ongoing process. / 96.50%
9. The Candidate is willing to work with other professionals to improve the overall learning environment for students. / 96.50%
10. The Candidate appreciates that "knowledge" includes multiple perspectives and that development of knowledge is influenced by the perspective of the "knower." / 92.75%
11. The Candidate is dedicated to teaching the subject matter and to keeping informed and is competent in the discipline and its pedagogy. / 95.75%
12. In designing curriculum, the Candidate appreciates both the particular content of the subject area and the diverse needs, assets, and interests of the students and values both short and long-term planning. / 93.75%
13. The Candidate is committed to the expression and use of democratic values in the classroom. / 100.00%
14. The Candidate takes responsibility for making the classroom and the school a "safe harbor" for learning, in other words, a place that is protected, predictable, and has a positive climate. / 100.00%
15. The Candidate recognizes the fundamental need of students to develop and maintain a sense of self-worth and that student misbehavior may be attempts to protect self-esteem. / 95.75%
16. The Candidate believes that all children can learn and persists in helping every student achieve success. / 100.00%
17. The Candidate values all students for their potential as people and helps them learn to value each other. / 100.00%

This strong inter-rater reliability contributes to our confidence in the clarity of the rubric and items.

Analysis The following charts provide mean ratings of candidate dispositions disaggregated by initial and advanced programs. Examples of individual program ratings are provided following the analyses. Means reported are the averages of ratings on the rubrics provided for individual disposition items. Analyses are by disposition alignment with the items.

Mean ratings suggest that both initial and advanced program candidates demonstrate strong attendance and punctuality. With a rating of 4 being exemplary (the rubric indicates perfect attendance and never late as the “4” rating for these items) candidates appear professional in attendance and punctuality. Though the scale is different for the classroom disposition report (3 indicates that the candidate meets expectations) there are high mean ratings of attendance and punctuality.

Both initial and advanced candidates’ mean ratings for commitment to high ethical and professional standards are consistently high. Raters are asked to indicate if candidates demonstrate these standards as yes, no, or unable to rate. The ratings are mean percent of yes responses. Advanced candidates’ mean ratings are 100% across the board; we would anticipate ratings such as these from candidates who are practicing professionals.

Consistently high ratings are reported for both initial and advanced programs. No true trend line was generated in the initial program; there has been a positive trend in the mean ratings of advanced candidate reflection, assessment, and recognition of learning as an ongoing process.

Mean ratings of initiative and responsibility are also consistently high across initial and advanced programs.

Mean ratings on these items, essential dispositions in our transformation initial and our conceptual framework, are consistently high. With our program emphases, we anticipate that these ratings will become even higher.

Advanced programs do not assess the general education baccalaureate competencies. However, oral and written communications are assessed on the classroom disposition form. Mean ratings indicated that the outcomes are met, approaching strongly met.