STEPMT Notes for Evaluation Plan

STEPMT Notes for Evaluation Plan

From Kathy Whatley’s Site Visit

October 20-12, 2004

Kathy Whatley and Ginger Rowell discussed the plans for both formative and summative evaluation for the STEPMT project. The notes from these discussions follow.

General Notes:

·  Show documents to Kathy before using

·  Meet with psychology professor to look at design of study

·  For studying graduation and retention rates

o  Follow cohorts

o  Study groups of new fresh, transfer, gender, race

§  The purpose of studying these groups is to look for places to target our work

·  Note: Remember in the proposal that we indicated that retention of male biology majors is an area to target. Another area to target is female ETIS majors.

Curriculum

Discovery-Based Modules, Research Class, and Laboratory Internship Class

·  Try to do the same type of assessments for each of these groups. Make minor changes to the instrument.

·  Faculty (set up on web)

o  BEFORE starting: have faculty write a short paragraph on their expectations of what they hope to get out of the development process.

o  AFTER various development stages: faculty write short paragraph on what they got out of it

o  A qualitative analysis will be conducted on these paragraphs (Note: there are softwares which make qualitative analysis of typed materials easier. We may need to check on these items.)

·  Students

o  I don’t have any notes on this. **************************

Research

Academic Year Lower Division and Upper Division Students

·  Instruments: Attitude survey plus brief narrative paragraph

·  Conduct ONLY twice a year (i.e. start and end).

o  Kathy Whatley advised that we only conduct these surveys twice a year. Note, this is less than is in the proposal. We will follow her guidance on this.

·  Kathy suggested

o  An idea for implementation: UNCA has a pre/post attitude survey in their university studies class, it is tied to their computer account and that is how they get them to do the pretest and the post-test is part of their final exam

o  Web-based if possible for students

o  Tie pre/post-tests to student payments

Academic Year Faculty Salary (Note: we need to set (or clarify) criteria)

·  PRE: Faculty should write a paragraph projecting what they expect the students to get out of the experience

·  POST: Faculty write a paragraph telling what they think the students got out of the experience

·  Have the same question for the students as faculty have. Compare student and faculty responses.

·  Idea: Have these statements for the students on the application for the students

·  For the faculty, we could have them write a support statement that includes what students will get out of project. In addition to the faculty stating if the students are capable, responsible, and have follow-through, the faculty could also include a statement about how the project will benefit the student

·  We could have the faculty letters be turned in to the students in a signed sealed envelop. The project coordinator could be responsible

Summer Research Teams

·  Have the team’s goals statement in proposal include expectations for each member of team

·  Have the team final report include specifically what the team members got out of the experience

·  MTSU & HS STUDENTS: Use the same general evaluation from academic year evaluation (attitude & paragraph)

·  HIGH SCHOOL TEACHERS: Make slight adjustment to the MTSU student evaluation that would reflect the mentoring role of the HS teacher and include what the HS teacher would take back to their classroom. Would use the same general evaluation from academic year evaluation (attitude & paragraph).

·  Follow-up with HS teacher the following year to see if they actually were able to include what they had proposed in their high school class

·  Track high school students to see where they go to college and what their major is

·  Track high school teachers to see if they return for their master degrees

Research Content Evaluation

·  Keep a count of papers, presentations, etc.

·  Kathy suggested that we need to get faculty to report this. She also suggested that a database is good for this, BUT we need to publish this list (librarians are good at this)

Travel

·  Faculty: Need reports from faculty in order to get travel reimbursement

·  Students: We need to request a report from student to get reimbursed

Supplies

·  STEPMT Coordinator needs to keep records of supplies purchased from the grant.

Summer Internships

·  Student provides a written assessment of their experience, this will let you know how things are going with that internship.

·  Each internship person needs a monitoring faculty to make sure no problems occur

·  Carefully screen interns for the positions (content knowledge and maturity)

·  Intern partners must evaluate students as well

Other Ideas and Suggestions:

·  Funding for science and math HS teachers after the grant runs out: show data on the way that this is helping change and improve teaching, see if County Board of Education can fund the summer research interns.

·  At end of academic year (or appropriate time) meet with UR Council to review application process went. Revise instructions based on this feedback (formative evaluation and find a way to fix things that are not working and actually fix them). One EXAMPLE: How to increase minority applicants. When the committee meets, take minutes, keep them on file, make a list of action items, assign a person for the tasks, and follow-up that it is done.

·  Keep a check on minority applicants and participants. Develop a plan to recruit these groups.

·  Report to Kathy things that we may want to change from the way they were proposed, or another time to report to her is when an event happens that we don't know what to do, or we can report to her that we have an idea and we could like her feedback.

·  Assessing the Teaching/Learning seminars: send out an email to participants (maybe randomly select some) and have a little checkbox form. Example follows:

o  Here are the 4 sessions, which ones did you attend, rate them (fair, good, excellent)

o  Why did you not attend any

o  Suggestions for improvements in the future

·  Assessing the larger workshops or seminar speakers:

o  Randomly selected sample by email afterwards

o  Get numerical data on attendance

·  Get yearly report forms from NSF for annual and final reports (drive assessment)

3