OSU Academic Alert Instructor Survey
Prepared by the Academic Alert Task Force[1], January 23, 2012
The purpose of the Academic Alert Instructor Surveywas toassist with evaluating the pilot of the Academic Alert System by gathering information from instructors who used the system to report student concerns during the Fall 2011 term. The survey was designed to: (1) assess the level of instructor satisfaction with the system; (2) assess instructor perceptions regarding the effectiveness of the system—whether they detected student behavior changes as a result of academic alert notices; and (3) determine whether aspects of the system should be changed for future terms.
The 408 Fall 2011 instructors who used the Academic Alert System between September 7 and November 11, 2011,were invited to participate in the online survey through an initial email notification on November 14, 2011, and a reminder notification on November 29, 2011. During the time period November 14 – December 9, a total of 157 completed surveys were obtained, yielding a response rate of 38.5%.
Results
Instructors were asked whether the five alert categories should be retained for future terms. The vast majority of responses favored retaining all categories:
Attendance98.7% retain1.3% delete
Poor Quality Work 82.4% retain17.6%delete
Missing Work 92.8% retain7.2% delete
Cannot Pass with Remaining Coursework 91.7% retain8.3% delete
Instructor Comments 96.7% retain3.3% delete
Approximately 20% of the instructors (N = 34) suggested additional categories for reporting alerts. The complete set of responses is provided in Appendix C.
Approximately 75% of the instructors rated their satisfaction with the Academic Alert system as a 4 or 5 on a 5-point Likert scale. See Figure 1. The mean response was 3.90.
Figure 1. Instructor Satisfaction with Academic Alert System
Instructors were asked to indicate the observed level of improvement in students’ academic success after submitting Academic Alert notices. Approximately 75% of the respondents reported improvement in at least some students. When asked whether they detected significant improvement, the percent decreased to approximately 45. See Figures 2 and 3.
Figure 2. Some Improvement in Student Academic Success Detected after Alert
Figure 3. Significant Improvement in Student Academic Success Detected after Alert
Approximately two-thirds of the instructors(N = 97) responded to Question 5: “What comments do you have regarding the Academic Alert system?” The complete set of responses is provided in Appendix C.
Approximately half of the instructors (N = 77) responded to Question 6: “What suggestions do you have to improve the Academic Alert system?”The complete set of responses is provided in Appendix C.
Appendix A. Survey Instrument
Q1. Currently instructors may use the Academic Alert system to report concerns in the following categories. Please indicate whether you think these categories should be retained in the alert system during future terms.
Attendanceretaindelete
Poor Quality Work retaindelete
Missing Work retaindelete
Cannot Pass with Remaining Coursework retaindelete
Instructor Comments retaindelete
Q2. If you would like to see additional categories for reporting options, please list them here.
Q3. Please indicate your level of satisfaction with the Academic Alert system
Not satisfiedVery satisfied
12345
Q4. In general, what level of improvement in students’ academic success did you observe after you submitted academic alert notices?
Most students / Some students / No studentsSignificant improvement detected / / /
Some improvement detected / / /
Q5. What comments do you have regarding the Academic Alert system?
Q6. What suggestions do you have to improve the Academic Alert system?
Thank you for participating in this survey!
Appendix B. Response Frequencies
Q1a. Currently instructors may use the Academic Alert system to report concerns in the following categories. Please indicate whether you think these categories should be retained in the alert system during future terms.
Attendanceretaindelete
Frequency / Percent / Valid PercentValid / delete / 2 / 1.3 / 1.3
retain / 153 / 97.5 / 98.7
Total / 155 / 98.7 / 100.0
Missing / System / 2 / 1.3
Total / 157 / 100.0
Q1b. Currently instructors may use the Academic Alert system to report concerns in the following categories. Please indicate whether you think these categories should be retained in the alert system during future terms.
Poor Quality Workretaindelete
Frequency / Percent / Valid PercentValid / delete / 27 / 17.2 / 17.6
retain / 126 / 80.3 / 82.4
Total / 153 / 97.5 / 100.0
Missing / System / 4 / 2.5
Total / 157 / 100.0
Q1c. Currently instructors may use the Academic Alert system to report concerns in the following categories. Please indicate whether you think these categories should be retained in the alert system during future terms.
Missing Workretaindelete
Frequency / Percent / Valid PercentValid / delete / 11 / 7.0 / 7.2
retain / 142 / 90.4 / 92.8
Total / 153 / 97.5 / 100.0
Missing / System / 4 / 2.5
Total / 157 / 100.0
Q1d. Currently instructors may use the Academic Alert system to report concerns in the following categories. Please indicate whether you think these categories should be retained in the alert system during future terms.
Cannot Pass with Remaining Courseworkretaindelete
Frequency / Percent / Valid PercentValid / delete / 13 / 8.3 / 8.3
retain / 143 / 91.1 / 91.7
Total / 156 / 99.4 / 100.0
Missing / System / 1 / .6
Total / 157 / 100.0
Q1e. Currently instructors may use the Academic Alert system to report concerns in the following categories. Please indicate whether you think these categories should be retained in the alert system during future terms.
Instructor Commentsretaindelete
Frequency / Percent / Valid PercentValid / delete / 5 / 3.2 / 3.3
retain / 147 / 93.6 / 96.7
Total / 152 / 96.8 / 100.0
Missing / System / 5 / 3.2
Total / 157 / 100.0
Q2. If you would like to see additional categories for reporting options, please list them here.
Open-ended question. See Appendix C.
Q3. Please indicate your level of satisfaction with the Academic Alert system
Not satisfiedVery satisfied
12345
Frequency / Percent / Valid PercentValid / not satisfied / 3 / 1.9 / 1.9
2 / 8 / 5.1 / 5.1
3 / 31 / 19.7 / 19.7
4 / 74 / 47.1 / 47.1
very satisfied / 41 / 26.1 / 26.1
Total / 157 / 100.0 / 100.0
Mean = 3.90
Q4a. In general, what level of improvement in students’ academic success did you observe after you submitted academic alert notices?
Most students / Some students / No studentsSignificant improvement detected / / /
Frequency / Percent / Valid Percent
Valid / no students / 79 / 50.3 / 56.0
some students / 54 / 34.4 / 38.3
most students / 8 / 5.1 / 5.7
Total / 141 / 89.8 / 100.0
Missing / System / 16 / 10.2
Total / 157 / 100.0
Q4b. In general, what level of improvement in students’ academic success did you observe after you submitted academic alert notices?
Most students / Some students / No studentsSome improvement detected / / /
Frequency / Percent / Valid Percent
Valid / no students / 37 / 23.6 / 25.5
some students / 93 / 59.2 / 64.1
most students / 15 / 9.6 / 10.3
Total / 145 / 92.4 / 100.0
Missing / System / 12 / 7.6
Total / 157 / 100.0
Q5. What comments do you have regarding the Academic Alert system?
Open-ended question. See Appendix C.
Q6. What suggestions do you have to improve the Academic Alert system?
Open-ended question. See Appendix C.
Response Submissions by Date
Frequency / Percent / Valid PercentValid / 14-Nov-2011 / 76 / 48.4 / 48.4
15-Nov-2011 / 9 / 5.7 / 5.7
16-Nov-2011 / 6 / 3.8 / 3.8
17-Nov-2011 / 4 / 2.5 / 2.5
18-Nov-2011 / 4 / 2.5 / 2.5
20-Nov-2011 / 1 / .6 / .6
21-Nov-2011 / 1 / .6 / .6
23-Nov-2011 / 1 / .6 / .6
29-Nov-2011 / 47 / 29.9 / 29.9
30-Nov-2011 / 5 / 3.2 / 3.2
01-Dec-2011 / 2 / 1.3 / 1.3
07-Dec-2011 / 1 / .6 / .6
Total / 157 / 100.0 / 100.0
Appendix C. Verbatim Responses to Open-Ended Questions
Q2. If you would like to see additional categories for reporting options, please list them here. (34 responses)
Respondent ID / Response5 / Willingness to seek assistance, participation
6 / I think poor attendance and missing work go hand in hand: if they're not in class they don't get the homework and so they can't hand it in. I think the two cateories are somewhat redundant.
I also believe that the students are well aware of their overall grade and figure out exactly how much work they have to do to get the grade they want, that's why the category "cannot pass with remaining coursework" could be deleted, in my opinion. I have students who do minimal work for the final, because they've figured out that whatever low grade they might get on the final is good enough to keep their overall grade where they want it to be.
9 / None that I can think of.
21 / Disciplinary concerns
43 / I would like Instructor Concerns as a category. Or maybe that's what you want in Comments...?
50 / Disruptive to class.
53 / Would be nice if the instructor comments were not limited in space as much as they are. Can't always get the problem across in that small of space.
59 / It would be nice to have separate categories under "Missing Work" such as missing tests, missing quizzes, and missing classwork. I think this would let the advisers and student know specifically what the student is missing.
64 / Maybe poor test scores, right now I reported that as part of Poor Quality of Work.
66 / Missed Exam
70 / I would like the "Quality of Work" category offer the following choices: poor, marginal, and clear. "Clear would, by default indicate that the work is good.
76 / I think there is a difference between poor attendence and stopped attending ... I think it would be nice to be able to indicate poor attendence as well as stopped attending.
84 / Include a category about the student's ATTITUDE TOWARD LEARNING.
86 / Extremely Difficult to Pass with Remaining Coursework or Significant Improvement Required to Pass with Remaining Coursework
93 / Some students need specific kinds of assistance: time management skills, writing skills, tutoring in particular areas -- we need room to comment on this.
98 / Instead of 'Cannot Pass', perhaps a more appropriate category would be 'Highest Reasonably Possible Grade' with a space to enter a grade. That gives some leeway in saying F or D when a C is numerically possible but not going to happen. It also provides information to an advisor about whether the student is going to fail with an F or not pass with a D.
I am not happy with the two choices "Poor Quality Work" and "Missing Work". I would prefer to separate based on type of work. Some students don't bother doing their homework well. Other students have trouble with exams after doing all their homework well. The two need different kinds of advice from their advisors and "Poor Quality Work" seems to be the only way to tag both these situations.
101 / No
104 / Allow for positive input from instructors for the academic advisor's use.
109 / Online Course: Not reading/responding to Instructor's email.
110 / Student does not respond to email
111 / You may want to include "missing lab reports" which was one of the items my alert involved and perhaps "N/A" if that is not a component of a particular class. I emphasize the 'comments' allow me to explain where the problem is that can't be simply addressed by check boxes.
114 / (1) Lack of understanding of background material from prior courses.
(2) Perhaps we could add a category that indicates the student is not attending office hours or seeking help even when they are struggling.
120 / Current Class Grade field, a check box to indicate if the teacher wishes for the adviser to contact the student.
126 / na
128 / I've used this primarily for "attendance", which in an online class means that the student hasn't accessed the documents I provide for the course. I'm not totally sure whether or not the students use the email system. I'd like to start requiring a "read" receipt when the student reads the email, but don't know if D2L's email function allows for that.
130 / Some notion of the time of grading scheme would be helpful so if a student can drop the lowest test score that is indicated by the instructor.
132 / A dropping grade alert due to poor attendance
136 / howbout "improving", or some other positive feedback as a result of previous intervention?
138 / Perhaps an optional field where an instructor could post a current projection of the student's final grade. I already do this on D2L for the student, but I had to put it in the comments field to get it to the advisor.
144 / Frequent distraction of instruction from some students during the lecture time due to their non-course related activities such as texting, excessive talking when the class is not in discussion time, etc.
146 / Poor participation (sleeping, unapproved use of electronics, lack of attention) in classroom.
149 / current score
150 / Attitude and willingness to work cooperatively with instructor and other students.
152 / Can pass but only with serious improvement.
Q5. What comments do you have regarding the Academic Alert system? (97 responses)
Respondent ID / Response2 / has alerted both student and advisors and we have been able to get students back on track....LOVE IT
3 / I think that it is a great service for faculty and students. My only concern is those students who take no action. Are they being notified by the advisors? Or are they just not taking action. I wish there was a way to have the advisor report or indicate that a conversation and action plan was established by the advisor and the student.
6 / It's the first time that it worked, although not with all students. Maybe, because the students got the messages as well and not just the advisors. Keep it going longer, after the withdrawl period. But maybe the new system "First grade???" (I can't remember the name) will do that.
7 / Some advisors are not responsive to academic alerts being filed, which reduces the effectiveness of the academic alerts system. In some cases, there was no indication that the advisor or the student had read the alerts and comments I submitted.
9 / It was both easy and effective.
10 / The system is not very user friendly.
12 / I teach at Tulsa Community College and at the University of Oklahoma. I wish they would adopt a similar system. I had several students who salvaged their grade after they were made aware of what they were lacking.
15 / A very effective tool
16 / The AAS is overkill that further adds to the problem by removing more responsibilities from students for their own academic career. Reported information is simply duplication since students already have access everything via D2L. Perhaps it helps information flow past the barrier of FERPA. Students do not need to be coddled any more than they already are.
18 / a waste of time
19 / I appreciate knowing I have properly reported issues with student attendance and grades. As a first-time instructor this semester, this was helpful.
20 / I used the system for two of my students who did not respond to the alerts in any fashion. I think the alerts are good though.
23 / The students I was concerned about had work related issues that could not be resolved with this system. I think the Academic Alert system is useful.
24 / I've only used the system for a master's class (student had stopped attending early in the semester and never came back), so I can't comment very well just yet. I do, however, think this system will greatly benefit my undergraduate students in the spring.
26 / I only had to use it for one student who did not respond to either an email I sent her earlier or to what was sent through the academic alert system.
27 / The comment field is limited to 250 characters. I must have rewritten my comments 20 times before I could final get the system to accept them. An open number of characters would be preferred.
28 / This seems a system to justify so-called student retention. I had several students who have not appeared this semester yet whose names are on my roll sheet. I used the AA system, and nothing happened, save a note from an administrator wondering whether a student should get full refund for never attending a class. Was I sure he never attended? Yes, I am. This is a meaningless gesture, and despite faculty filling in forms and advisors sending emails and making calls, nothing happened in my class.
30 / 1 - It would be nice to see what was actually sent to the student and advisor. It would make responding to both easier.
2 - I have only used it one time for one purpose, to indicate that it appeared a student had no chance of passing the class. I hadn't thought about using it for other purposes. Releasing the pilot results would be beneficial in terms of helping me decide whether or not to use it for the less serious cases.
Thanks.
31 / It is a great idea. However, I did not notice any changes in the students I marked on the system. At the very least, I was expecting a call or e-mail from both the students and their respective academic adviser. Does the system actually work? After I submitted my alerts, I had no feedback: were the students contacted, did their academic adviser talk to them?
33 / It seems weird to be sending information to a student and their advisor at the same time. For example, I wasn't sure what I could include due to Ferpa.
35 / It is worthwhile
36 / There should be a follow up email sent by the student advisor to the faculty who submitted the academic alert to indicate if a meeting between the student and his/her advisor occurred and the steps in place that were communicated with the student to correct the issues that caused the alert.
37 / I think it's a great idea to have a system that allows faculty to communicate directly with a student's advisor. It would be interesting to know what advisors do with this information once they receive it. For example, do they summon students to a meeting to discuss their performance?
39 / The two students who received academic alerts ended up dropping the class, so I guess that might count as "improvement."
40 / I would like to see a printable report of the messages I sent.
42 / The system is valuable and should be maintained.
43 / The system works for me. Do the students respond to it? Their advisors?
45 / There is no feedback to the instructor. I have students disappear who are still on my rolls, and I wonder what happened to them.
48 / I think that it is a good idea, but the follow up done by the advisors needs to be a little more robust. In cases where I submitted something through the academic alert system the advisor only sent an email. This is not sufficient. I can (and do) sent emails to students that are struggling. I need the advisor to meet with the student and discuss the problem and then I would like to receive a report, either from the student or from the advisor.
50 / It gave me an opportunity to bring a student's absences, etc. to her counselor's attention.
51 / It would be nice if you enter the system to make a second annotation that you can overwrite or replace the date of the annotation with the new date. My memory is that I was able to put in new text for a comment, but it still carried the same date as my original comments, which was confusing.
54 / I'd like more information about when to use the system. I talk to my students about the issues in the Alert Systems on a regular basis. Do I still need to use the Alert System? I'd prefer not to. If alerts become a requirement, I'll just use the alert system and stop pulling students aside to talk about their performance.