QEP Survey #3 Findings

QEP Survey #3 Findings

April 10th, 2015

Prepared by:

Jaime B. Henning, Ph.D.

Jean-Paul Philippe, I-O Psychology MS Student

Summary of Methodology and results

In February 2015, an online survey was sent to department chairs and program coordinators in order to gather program-level evidence for student performance in three areas identified via an earlier survey as potential QEP topics (Critical Reading Skills, Information Literacy, and Applying Discipline Specific Concepts). Department chairs were instructed to seek faculty input to complete the survey. Data were provided from 47 program representatives. Of these participants, the majority (76.6%) represented undergraduate programs.

When asked about students’ critical reading skills, 36.1% of respondents indicated that students’ general reading skills are “Good” or “Very Good,” while 34.5% of respondents indicated that students’ reading skills are “Weak” or “Extremely Weak.” Additionally, 31.9% of respondents indicated that students’ discipline-specific readings skills are “Good” or “Very Good,” while 38.3% of respondents indicated that students’ discipline-specific reading skills are “Weak” or “Extremely Weak.” Furthermore, when respondents were asked to indicate sources of evidence used to support their opinions regarding students’ critical reading skills,the majority of participants indicated relying on faculty observationandthe evaluation of student work, while far fewer reported relying on standardized or professional exams, employer/practicum supervisor observations, and student perceptions of their own skills.

Participants were also asked to evaluate students on general information literacy skills. Of the 47 respondents, 21.3% indicated that students are“Good” or “Very Good” with regard to these skills, while 25.5% indicated students are “Weak” in this area. Additionally, when asked about discipline-specific information literacy skills, 51.1% of respondents indicated students are“Ok” in regard to these skills, while 21.3% rated students as “Weak” or “Extremely Weak.” Furthermore, the majority of respondents indicated relying on faculty observationand evaluation of student work to support their opinions, with few relying on standardized or professional exams, employer/practicum observations, and student perceptions of their own skills.

Finally, respondents were asked to rate students’ ability to apply discipline-specific concepts to issues of importance to their field. Of the 46respondents to this questions, 40.5% indicated that students’ability was either “Good” or “Very Good,” while only 12.8% indicated students’ ability as “Weak” or “Extremely Weak.” Furthermore, the majority of participants reported faculty observation and evaluation of student work as the types of evidence used to support their opinions.

Based on these results, it appears respondents more frequently rated students as “Weak” or “Extremely Weak” on Critical Reading Skills (34.5% for general skills, 38.3% for discipline-specific skills), followed by Information Literacy Skills (25.5% for general skills, 21.3% for discipline-specific skills), and Ability to Apply Discipline-Specific Concepts (12.8%). Furthermore, in all cases the majority of respondents indicated relying most frequently on faculty observation and evaluation of student work to provide evidence for their opinions. Far fewer respondents reported standardized or professional exams, employer/practicum observations, and student perceptions of their own skills as available forms of evidence.

Charts and comments are presented below to provide further information. These include greater detail regarding ratings of students’ skills and abilities on each of the three topics, and on the forms of evidence programs rely on to inform these evaluations.

Programs Represented

• Academic Readiness & Eastern Bridge program
• Academic Tutoring
• Agriculture
• Animal Studies
• Anthropology Program Department of Anthropology, Sociology, and Social Work
• Associate Degree Nursing
• Associate of General Studies
• B.S. Psychology
• BA Geography
• BGS: College Tracks
• Biology BS
• Biology MS
• Biology Teaching BS
• BS Earth Science Teaching
• BS Geology
• Clinical Psychology
• CMS (Communication Studies)
• College of Business and Technology - Marketing Department
• Corporate Communication and Technology
• Economics
• Education Pays
• EKU Honors Program
• EKU Libraries
• English 101/102 (First-Year Writing)

• Environmental Health Science
• Environmental Studies BS
• Fly
• General Psychology Masters Program
• Globalization and International Affairs (GLO)
• Government -- Paralegal, Political Science, Public Administration
• History and History Teaching
• I-O Psychology MS Program
• Journalism
• Justice Studies
• McNair Scholars Program
• Medical Laboratory Science
• Noel Studio for Academic Creativity
• NOVA Student Support Services Trio Program
• Pre-Law Pre-Professional Advising
• Pre-Professional Advisor for Health Programs
• Professional Track of BGS: (Adult Completer Degree)
• Public Relations
• Safety, Security, Emergency Management
• Social Work
• Sociology
• Teaching & Learning Center
• Wildlife Management BS

Topic #1: Critical Reading Skills

***Note – Forms of Evidence Indicated by Respondents who Rated Students’ Reading Skills as Weak or Extremely Weak are reported as frequencies.

***Note – Forms of Evidence Indicated by Respondents who Rated Students’ Reading Skills as Good or Very Good are reported as frequencies.

Other Evidences Indicated byRespondents who rated students’ Reading Skills as:

Very Weak:

• Tutors: Students are required to attend study sessions with tutors. Tutors report that poor reading skills affects almost every assignment.

• Consultation records from Writing Fellows program

Weak:

• Some students writing theses can't seem to figure out what is important in what they read, and don't seem to know how to integrate what they read into a coherent introduction.

• We see their writing in reaction to what they’ve read, which exhibits they often may have misread the text, misunderstood the author’s motives, may not be able to identify main points or thesis statements, or find college-level reading too difficult. They often have not been taught to annotate their reading or practiced that until they get to our classes, and then they may have difficulty making that a common practice. They often have even more difficulty reading something they potentially disagree with, finding it difficult to separate their own opinions on the topic from analyzing the work at hand.

• These responses are based on feedback Lara Vance has received from tutors: Reviewing student notes. Dialogue during tutoring sessions. Review of notes/grading comments made by instructors. Since receiving instruction in how to recognize a reading issue (from Dr. Lisa Bosley), more tutors recognize when they are working with someone who has difficulty reading (comprehension, vocabulary, speed, etc.).

• Evaluation of note taking skills. Evaluation of classroom participation.

Ok:

• Data are scattered among different departments that teach required majors courses. I’ve thought about how to address the questions and I honestly have to respond that I do not know how to answer the questions in a why in which I can provide evidence to support my assessment. For example, of the 25-26 classes required in the Land Resources Option (range of numbers because the student may take one or two math classes), Biology teaches 6 of the classes (BIO 111, 112, 316, 320, 495, 521 or 536)….the rest of the courses, about 75%, are distributed among 8 departments (CHE, GLY, GEO, EHS, PHI, STA, MAT, AGR)…and if the student doesn’t do a coop (BIO 349), the required independent study class will be taken in CHE, ENV, GEO, or GLY. I have no idea how faculty in the other programs would rate the students in terms of reading, literacy, or ability to apply discipline-specific concepts. In my opinion, the best class in which to assess this material would be in the required independent study class….but to my knowledge, Biology has never received any feedback regarding these students from the other programs. The same general trend applies to the Natural History Option. I’m not sure how to answer the QEP survey’s questions for a degree program that was designed to be interdisciplinary…maybe Rose Perrine can answer that?? Chuck Elliott

• Advising.

• Observations in the library instruction classroom, while answering reference questions, and during research appointments

• Our assessment is of skills achievement in the areas of critical thinking and written and oral communication, particularly analytic essays. Therefore each year we collect reading response papers, primary source analysis essays, and research papers at various levels in our program to assess our students achievement. PLEASE NOTE THAT THIS SURVEY WILL NOT ALLOW FOR A DIVISION AMONG FACULTY. ON QUESTIONS 3 AND 4 FACULTY EVENLY SPLIT BETWEEN OK AND GOOD. FACULTY IN MAKING THESE RATINGS NOT THAT BECAUSE WE ARE A READING AND WRITING INTENSIVE DISCIPLINE WE DO NOT ATTRACT THOSE WITH VERY WEAK SKILLS. OUR MAJORS TEND TO COME IN WEAK AND BECOME OK or COME TO OUR MAJOR WITH OK SKILLS AND PROGRESS TO GOOD SKILLS.

Good:

• Students in our major preform adequately to quite well on current events quizzes, when they choose to read. The choosing to read is the variable that provides the evidence. Students frequently admit to not reading when they have a low score. Rarely if ever do students state they did not understand the news stories assigned once they take time to read them.

• Reading course examinations

• We have many opportunities in our courses to observe students ability to synthesize and analyze assigned reading materials which suggest good (not very good) read skills for our students

• Semester academic progress reports from participants' instructors.

Very Good:

• Most of our graduate students have scored over 460 on the verbal portion of the Graduate Record Exam. That score places students at 51st percentile, meaning that, among all those who take the exam, our graduate students rank in the top 50% (and most much higher than that). As such, even when entering our program, our graduate students have demonstrated competence in their reading and verbal reasoning skills. In addition, all of our graduate-level courses require significant amounts of reading (of texts as well as, in some classes, the primary scientific literature) and students would not be able to do well in those classes without excellent reading skills. Finally, all of our graduate students must prepare and research proposal and, prior to graduating, a thesis. Doing so requires that students are very familiar with the primary literature which necessary involves much reading.

Topic #2: Information Literacy

***Note – Forms of Evidence Indicated by Respondents who Rated Students’ Information Literacy Skills as Weak or Extremely Weak are reported as frequencies.

***Note – Forms of Evidence Indicated by Respondents who Rated Students’ Information Literacy Skills as Good or Very Good are reported as frequencies.

Other Evidences Indicated byRespondents who rated students’ Information Literacy skills as:

Very Weak:

• Our assessment for the past two cycles has shown that students in our program struggle with integration and cohesion, which indicates a greater issue at hand with information literacy skills. They may be able to find information but then not know what to do with it once they have it. For example, they may appropriately insert a quote and documentation but not integrate smoothly. In the worst cases, they accidentally plagiarize because they struggle with intellectual property concepts requiring them to quote, paraphrase, and cite. Even in the best case scenario, they struggle with the idea that information should be used to enter into a conversation and build their own evidence for development; in other words, their own authorship gets overwhelmed with their research.

• These responses are based on feedback Lara Vance has received from tutors: Reviewing student notes. Dialogue during tutoring sessions. Review of notes/grading comments made by instructors. Since receiving instruction in how to recognize a reading issue (from Dr. Lisa Bosley), more tutors recognize when they are working with someone who has difficulty reading (comprehension, vocabulary, speed, etc.).

• Inability to apply information found in reading assignments to classroom activities.

Weak:

• Advising.

• Students struggle in a media-saturated environment to identify appropriate sources of material in first drafts for news stories and for media law and media research papers. In news classes and in research papers, much time must be devoted to dissecting source credibility so that the second drafts of stories are more in-line with discipline expectations. Weaker students flounder because they do not know where to begin to seek out information. They state they are accustomed to having information provided to them and do not know where to begin to seek out information independently. In response to this need, a faculty member is developing a course in information gathering.

Ok:

• Our assessment is of skills achievement in the areas of critical thinking and written and oral communication, particularly analytic essays. Therefore each year we collect reading response papers, primary source analysis essays, and research papers at various levels in our program to assess our students achievement. PLEASE NOTE THAT THIS SURVEY WILL NOT ALLOW FOR A DIVISION AMONG FACULTY. ON QUESTIONS 3 AND 4 FACULTY EVENLY SPLIT BETWEEN OK AND GOOD. FACULTY IN MAKING THESE RATINGS NOT THAT BECAUSE WE ARE A READING AND WRITING INTENSIVE DISCIPLINE WE DO NOT ATTRACT THOSE WITH VERY WEAK SKILLS. OUR MAJORS TEND TO COME IN WEAK AND BECOME OK or COME TO OUR MAJOR WITH OK SKILLS AND PROGRESS TO GOOD SKILLS.

• Some students had difficulty finding materials to use in writing their thesis work, even though the materials were easily available.

• Tutors' reports regarding the areas in which they are most asked for help.

• Ability to critically analyze professional writing; verbal communications

• Term paper and other short research exercises suggest that students are only okay in both their general and discipline-specific literacy skills.

• I should qualify that generalizations of this sort would depend upon the level of the student in the Honors Program. In Honors Rhetoric (first-year honors students), reading comprehension skills and information literacy skills would be rated by our faculty as "Developing." Most of our honors thesis mentors, however, would regard their thesis mentees' skills in disciplinary reading comprehension and information literacy as "good" or even "very good," particularly by the end of the thesis process.

Good:

• 2015 Program Review data

Very Good:

• Expanding on our response to question 1C, we require all of our M. S. students to take BIO 801 – Scientific Writing and Literature. In this course, students are required to write research proposals and research papers (in scientific journal format), critique the writing of others, give oral presentations, and prepare a poster. The oral presentations and poster are presented and prepared as they would be at a scientific meeting. Many of our graduate students have little or no prior experience with scientific writing and must quickly become reasonably adept at these skills to receive a passing grade. Although students may not be accustomed to having their writing and presentations critically evaluated, such critiques are an essential part of becoming better writers and presenters. Writing proposals and papers and giving presentations also require critical and creative thinking. Students must become familiar with relevant scientific papers and must be able to place their work within the context of previous work, explaining how their results relate to those published previously and what their results have added to what was known previously. As an indication of our effectiveness at improving the creative thinking and effective communication skills of our graduate students, 98 graduate students have taken BIO 801 since 2008 and 93 have earned either an ‘A’ (62, or 63.3%) or a ‘B’ (31, or 31.6%). Only five (5.1%) earned a grade of ‘C.’ These grades are one indication of the very good writing skills of our graduate students In addition, many, if not most, of our graduate courses require some writing in the form of essays or reviews and doing well in the courses requires, among other things, good writing skills. Also, all of our graduate students must write a research proposal and, ultimately, a thesis. These must be approved by their three committee members and such approval requires that a proposal and thesis be well written. Our high M. S. degree graduation rates (indicating successful completion of a proposal and thesis) provide further evidence of the very good writing (and reading) skills of our graduate students.

• Semester academic progress reports from participants' instructors.

Topic #3: Application of Discipline Specific Concepts

***Note – Forms of Evidence Indicated by Respondents who Rated Students’ Ability to Apply Discipline-Specific Concepts as Weak or Extremely Weak are reported as frequencies.

***Note – Forms of Evidence Indicated by Respondents who Rated Students’ Ability to Apply Discipline-Specific Concepts as Good or VeryGood are reported as frequencies.

Other Evidences Indicated byRespondents who rated students’ ability to Apply Discipline Specific Concepts as:

Extremely Weak:

• These responses are based on feedback Lara Vance has received from tutors: Reviewing student notes. Dialogue during tutoring sessions. Review of notes/grading comments made by instructors. Since receiving instruction in how to recognize a reading issue (from Dr. Lisa Bosley), more tutors recognize when they are working with someone who has difficulty reading (comprehension, vocabulary, speed, etc.). Also, feedback from students attending tutoring sessions. For example, students will say to tutors, "I don't know why I need to know this."

• Most students in the Eastern Bridge program have not yet had the opportunity to experience discipline-specific concepts in particular major areas. However, they exhibit a lack of critical thinking, critical understanding, and critical reading skills.

Weak:

• Assessment of courses.

Ok:

• Advising. Assessment of courses.

• Students who have succeeded in courses where they are expected to engage in higher-order thinking skills and who have been exposed to multiple worldviews tend to be successful in making discipline-specific content applications in Media Ethics and Media Law. Test scores indicate that students who have not successful completed previous courses where they were expected to do high levels of analysis simply do not fare well.

• Our assessment is of skills achievement in the areas of critical thinking and written and oral communication, particularly analytic essays. Therefore each year we collect reading response papers, primary source analysis essays, and research papers at various levels in our program to assess our students achievement. PLEASE NOTE THAT THIS SURVEY WILL NOT ALLOW FOR A DIVISION AMONG FACULTY. ON QUESTIONS 3 AND 4 FACULTY EVENLY SPLIT BETWEEN OK AND GOOD. FACULTY IN MAKING THESE RATINGS NOT THAT BECAUSE WE ARE A READING AND WRITING INTENSIVE DISCIPLINE WE DO NOT ATTRACT THOSE WITH VERY WEAK SKILLS. OUR MAJORS TEND TO COME IN WEAK AND BECOME OK or COME TO OUR MAJOR WITH OK SKILLS AND PROGRESS TO GOOD SKILLS.

• Since FYW is a General Education program with a broad focus across disciplines, we have no specific disciplinary concepts for students to apply, per se. However, if we look from the perspective of some generalizable writing applications, then this seems an accurate assessment (OK).

• We have little-no evidence of application of discipline concepts for AGS students, as most of their courses these students take are Gen Ed. Most do not have a concentration in a major.

• Teaching plans are very difficult for students to complete because of their inability to understand and apply professional writing to their practice.

• Again--since our program involves development across the undergraduate careers of the students, assessment would vary by student level. Early evidence from our new assessment device indicates, however, that we may do well to focus on general reading comprehension in particular--even for our honors students.

Good: