Achieving College Success (ACeS) Now

PI: LaraineDemshock, Northampton Community College

Final Report

July 2012

Submitted by Grace I. L. Caskie, Ph.D.

Table of Contents

  1. Introduction and study overview
  2. Description of faculty participants
  3. Description of student participants
  4. Description of courses included
  5. Professional development seminars
  6. Evaluation of Faculty Learning Circle (FLC)
  7. Faculty perception of use of UID strategies
  8. Student perception of faculty use of UID strategies
  9. Comparison of faculty and student perceptions of UID strategy use: By semester
  10. Course grades and course completion: By semester
  11. Comparison of student self-report of disability status with Disability Services Offices records
  12. Course completion rates for students with a disability registered with the Disability Services Office
  13. Pre-test to post-test change in students’ academic self-efficacy
  14. Overall conclusions
  1. Introduction and study overview

ACeS Now was designed to impact the postsecondary academic success of students with disabilities by enhancing their classroom experience through faculty adoption and infusion of transition support information and Universal Instructional Design (UID) strategies. This research project was conducted by members of Disability Services staff (LaraineDemshock, Kathi Jo Weinert, and Leigh Cundari) at Northampton Community College (NCC) in Bethlehem, PA.

Faculty were trained on UID strategies via two professional seminar workshops conducted by project staff in May 2009. The Fall 2009 semester was used as the control semester; during this semester, faculty were engaged in a Faculty Learning Circle (FLC), planning how the UID strategies could be integrated into their teaching. A focus group was conducted by this evaluator in December 2009 to evaluate the functioning of the FLC. In the Spring 2009 and Fall 2010 semesters, the UID strategies were implemented by faculty.

In all three semesters, data were collected on faculty perceptions of UID strategy use and student perceptions of UID strategy use; also, student grades in participating courses and student demographic information were obtained. In the two implementation semesters, a pre-test and post-test assessment of academic self-efficacy was also administered to all participating students. Faculty demographics were also assessed at the faculty development seminars conducted at the beginning of the study.

  1. Description of faculty participants

A group of 13 NCC faculty were recruited to participate in ACeS Now. One faculty member participated in the control semester (Fall 2009) and first implementation semester (Spring 2010) but dropped out of the project prior to the second implementation semester (Fall 2010). Demographic characteristics of the 13 faculty participants are shown in Table 1.

Table 1.Demographic characteristics of the participating faculty (N = 13).

Status / Full time / n = 12
Part-time / n = 1
Rank / Professor / n = 1
Associate Professor / n = 4
Assistant Professor / n = 7
Other / n = 1
Experience / 30-39 years / n = 1
20-29 years / n = 2
10-19 years / n = 3
5-9 years / n = 5
1-5 years / n = 1
1st year / n = 1
Degree / Doctoral degree / n = 5
Master’s degree / n = 7
Other professional degree (e.g,. MD, DDS, JD, DVM) / n = 1
NCC Campus / Main / n = 10
Monroe / n = 2
Mt. Pocono / n = 1
Age Group / 50-64 years / n = 3
40-49 years / n = 5
30-39 years / n = 5
Gender / Female / n = 11
Male / n = 2
  1. Description of student participants

A large number of NCC students participated over the course of the project; however, not all students provided data for demographic characteristics, shown in Table 2. Demographic characteristics were collected via the UID measure only in Fall 2009; however, for student participants in Spring 2009 and Fall 2010, demographic characteristics were collected via three sources: (1) UID post-test, (2) ASES pre-test, and (3) ASES post-test. These information sources were merged to provide the most complete demographic picture. It should be noted, in general, that some non-independence of observations occurred in the ACeS Now datasets due to students being enrolled in courses with multiple participating faculty; this occurrence is noted throughout the document where relevant and is described in greater detail in interim reports made by this evaluator. Despite some duplication of individuals, the records of grades earned in each course and student reports of faculty use of UID strategies were considered to be independent, given that the grades and perceptions were for different courses; however, duplicate student reports of self-efficacy were not considered, and the sample size was reduced accordingly for analyses of that variable.

Table 2.Demographic characteristics of student participants: By semester.

Fall 2009
Control
(N = 724) / Spring 2010
Implementation 1
(N = 628) / Fall 2010
Implementation 2
(N = 582)
Gender (% female) / 62.3%
(225 of the 362 who responded) / 53.1%
(306 of the 576 who responded) / 57.9%
(309 of the 534 who responded)
Disability
(% yes-self report) / 11.6%
(42 of the 361 who responded) / 12.8%
(74 of the 578 who responded) / 12.5%
(67 of 535 who responded)
Race (% white) / 65.9%
(234 of the 355 who responded) / 58.7%
(336 of the 572 who responded) / 56.4%
(300 of the 532 who responded)
  1. Description of courses included

A total of 15 courses (some with multiple sections) were used in the ACeS project during the course of its implementation. As shown in Table 3, although enrollment and the number of sections for some courses (e.g., CISC101, CJUST101) were relatively stable, the number of students and sections per course was much smaller in Spring 2010 for certain courses (e.g., EDUC115, MATH140, ENGL151C) in comparison to Fall 2009 and Fall 2010. This shift may have influenced the comparability of the three semesters slightly. In addition, several other course changes during the implementation of the three-semester project should be taken into account when comparing the three semesters’ data. Specifically, the EARL106 course that was included in Fall 2009 and Spring 2010 was replaced by EARL217 in Fall 2010 due to a change in what the instructor was able to teach. The ENGL020 course that was included in Fall 2009 was replaced by the new course number ENGL026 in Spring 2010 and Fall 2010; however, ENGL020 and ENGL026 were considered comparable courses. PSYC103 was taught in an online format during Fall 2010 only and had about 1/3 the enrollment of the previous two semesters. SOCA103 was not used in Fall 2010.

Table 3.Number of students and sections per course: By semester.

Fall 2009 / Spring 2010 / Fall 2010
Course / Total n
per course / Number of sections / Total n
per course / Number of sections / Total n
per course / Number of sections
BIOS105 / 45 / 2 / 47 / 2 / 70 / 3
BUSA101 / 32 / 1 / 60 / 2 / 33 / 1
CISC101 / 33 / 1 / 32 / 1 / 32 / 1
CJST101 / 70 / 2 / 67 / 2 / 69 / 2
CMTH102 / 79 / 3 / 104 / 4 / 107 / 4
CMTH110 / 26 / 1 / 51 / 2 / 52 / 2
EARL106/217* / 61 / 2 / 30 / 1 / 32 / 1
EDUC115 / 57 / 3 / 19 / 1 / 52 / 2
ENGL020/026* / 14 / 1 / 47 / 3 / 17 / 1
ENGL151C / 92 / 4 / 23 / 1 / 65 / 3
MATH140 / 53 / 2 / 34 / 1 / 52 / 2
PSYC103* / 70 / 2 / 70 / 2 / 23 / 1
SOCA103* / 101 / 3 / 69 / 2 / 0 / 0
Total N / 733 / 27 / 653 / 24 / 604 / 23

*Notes: EARL106 that was included in Fall 2009 and Spring 2010 was replaced by EARL217 due to a change in what the instructor was able to teach. The ENGL020 course that was included in Fall 2009 was replaced by the new course ENGL026 in Spring 2010 and Fall 2010; ENGL020 and ENGL026 are considered comparable courses. PSYC103 was taught in an online format during Fall 2010 only. SOCA103 was not used in Fall 2010. Sample sizes are slightly higher here than in Table 1 due to students being enrolled in multiple courses per semester.

  1. Professional development seminars

A key component of the ACeS Now project was to deliver two faculty professional seminars to the participating faculty that covered these topics: (1) Understanding the Transition Needs of College Students with Disabilities and (2) Universal Instructional Design in the College Classroom. The content of these seminars was developed and presented to faculty participantsby Disabilities Services staff; all handouts were provided in a well-organized binder that faculty reported finding to be a helpful resource (see evaluation of Faculty Learning Circle below) as they considered implementing Universal Instructional (UID) principles into their courses. This evaluator attended and observed both seminars.

Before each seminar, the 13 participating faculty completed a survey assessing knowledge of the topic; the same survey was again completed at the end of each seminar as a post-test.

Analysis of the data collected from the 15-item survey for the first seminar indicated that statistically significant gains occurred for faculty with regard to knowledge about the transition needs of college students (t(12) = 4.15, p = .001). The average score at pre-test was 11.62 (range = 9 to 14) and at post-test was 13.38 (range = 12 to 15). However, for the data collected from the 10-item pre/post survey for the second seminar, no significant difference was found (p = .104). This result is likely due to a ceiling effect; the average scores on both pre-test and post-test were quite near the maximum of 10 points possible (pre-test mean 8.69; post-test mean 9.00).

Faculty were also surveyed regarding their evaluation of and satisfaction with the two workshops. All survey items were rated on a four-point scale, with higher scores indicating better responses (e.g., more knowledge, better rating). Overall, faculty were highly satisfied with both workshops; the average overall rating of Workshop #1 was 3.89 out of a possible 4 points and of Workshop #2 was 4 out of a possible 4 points. In addition, statistical analysis of the participants’ ratings of their knowledge of the seminar topics before and after the workshop indicated a significant gain in self-ratings of knowledge after Workshop #1 (t(12) = 8.39, p < .001) and after Workshop #2 (t(12) = 6.88, p < .001). The average, minimum, and maximum values for responses to both surveys are shown in Table 4.

Table 4.Faculty evaluations of professional development seminars (N = 13).

Abbreviated Item / Workshop #1: Understanding Transition Needs of College Students with Disabilities / Workshop #2: Universal Instructional Design in the College Classroom
M / Min / Max / M / Min / Max
1. Knowledge before attending / 2.38 / 1 / 3 / 2.54 / 2 / 4
2. Knowledge after attending / 3.85 / 3 / 4 / 3.81 / 3 / 4
3. Instructor’s knowledge of subject material / 4.00 / 4 / 4 / 3.92 / 3 / 4
4. Instructor’s ability to actively involve participants / 3.54 / 3 / 4 / 3.77 / 3 / 4
5. Organization of presentation / 3.85 / 3 / 4 / 4.00 / 4 / 4
6. Examples provided in workshop / 3.85 / 3 / 4 / 3.85 / 3 / 4
7. Handouts provided in workshop / 3.69 / 3 / 4 / 3.92 / 3 / 4
8. Length of workshop / 3.38 / 3 / 4 / 3.77 / 3 / 4
9. Overall Rating of Learning Experience / 3.89 / 3 / 4 / 4.00 / 4 / 4
  1. Evaluation of Faculty Learning Circle (FLC)

Another key element of the ACeS Now project was the creation of the FLC with regular meetings of FLC members to collaborate and discuss ways to integrate UID principles into their courses. This evaluator observed a FLC meeting held in October 2009. The faculty were engaged and showed a high level of collaboration. To evaluate the functioning of the Faculty Learning Circle within the larger Achieving College Success Now (ACeS) project, several activities were undertakenby this evaluator to gain the perspectives of various invested parties; specifically: (1) the two faculty leaders of the FLC were interviewed on December 9, 2009, (2)a brief 8-item survey was distributed to the 8 individuals present at the December 14, 2009 focus group and to the remaining 5 individuals at a later time, and (3) a focus group was conducted of 8 faculty participants including one FLC leader on December 14, 2009. A summary of the findings of these three assessments was discussed with the project staff on February 3, 2010, who were then interviewed regarding their experience with the ACeS Now project activities. A written report of all of the above activities was submitted to the project staff in March 2010 and is also summarized below.

(1)Interview with Faculty Leaders. The interview conducted with the two faculty leaders identified a number of strengths of the FLC and several areas of growth. Strengths included the following: (a) Focusing on new teaching strategies, brainstorming with other faculty, and getting immediate feedback and support from others; (b) Receiving support from ACeS project staff made it easy for them to participate as leaders; (c) Participating as a leader resulted in having an even richer experience as they spent even more time than the other faculty participants thinking about these issues; (d) Including the faculty leaders at the planning meetings was essential to planning good discussions at the FLC to provide the perspective of the faculty in the classroom; and (e) Learning about the IEP process in which students with disabilities participate prior to arriving at college was helpful to understanding what students may expect in terms of “hand-holding” or specific accommodations. Two areas for possible growth were identified as (a) Finding ways to improve the FLC experience for faculty at the Monroe campus, for example increasing the number of faculty participating so that different groups could be formed there each week similar to what was done at the Main campus or possibly budgeting funds to have dinner meetings in the evening that might be easier for people from both campuses to attend, and (b) Learning about specific disabilities (e.g., ADD) and the best strategies to use for that specific disability would be a good topic for future FLC discussions.

(2)Survey Results. As shown in Table 5, the 8-item survey on the Faculty Learning Circle (FLC) experience showed that the faculty participants found participation to be valuable, their expectations for the FLC were met, they were more knowledgeable about Universal Instructional Design (UID) and the needs of students with disabilities, and the FLC would have a lasting impact on their teaching. However, fewer faculty endorsed the usefulness of Blackboard as part of the FLC – particularly for discussion. Only half of the participants (54%) indicated that they had shared FLC information with other colleagues.

Table 5.Results from survey of Faculty Learning Circle experience

Item / Median / Mode / Strongly
Agree / Somewhat
Agree / Neutral / Somewhat
Disagree / Strongly
Disagree
1 / Participating in the FLC was a valuable experience. / 5 / 5 / 100%
2 / My expectations were met for participating in the FLC. / 5 / 5 / 75% / 25%
3 / Participating in the FLC has helped me to be more knowledgeable about the needs of students with disabilities. / 5 / 5 / 85% / 8% / 8%
4 / Participating in the FLC has helped me to be more knowledgeable about UID. / 5 / 5 / 85% / 15%
5 / I have shared information from the FLC with other colleagues. / 5 / 5 / 54% / 31% / 15%
6 / Blackboard was a useful way to share information with other FLC members. / 4 / 5 / 39% / 23% / 31% / 8%
7 / Blackboard was a useful way to discuss topics with other FLC members. / 4 / 4 / 15% / 39% / 15% / 23% / 8%
8 / Participating in the FLC will have a lasting impact on my teaching. / 5 / 5 / 92% / 8%

(3)Focus Group. The focus group of 8 of the 13 participating faculty that was conducted by this evaluator also identified several strengths and areas of possible growth for the FLC. Strengths were identified as: (a) the support and resources provided by project staff (especially the binder of training materials from the workshops) in figuring out how to implement the UID strategies; (b) the support of other faculty in collaboration on ideas for teaching and in lessening the typical isolation associated with teaching; (c) having discussion questions posted on Blackboard ahead of time as well as the summaries posted after the meeting; (d) the opportunity to enjoy the creative aspect of teaching as they re-thought their course. Areas for possible growth were identified as: (a) finding better ways to manage both the large group discussion and the small group discussion in the same setting; (b) utilizing the Blackboard software more effectively; (c) working to reduce/eliminate frustration associated with not being able to implement the UID strategies immediately; (d) obtaining more information about specific disabilities and which of the UID strategies works best for those specific disabilities; and (e) learning how to anticipate the student needs that may arise in a particular upcoming course.

(4)Interview with ACeS project staff. The three project staff (Demshock, Weinert, Cundari) were interviewed in February 2010. Staff also identified a number of strengths and challenges with the FLC and the project as a whole. These were reported in the March 2010 report submitted by this evaluator and are reproduced in their entirety here in this report in Table 5 below.

Table 5.Strengths and challenges related to the FLC as identified by project staff.

Strengths / Challenges
The ability to be flexible was essential to the success of the grant. / Balancing the sometimes-competing forces of (a) needing to follow the research plan for grant activities and (b) faculty preferences for the timing of certain activities was a challenge; an example of a challenge was having planned to hold the UID workshops soon after the start of the fall semester so faculty would not be frustrated by learning strategies they could not implement during their fall course planning, but also wanting to be responsive to faculty requests that the workshops take place in May.
Collaboration and contact between faculty occurred that would have been unlikely under normal circumstances. / Obtaining the deans’ commitment for the participating faculty (especially the one adjunct) to teach the same courses for three semesters in a row to conform with the planned research design was a difficult political challenge.
Setting a “team atmosphere” was very important. Similarly, having faculty view the discussions as a “safe” place to air concerns was helpful in having honest opinions shared. / Having faculty on two campuses made it difficult to find a good meeting time, and although using the distance technology helped, it did not completely replicate the feeling of a meeting with all persons present in the same place.
Individually selecting the faculty to participate made it feel “special” and helped to get faculty to participate better than if a general call for anyone interested had been made. / The size of the group was a challenge at times to get everyone to participate.
Having the Disability Services office had at least two benefits: (a) information about disabilities was able to be readily accessed and (b) faculty did not feel they were being evaluated, but were rather partners in this project.
An unanticipated benefit of the grant was the strengthened relationships between the participating faculty and the project staff in their roles in the Disability Services office and a sense that the faculty better understood that Disability Services staff do have knowledge about teaching that could be of use to them. These new relationships have opened up better communication between the faculty and staff, even about issues unrelated to the grant or a specific student.
Learning about what faculty are doing in their classrooms has also been helpful for the Disability Services staff to improve what they do and their understanding of the challenges that students and faculty may face (e.g., use of Blackboard).
  1. Faculty perception of use of UID strategies

At the end ofeach of the three semesters (i.e., Fall 2009, Spring 2010, Fall 2010), faculty reported their perception of how much they had used various UID strategies in the relevant courses using a 30-item survey format. Twenty-one of these items also appeared on the student perception of UID strategy use; 9 items were unique to the faculty survey. The mean response for each item by semester is shown in Table 6.