PREDICTING STUDENT SUCCESS IN A DEVELOPMENTAL ENGLISH LANGUAGE PROGRAM: CAN IT BE DONE?
This paper reports on the relative utility of different kinds of data about student skills and performance, including ACT scores, entry placement tests, faculty hunches, past grades, and outcomes assessment scores, for predicting student academic success, defined as graduation. The students in question are deaf and hard of hearing college students enrolled in a developmental English program.
The finding that such data are not as highly predictive as one might think is interpreted to mean we should “never say ever” when confronted with students with significant English language deficits.
The Bottom Line
• At the top of concerns for all institutions of higher education is student success.
• Graduation rates are a measure of student success.
• Therefore, all institutions of higher education are concerned about graduation rates.
• NTID is no exception.
The Student Body
•NTID has about 1,100 students, with varying degrees of deafness. The college offers Associate of Occupational Studies (AOS), Associate of Arts & Sciences (AAS) and Associate (AS) Degrees. Many deaf students, supported by NTID, are matriculated in baccalaureate programs in the other colleges of RIT.
•Each September, we accept somewhere in the region of 220 new students (this number excludes those NTID-supported students who are directly accepted to baccalaureate programs at RIT). Most of these freshmen are accepted into NTID associate-degree programs. Today’s presentation is based on data related to the performance of these students.
•In our recruitment information and during our open days for prospective students, we place a lot of emphasis on contrasting our graduation rates with those of our competition.
Impressive Graduation Rate
• Overall graduation rate for
RIT deaf and hard-of-hearing students55%
- Associate49%
- Baccalaureate70%
• RIT graduation rate (hearing students)64%
• Deaf and hard-of-hearing students from
other colleges25%*
*National Center for Educational Statistics, 2003
This Presentation
•Because graduation rates are such an important institutional statistic, it would be helpful if we had some method of predicting which students will or will not be successful in terms of graduation. If we could do that accurately, we could be more purposefully selective in who we accept and who, once accepted, we use scarce resources to encourage to stay. This presentation is about how difficult it is to do that.
Institutional Predicament
•About 80% of NTID’s funding comes from the Department of Education. As part of its oversight function, DOE requires regular reporting of enrollment, retention and graduation statistics. Some years ago, the Department established a schedule of improvement in these so-called “performance indicators” approximating to 1%/year growth in each of the categories.
•In addition to the pressure of federal oversight, NTID has had to face two further challenges over the last few years.
First, state vocational rehabilitation agencies, the primary source of financial aid to deaf students, have become significantly less ready to support students out of state, while at the same time tightening their criteria for continued funding of students who may take longer than the norm to graduate.
•Second, the institutional identity of RIT as a whole has begun to rapidly evolve over the last 15 years from a comprehensive university to a doctoral-granting institution with its eyes set on admitting an increasingly more highly qualified student body.
•Finally, despite the fact that 80% of our entering students are not directly accepted into RIT baccalaureate programs, our intake surveys indicate that 85% of them come with the expectation that they will graduate with baccalaureate degrees. Some are able to make the transition during their time at NTID, but for many of them, a baccalaureate degree is an impossible dream.
Strategic Vision2010
•As a result of the upward mobility of RIT, as well as the belief that more highly-skilled students will equate to a higher graduation rate, three years ago NTID engaged in a strategic planning effort whose major recommendation, couched in a document entitled Strategic Vision2010, was to shift the make-up of the student body from its existing proportion of approximately 55% associates students & 45% baccalaureate students to one of 35/65. An associated recommendation was to “raise the bar” on minimum entry requirements expressed in terms of ACT composite and ACT reading sub-test scores.
•This recommendation has been controversial insofar as many faculty believe that the Institute was created to offer deaf students who could not otherwise get a post-secondary education, an opportunity to earn a degree. Though it is true that students with low English skills do not graduate at a rate equal to that of students with higher English skills, it is also true that the Institute enjoys much greater success in developing the English literacy skills of such students than probably any other post-secondary institution. And it is also true that some of these students do go on to succeed and graduate with associate and, in a few cases, baccalaureate degrees.
•Supporters of the recommendations, on the other hand, argued that not only would they bring NTID more into line with RIT and improve graduation rates, but they would also, at least to a certain extent, address the problem of 85% of our students expecting to get a baccalaureate degree. They also argue that the AOS degree, while it does lead to entry level jobs, is not as successful in ensuring subsequent promotion in the workplace.
•In the event, the Institute to this point has been unsuccessful in attracting a higher proportion of more highly-skilled students. Because of the federal requirement that we maintain a student body of 1,100, entry requirements have not been raised and Strategic Vision2010 has become Strategic Vision2011.
Five Questions
Question #1
How well do entry placement test scores in general predict academic success?
Question #2
How well do entry writing test data, specifically of students with low English skills, predict academic success ?
Question #3
How well do pre-entry ACT scores predict academic success?
Question #4
How well do data associated with past course performance predict performance in subsequent courses?
Question #5
In general, does the widespread belief that deaf students don’t graduate primarily because of deficits in English literacy hold water?
Institute Developmental English Curriculum
•There are three developmental strands of courses in the NTID English curriculum. These strands are Academic Writing, Nonfiction Reading and Literature.
•Each strand includes courses at different levels of difficulty. There are four levels: Introductory Level (Level A); Fundamental Level (Level B); Intermediate Level (Level C) and Bridging Level (Level D). Courses in the Academic Writing and Nonfiction Reading strands are offered at each level of difficulty.
Placement Testing
Entry Writing Test
•This test measures students' writing ability. It is used to place students in the Academic Writing strand. Students are given 30 minutes to write a short essay in response to the prompt:
“You are in a new place. Write an essay on your opinions of NTID and the people here. Give reasons and examples.”
Each essay is read by three faculty members. Scores range from 0-100 points with 25 points for each of four categories: organization, content, language use and vocabulary.
Entry Reading Test
• This test measures students' reading ability at entry. It is used to place students in the Nonfiction Reading strand. Students take a two-hour reading and vocabulary test to demonstrate their skills in English reading comprehension and vocabulary. The test results in a score on a scale of 0-200.
Entering Class (Placement in Writing Levels)
1997 (N=255)
Writing I= 62 (24%)
Writing II = 92 (36%)
Writing III = 62 (24%)
Writing IV= 62 (11%)
Proficient = 62 (4%)
2006 (N=215)
Writing I= 25 (12%)
Writing II = 73 (34%)
Writing III = 68 (32%)
Writing IV= 31 (14%)
Proficient = 19 (9%)
Question #1
How well do entry placement test scores in general predict academic success?
Graduation Data by Entry Writing Level
CLASS OF 2000N=255
ENTRY / TOTAL / TOTAL / TOTAL / TOTAL / TOTAL / TOTAL
WRTNG / STUDENTS / GRADUATED / BAC. DEG. / AAS DEG. / AOS DEG. / NOT-GRAD
LEVEL A / 73 (29%) / 25 (34%) / 1 (1%) / 5 (7%) / 17 (23%) / 48 (66%)
LEVEL B / 66 (26%) / 28 (42%) / 5 (8%) / 8 (12%) / 14 (21%) / 38 (58%)
LEVEL C / 57 (22%) / 25 (44%) / 11 (19%) / 9 (16%) / 5 (9%) / 32 (56%)
LEVEL D / 36 (14%) / 16 (44%) / 13 (36%) / 2 (6%) / 1 (3%) / 20 (56%)
LEVEL PROF / 23 (9%) / 12 (52%) / 8 (35%) / 3 (13%) / 1 (4%) / 11 (48%)
Table One
Discussion
•The table shows that students who placed in Level A on entry graduated at the lowest rate of all five levels, and that students testing as proficient had the highest graduation rates. However, interestingly, the rates of graduation for Levels B-D show almost no difference.
•A closer look at Levels B-D reveals group differences in the kind of degree earned, with Level D students much more likely to graduate with a baccalaureate and Level B students with an AOS degree.
[It is worth noting, however, that when looked at from the point of view of what might be termed “degree acceptability” (with an AAS degree being much more readily accepted than an AOS degree), the difference between Level Ds and Level Cs is significantly less than that between Cs and Bs. 35% of students who enter at Level C graduate with an AAS or a BS compared to 42% at Level D, while Level B students at 20% lag far behind.]
•We can conclude therefore that although entry writing scores do not predict graduation per se except at the two extremes, they do have some group predictive value for kind of degree earned
Within Level Graduation Data
LEVEL A / LEVEL B / LEVEL C / LEVEL D / LEVEL PROF(N=73) / (N=66) / (N=63) / (N=36) / (N=23)
GRADUATED / 25 (34%) / 28 (42%) / 25 (44%) / 16 (44%) / 12 (52%)
AV. WRITING / 31.3 / 43.5 / 54.2 / 62.9 / 71.7
AV. READING / 83.5 / 98.8 / 124.0 / 144.0 / 170.0
NOT GRADTD. / 48 (66%) / 38 (58%) / 32 (56%) / 20 (56%) / 11 (48%)
AV. WRITING / 31.8 / 44.6 / 54.6 / 62.7 / 72.3
AV. READING / 79.4 / 102.1 / 124.3 / 146.2 / 153.6
Table Two
•Interestingly, when one looks within the levels, however, there is no pattern predictive of overall graduation. For Level A students, a higher reading score gives a slight advantage, but the converse is true for Level B, C & D students. There is also practically no difference between graduates and non-graduates on the writing test.
•The levels are, of course, artifacts of our curriculum design. Still, there is a natural tendency to think that a “high level C” is more likely to succeed than a “low level C”. These data suggest that we should be careful before jumping to that conclusion.
Question #2
How well do entry writing test data, specifically of students with low English skills, predict academic success ?
•The immediate impetus for this question originated in several aspects of the institutional debate alluded to at the beginning of this presentation under the heading “Institutional Predicament,” in particular, the question, should we continue to accept students with low levels of English literacy or should we “raise the bar” and make ourselves a little more like RIT as a whole? Since we know that Level A students do not graduate at the same rate as their peers at higher levels, should we not be more selective and thereby improve our graduation rate? On the other hand, since DOE is insistent that we maintain our student body at around 1,100, can we not develop some kind of predictive mechanism for selecting the students most likely to succeed from among the large number of Level A applicants?
Assessing the Predictability of Level A Writing Samples
Method
•To assess the extent to which the entry writing tests of low-English students (Level A students) might be predictive of subsequent academic success, eight groups of three essays each were assembled on the basis of the following criteria:
• The essays in each triad received the same entry score
• They had approximately the same number of words
• The students in each triad had very similar reading test scores
• One student in each triad had graduated with an AAS degree.
• The other two students either left the Institute with no degree or graduated with an AOS degree.
• Three groups of faculty were asked to read the essays and predict, based on whatever criteria they wished, which of the essays had been written by the AAS graduate-to-be.
• The three groups were:
Group One: English instructors teaching in the developmental program
Group Two: English and Liberal Arts instructors teaching deaf students in the College of Liberal Arts
Group Three: Academic administrators
• Of the three groups, only faculty in Group One have any significant experience teaching Level A students.
GROUP 1: STUDENT P
“What do you like about NTID and the people here”
When I arrives here on National Tech. Int. for the Deaf on campus, I knew that, this will give me the ideas what the college is adout. There are several things that I like adout NTID, first of all, various students from all over states that came here and other countries. I was in culture shock but slowly accept who they are. Where they come from (lifestyle background). Of course, you learned lots of things adout social thur your friends from your college.
Second things, NTID offer the students many different fields in majors for you to choose. To help the students to stay on the track until they graduated with major degree. This is a wonderfol oppintries for the students to reach their goals and to be successful in future. It will be worth to have a degree when the students finished their career then it will pay off. So the jobs out there can hired them. Understand the reality in college to get ready for the real “outside” world. Sometime it will not going to be easy but learn to challenge it, once you do that, you can be an a leadership or be a successful in many gifted ways.
The only way you learned adout college life is, thur by your own friends, no matter if that person is hearing or not. Accept who you are and move on.
GROUP 1: STUDENT T
As I am new student at the NTID this year. NTID were setting in Rochester, New York. The NTID were offering the many wonderful choose for everyone to pick for their major for the future. The students who are borning as fully deafs or hard hearings that they can getting more time with the focus than attending at the RIT. Many people who were students at the NTID, where they can be society with other deaf people. Most of the college, the number of the deaf students are less on the percentage of the attending. Let’s discuss about my opinions of NTID and the people here.
The NTID were founded by the president of the United State. His name is Lyndon Johnson. He got his express from his neapew. Then, he found the deaf college of technologies. Many of the deaf people who really needing the best education for the future with their major.
The NTID’s place were built by the whole bricks to make look like more on the technologies of the institutes. The classrooms at the NTID have different location in the RIT’s campus.
The RIT were hoping for the future where they wants to keeping up with the deaf students’ attending at the NTID. The NTID always to be there for the big helping with the education with the various of the subjects. They are many professors who receveing the high degree from the university with the wonderful experience. The deaf students will not be too isolation for self. They will be society for many hours.
The conclusion of this opinion is the NTID students who did made a right decision with their University’s choices. The NTID will be always be here in Rochester for next many years.
GROUP 1: STUDENT Z
“Welcome in NTID”
I like about NTID, because I looked at any where in NTID or RIT alike much very nice. But I will get to more look all of buildings on more time. I would feel so more comfortable and encourage for ready to work and study hard into some classes if I need to find very easy to hard levels for some classes. I know I want to interest about good choices which any classes of arts. Because I always like so I can do drawing my own idea or image about just like as comic books. And I like about the people in here. That’s ok, Because I would like be more enjoy to being alone and my life need to easily for improve into my skills. But, If I don’t understand about everything, Some people can help to me for problem slove about somethings. I have to learn with my experience about find new friends or roomate. I don’t like about NTID if someone give to me for more much difficult about some classes for grade or some hard work for final exams. Because I don’t want my grade will getting to flunking for some classes. But I’ll try to best doing that and trying to hardest study for good grades. I don’t like about the people in here, because I wasn’t feel so comfortable with new roommate same my room. I want to need feel so very much comfortable and better for being alone in new room single if I need it for new room. But I must to take my life with patient. I’ll be getting to find much better for beginning of this year in NTID and my goal need to take sucesse in my future.
Results
Faculty / Faculty / AdmnstrtrsGROUP A / GROUP B / GROUP C / TOTALS
N=8 / N=6 / N=7 / 21
Graduated w/
AAS degree
Student P / 6 / 5 / 6 / 17
GROUP 1 / Student T / ∆ / 1 / 0 / 1 / 2
Student Z / 1 / 1 / 0 / 2
Student E / ∆ / 0 / 0 / 0 / 0
GROUP 2 / Student M / 0 / 0 / 0 / 0
Student R / 8 / 6 / 7 / 21
Student C / 2 / 2 / 1 / 5
GROUP 3 / Student F / 5 / 2 / 4 / 10
Student V / ∆ / 1 / 2 / 2 / 6
Student B / 3 / 2 / 2 / 7
GROUP 4 / Student G / 0 / 3 / 3 / 6
Student N / ∆ / 4 / 1 / 2 / 7
Student A / ∆ / 0 / 0 / 3 / 3
GROUP 5 / Student D / 3 / 1 / 0 / 4
Student J / 4 / 5 / 4 / 13
Student L / ∆ / 1 / 0 / 2 / 3
GROUP 6 / Student O / 1 / 0 / 0 / 1
Student S / 6 / 6 / 5 / 17
Student Q / 1 / 0 / 0 / 1
GROUP 7 / Student X / 7 / 5 / 6 / 18
Student Y / ∆ / 0 / 1 / 1 / 2
Student H / 0 / 0 / 1 / 1
GROUP 8 / Student K / ∆ / 8 / 4 / 3 / 15
Student U / 0 / 2 / 3 / 5
Table Three