Central Washington Unviversity

CENTRAL WASHINGTON UNVIVERSITY

2013 NOEL-LEVITZ PRIORITIES SURVEY OF ONLINE LEARNERS

A COMPARISON OF RESPONSES FROM STUDENTS ENROLLED PRIMARILY ONLINE

TO STUDENTS ENROLLED PRIMARILY ON CAMPUS

I. / Abstract …………………………………………………. / 2.
II. / Executive summary ……………………………….. / 2.
III. / Analysis …………………………………………………. / 2.
A. Importance ………………………………………… / 3.
B. Satisfaction ………………………………………… / 3.
C. Enrollment Factors …………………………….. / 5.
D. Summary Questions ………………………….. / 5.
IV. / Methodology & Detailed Results …………… / 5.
V. / Summary ……………………………………………….. / 14.
Appendix 1 – Terms and Abbreviations …. / 15.
Appendix 2 – An Approximate Mock-Up
of the PSOL Survey ……………. / 16.

I. ABSTRACT
Central Washington University administered the Noel-Levitz (N-L) Priorities Survey of Online Learners (PSOL) survey from 4/29/13 through 5/15/13. The survey was sent to all CWU students enrolled in a course being taught 100% online. 253 students responded from a survey population of 2,392 students for a response rate of 11%.
The 62nd question on the survey (the 4th demographic question) asked students their current enrollment status. This analysis compares the responses of the 157 students who selected “primarily online” to the 90 students who selected “primarily on-campus.” The six responses by students who did not answer question 62 were not included in this analysis.
II. EXECUTIVE SUMMARY

IMPORTANCE –only a few differences between online and on-campus students
Primarily online students’ average response was moderately higher to two of the 36 questions as compared to primarily on-campus students.
Primarily online students’perceive the importance of course availability greater than primarily on-campus students(See Table 1 on page 3)
SATISFACTION–major differences between online and on-campus students
Primarily online students rated their satisfaction moderately higher on nine of 36 questions
Primarily online students gave moderately lower responses to 4 of the 36 questions.
Primarily online student averages were much higher on six of the questions
Primarily online student averages were much lower on one question. See Tables 2 and 3 on page 4.
ENROLLMENT FACTORS–major differences between online and on-campus students
Primarilyonline students rated 6 of the 18 questions moderately higher than primarily on-campus students
Primarily online students rated one question moderately lower. See Table 4 on page 5.
THREE SUMMARY QUESTIONS – major differences

Primarily online students were much higher on “has the online experience met your expectations” but were moderately lower on “overall satisfaction.” than primarily on-campus students.
See Table 5 on page 5.

DEMOGRAPHICS

Primarily online students tended to be older, working, and have family responsibilities. See Table 11 on page 9.

III. ANALYSIS

The PSOL survey consists of 73 questions:

26 standard items rated for both importance and satisfaction
10 campus defined items rated for both importance and satisfaction
18 items assessing pre-enrollment factors – only rated for importance
3 summary questions
15 demographic questions
1 question asking students their general major, or “group code”

The PSOL is partially designed to provide “gap analysis” between importance and satisfaction. This report compares responses of students enrolled “primarily online” to students enrolled “primarily on-campus.”

Questions 1 through 36 ask: “Each item below describes an expectation about your experiences with this program. 'On the left tell us how important it is for your institution to meet this expectation. On the right tell us how satisfied you are that your institution has met this expectation.

The PSOL “importance” and “satisfaction” questions use a seven-point Likert scale.

The scale for importance is: / The scale for satisfaction is:
1 - not important at all / 1 – not satisfied at all
2 - not very important / 2 – not very satisfied
3 - somewhat unimportant / 3 – somewhat satisfied
4 - neutral / 4 - neutral
5 – somewhat important / 5 – somewhat satisfied
6 – important / 6 - satisfied
7 – very important / 7 – very satisfied

IMPORTANCE – ONLY MODERATE DIFFERENCES

Primarily online students rated two questions moderately more important than primarily on-campus students and one questions very much more important thanprimarily on-campus students. Note: Cohen’s d statistic was used to estimate the difference in means. See appendix 1 for a short description of Cohen’s d and “Effect Size.”

Table 1. Differences in “Importance” questions

Primarily Online average / Primarily On-campus average / Primarily Online higher or lower?
12. There are sufficient offerings within my program of study. / 6.51 / 6.12 / ↑
14. I receive timely information on the availability of financial aid. / 5.74 / 5.05 / ↑
36. Campus item: Courses necessary to meet my degree objectives are offered online. / 6.71 / 5.66 / ↑↑

SATISFACTION – QUITE DIFFERENT

Online students are moderately or largely different than on-campus students on 20 of the 36 questions in this question bank.

13 of the 36 questions were moderately different. Online students rated six of the nine questions higher in satisfaction

Table 2. Moderate differences in “Satisfaction” questions(a Cohen’s d between 0.3 and 0.8)

Primarily Online average / Primarily On-campus average / Primarily Online higher or lower?
2. My program advisor is accessible by telephone and e-mail. / 3.98 / 5.42 / 
4. Faculty provide timely feedback about student progress. / 4.33 / 5.16 / 
8. Student-to-student collaborations provide valuable experiences. / 3.96 / 4.74 / 
10. This institution responds quickly when I request information. / 5.94 / 4.75 / ↑
12. There are sufficient offerings within my program of study. / 5.48 / 4.82 / ↑
18. Registration processes are convenient. / 6.37 / 5.61 / ↑
20. The quality of online instruction is excellent. / 5.52 / 4.61 / ↑
22. I am aware of whom to contact for questions about programs and services. / 5.50 / 4.80 / ↑
24. Tutoring services are readily available. / 4.64 / 3.41 / ↑
26. The bookstore provides timely service to students. / 5.54 / 4.91 / ↑
29. Campus item: Faculty-to-student interaction is fostered and encouraged in my online course(s). / 5.43 / 4.77 / ↑
31. Campus item: General academic advising is available to online learners. / 4.52 / 4.83 / 
36. Campus item: Courses necessary to meet my degree objectives are offered online. / 5.51 / 4.68 / ↑

Table 3. Large differences in “Satisfaction” average responses (a Cohen’s d of 0.8 or larger)

PrimarilyOnline average / PrimarilyOn-campus average / Primarily Online higher or lower?
1. This institution has a good reputation. / 6.71 / 5.36 / ↑↑
5. My program advisor helps me work toward career goals. / 6.17 / 4.59 / ↑↑
6. Tuition paid is a worthwhile investment. / 2.93 / 4.71 / 
9. Adequate financial aid is available. / 6.32 / 4.03 / ↑↑
14. I receive timely information on the availability of financial aid. / 6.13 / 3.97 / ↑↑
15. Channels are available for providing timely responses to student complaints. / 6.10 / 3.72 / ↑↑
16. Appropriate technical assistance is available. / 6.63 / 4.52 / ↑↑

IMPORTANCE ON DECISIONS TO ENROLL IN CURRENT PROGRAM – Online students are quite different than on-campus students

Table 4. Seven of the 18 questions relating to the “importance of factors in your decision to enroll at CWU” had average responses that were moderately different.

…tell us how important each ofthe following factors were in your decision toenroll in this program. / PrimarilyOnline average / PrimarilyOn-campus average / Primarily Online higher or lower?
40. Web site / 6.17 / 5.60 / ↑
43.Contact with current students and / or recent graduates of the program / 3.96 / 4.93 / 
44. Ability to transfer credits as factor in decision to enroll. / 6.32 / 5.43 / ↑
46. Financial assistance available as factor in decision to enroll. / 5.65 / 4.56 / ↑
49. Work schedule as factor in decision to enroll. / 6.13 / 5.20 / ↑
51. Convenience as factor in decision to enroll. / 6.63 / 6.28 / ↑
53. Program requirements as factor in decision to enroll. / 6.37 / 5.91 / ↑

THREE “SUMMARY” QUESTIONS

The PSOL includes three summary questions to provide overall ratings.

the average difference between online and on-campus students on question 55 is large
there is a moderate difference on question 56
there is not a significant different to question 57

Table 5. Differences in average responses to “Summary“ questions

IV. METHODOLOGY and DETAILED RESULTS

The Noel-Levitz Priorities Survey of Online Learners is a national survey tested thoroughly to give reliable and valid questions. CWU administered the survey to all students enrolled in “WW” courses (100% online) during the spring 2013 quarter. The responses are self-selected and the response rate (11%) is small so unfortunately CWU cannot infer any population characteristics from the sample results. However, online student responses are so different than on-campus student responses in satisfaction and factors in deciding on a program that further research is merited.

Table 6. The follow screen shot from the Noel-Levitz web site shows the response rates over time to CWU’s administration of the PSOL.

Appendix 2 has an unofficial mock-up of the PSOL survey,

Demographic question #4 (the 61st question overall) asks students their current enrollment status. Of the 253 completed surveys 157 students selected “primarily online”, 90 students selected “primarily on-campus” and 6 students did no answer the question.

This analysis compares the responses on all other survey questions of the 157 students who select “Primarily online” tothe 90 students who selected “Primarily on-campus.”

Effect size was used to categorize the difference in means. Cohen’s d effect size was used. It is the difference in means divided by the pooled standard deviation and it may provide better estimates of difference when sample sizes are different.

Cohen gave a “rule-of-thumb” that an effect size of .2 to .3 might be “small,” between .3 and .8 “moderate,” and larger than .8 “large.” For this analysis an effect size (absolute value) of .3 to .8 is moderate and .8 or larger is large.

Table 7. Detailed Average Responses to “Importance” Questions

Table 8. Detailed Average Responses to “Satisfaction” Questions

Table 9. Detailed Average Responses to “Enrollment Factor” Questions

Table 10. Detailed Average Responses to Three “Summary” Questions

Table 11. Demographic Details

Table 11. Demographic Details - continued

IV. SUMMARY

Students who are enrolled online answer questions on Satisfaction, Enrollment Factors, Summary Ratings, and Demographics quite a bit different than students who are enrolled on-campus.

Students enrolled “primarily online” provide moremoderately to much higher (better) responses to Satisfaction questions than students enrolled on-campus. However, online student average responses were lower on some Satisfaction questions.

It is worrisome that students enrolled online provide, on average, a moderately lower response to “Rate your overall satisfaction with your online experience so far” than students enrolled on-campus.

The differences in survey responses to Satisfaction, Enrollment, Summary, and Demographic questions by online students substantiate CWU’s research and highlight the importance of further studies in this area.

APPENDIX 1

TERMS AND ABBREVATIONS

N-L PSOL / Noel-Levitz Priorities Survey of Online Learners
This national online survey has been designed to “Assess the satisfaction and priorities of students in distance learning and online programs.” See

Cohen’s d / A statistical measure of “effect size.” “Effect size is a method of quantifying the difference between two groups that has some advantages over the use of tests of statistical significance alone.” A pooled standard deviation is used as the denominator in this analysis. A pooled standard deviation is called for when the sample sizes of the variables studied are significantly different. See this conference presentation on effect size:
Note: Some studies have indicated that Effect Size calculations with ordinal data may over-state the difference in means. The PSOL survey uses a seven point Likert scale for many of its questions. A seven point scale tends to provide statistics closer to real numbers than a four or five point scale.
Likert scale / Most of the questions on CWU’s course evaluations use a five point Likert scale arranged in a table format (a visual analog scale). For this analysis a response of “Strongly agree” is assigned a value of 5. A response of “Strongly disagree” is assigned a value of 1. Whether individual Likert items can be considered as interval-level data, or whether they should be considered merely ordered-categorical data is the subject of disagreement. This analysis assumes that the responses can be considered as interval data, especially since the Likert scale items are arranged in a visual analog format.
n / N stands for the number of students who answered a particular question. The sample size is needed to compute Cohen’s d Effect Size.
On-campus / For this study “on-campus” students are: (a) physically enrolled and located at the Ellensburg campus or one of CWU’s centers and (b) taking a course that is taught 100% online.
Online / Courses taught 100% online via Blackboard or Canvas, usually asynchronously.
St Dev / An abbreviation for “standard deviation” a statistical measure of variability

APPENDIX 2

AN APPROXIMATE MOCK-UP OF THE NOEL-LEVITZ PRIORITIES SURVEY OF ONLINE LEARNERS

SURVEY INSTRUMENT

July 26, 2013Online vs. On-campus PSOL Responsespage 1 of 21