Learning Management System Insights from the 2016 Faculty Technology Survey

Overview

UNM conducts annual surveys of students and faculty to gauge technology trends/needs and satisfaction with technology-related services provided by various units at UNM. This document examines the responses to those questions to inform a decision about whether to proceed with an upgrade or replacement of UNM Learn.

Survey Specifics

The survey was conducted online during the period March 11 through April 8. 1828 full-time faculty from main and branch campuses were asked to complete the survey. 387 faculty responded to the survey (21% response rate) with 328 completing all questions asked. The 2016 response rate is typical for annual faculty technology surveys. Faculty email invites included web links specific to each faculty member which will enable survey responses to be cross-referenced to Banner-related data such as faculty rank, academic unit, and hire date.

The 2016 faculty survey included 27 questions in the following groups:

·  10 questions - instructional use of technology other than LMSs and related UNM services

·  5 questions - satisfaction with IT services and use of cloud, email, and calendaring services

·  12 questions - experience with LMSs including both UNM Learn and other products/services

LMS-Related Questions and Responses

For purposes of this survey, an LMS was operationally defined as follows:

A learning management system (LMS) is a web application that provides an instructor with a way to create and deliver educational content/experiences, support and monitor student participation, post and receive student assignments.

LMS-Related Questions

The 12 LMS-related questions are summarized here (numbered 16-27 as they were in the survey):

16.  I use an LMS for class sizes of _____ students. (0-9, 10-24, 25-49, 50-99, 100-199, 200+)

17.  Please indicate your level of agreement with the following statements about the LMS: The LMS

  1. is critical to my teaching
  2. is very useful as a tool to enhance my teaching
  3. is very useful as a tool to enhance student learning
  4. creates efficiencies for my instruction

18.  How many years of experience do you have using LMS technology? (0-1, 2, 3, 4, 5-10, 11+)

19.  Do you currently use UNM Learn? (Yes, No)

Q19 NO Branch

20.  What factors influence your decision not to use UNM Learn (Check all that apply)?

  1. instructor ease of use
  2. student ease of use
  3. ease of maintaining personal web sites
  4. training availability
  5. support quality
  6. LMS tool usefulness
  7. LMS features
  8. LMS flexibility
  9. LME reliability/stability
  10. class size
  11. lack of 3rd party content integration
  12. prefer a different system

Q19 YES Branch

21.  Describe how you use an LMS to support your instructional activities (Select all that apply).

  1. distribute documents and learning materials to students
  2. distribute video to students
  3. web conferencing
  4. gradebook
  5. quiz//exam delivery
  6. online assignment submission
  7. graded online discussions
  8. communication between instructor and student
  9. student-to-student messaging
  10. group collaboration
  11. learning analytics
  12. fully online class delivery
  13. hybrid class delivery
  14. flipped classroom

22.  Please indicate your satisfaction with the following aspects of UNM Learn:

  1. ease of use in general
  2. ease of use from a mobile device
  3. supports meaningful interaction with students
  4. system availability
  5. system response time
  6. initial use training
  7. ongoing training and professional development
  8. posting content
  9. managing assignments
  10. monitoring/management enrollment
  11. entering grades and other progress information
  12. reliable assignment receipt
  13. integrating 3rd party content
  14. integration with other UNM systems
  15. overall satisfaction

23.  Do you have experience with an LMS that you prefer over UNM Learn? (Yes, No)

Q23 YES Branch

24.  What is your preferred LMS? (Angel, Canvas, Desire2Learn/Brightspace, Moodle, WebCT, Faculty/locally-developed, Other)

25.  Why is the above your preferred LMS (Check all that apply)?

  1. instructor ease of use
  2. student ease of use
  3. usefulness of LMS tools
  4. features
  5. flexibility
  6. reliability/stability
  7. 3rd-party content integration

26.  If my preferred LMS was available for use at UNM, I would invest time in moving my content to that LMS from UNM Learn. (Yes, No)

27.  Please provide additional comments comparing your experience with the other LMS to UNM Learn.

Survey Respondents

A few observations about respondent characteristics can be gleaned from the survey responses:

·  43% claim 5 or more years of LMS experience (Q18). Thus, it appears that survey respondents are more likely than non-respondents to be users of technology in general and the LMS in particular. If that characterization is true, the good news is that survey respondents are probably well-informed. The bad news is that they probably don’t represent the faculty as a whole.

·  287 respondents (approximately 77%) currently use UNM Learn. This is a fairly large sample from which to gather responses to the detailed UNM Learn questions. However, there’s no way to know how (un)representative of the faculty as a whole are those respondents.

·  53 respondents (approximately 16%) have experience with an LMS that they prefer to UNM Learn. The sample is large enough to gather some useful information about comparisons between UNM Learn and other LMSs. But it isn’t large enough to have high confidence in the accuracy of those comparisons nor whether they would reflect the attitudes of the entire faculty should UNM switch to a different LMS.

Faculty Use of UNM Learn

Q21 asked UNM Learn users how (which features) they use. The top responses in decreasing order of frequency are:

·  Distributing documents and learning materials (95%)

·  Gradebook (74%)

·  Online assignment/paper submission (63%)

·  Quizzes/exams (52%)

·  Class-related communication between the instructor and students (50%)

The least frequent responses were web conferencing (8%) and student-to-student messaging (14%). Use of group collaboration, graded discussions, and analytics were also relatively low (17-22%).

Note that Q21 was only asked of faculty members that currently use UNM Learn. Thus, it’s an open question whether past or present users of other LMSs use those LMS in ways similar to the way other faculty use UNM Learn.

Responses concerning class delivery modality (fully online, hybrid, and flipped) showed no clear pattern. Fully online and hybrid had a nearly equal number of responses (15-17%). Flipped classroom was about half as many responses (9%) but that isn’t surprising given the relative newness of that class delivery modality. By omission, we might be able to assume that the respondents that didn’t choose any of the other modalities use UNM learn to teach traditional face-to-face classes (up to 59%).

Q22 measured faculty satisfaction with UNM Learn feature-by-feature and overall. Overall, approximately 59% of respondents that currently use UNM Learn were satisfied or very satisfied, 21% were neutral, and 19% were dissatisfied or very dissatisfied. 53 respondents answered yes to Q23 (use another LMS they prefer over UNM Learn). If those respondents are assumed to be less-than-satisfied users of UNM Learn then the overall satisfaction for UNM Learn is somewhere between 49% and 59%. Exactly where in this range is the correct value is unclear since there’s no way to know how many respondents that answered yes to Q23 also answered Q22 (i.e., they currently use UNM Learn and currently use or previously used another LMS).

Faculty Non-Use of UNM Learn

88 respondents answered no to Q19 indicating that they do not currently use UNM Learn. Q20 asked why and provided a set of specific responses and an open-ended response option (choose all that apply). Of the specific responses, the most common in descending order of frequency were:

·  Ease of use for faculty (18.6%)

·  Ease of use for students (10%)

·  Usefulness (lack thereof) of LMS tools (10%)

·  Ease of maintaining personal websites (9%)

·  Availability of training (9%)

·  Flexibility of the system (9%)

Open-ended responses underscored the above responses with emphasis on poor ease-of-use, system clunkiness and inflexibility, and the utility of other automated tools such as Dropbox, Google Classroom, and traditional web sites for supporting instruction.

Comparing the latter point with the responses to Q21 and assuming that users of both UNM Learn and other LMS use them in similar ways, it’s apparent that many faculty prefer other tools to support the relatively small number of LMS functions that they employ (e.g., document distribution, gradebook, and communication with students). A few of the open-ended responses to this question and to Q27 also noted that utility of using tools such as Drobox that students already know and/or will use after graduation. Ease-of-use also factors into the reasons to use other tools and LMSs as shown by the large number of respondents that point out ease-of-use as either a reason not to use UNM Learn or to be less-than-satisfied with UNM Learn.

Faculty Experience with Other LMSs

49 respondents described other LMSs with which they have experience. The top responses were:

·  Canvas (16)

·  Other (14)

Specific descriptions accompanying the “other” responses were all over the map. Other specific alternatives such as Moodle and Desire2Learn had 5 or fewer responses. The sample sizes for the various UNM learn alternatives are all too small to draw definitive comparisons to Blackboard or among the alternatives.

Q25 asked why respondents preferred another LMS to UNM Learn. Similar to responses to earlier questions, ease-of-use (lack thereof) was the most frequently cited reason. Other frequent responses included LMS tool usefulness, flexibility, and reliability/stability.

Open-ended responses to Q27 underscored the pre-supplied response alternatives with many anecdotes describing UNM Learn’s poor ease-of-use and other shortcomings and praising the relative strengths of a chosen alternative.

Finally Q26 asked if current users of other LMSs would invest the time to transfer content from UNM Learn to their preferred LM. Of 46 respondents, 37 (80%) said yes.

Putting It All Together

The picture painted by the survey responses is complex. It’s provides neither a strong affirmation of UNM Learn nor a strong mandate for change. As such, the survey can be mined for pieces of information that will support one direction or another. But defending either choice will require further effort to fill in the missing pieces. As such, analysis and discussion of the survey results is only a starting point for deciding whether/how to improve, upgrade, or replace UNM Learn.

Some key takeaways from the survey include:

·  UNM Learn is underutilized

·  Ease-of-use, complexity, and training are problem areas

UNM Learn Under-Utilization

Some UNM faculty don’t use any LMS. Some use an LMS other than UNM Learn. Some faculty cobble together LMS-like capabilities using tools such as web sites and GoogleDocs. Among faculty that do use UNM Learn, many use only a handful of its functions.

Whether underutilization is a “problem” is a matter of perspective – faculty, student, or institutional. Since faculty aren’t mandated to use UNM Learn and since their opinions of its utility vary widely, it’s little surprise that there’s significant underutilization. Faculty teaching online and those teaching large classes have obvious motivations to use the LMS though some of those faculty choose to use tools other than UNM Learn. At the other end of the spectrum, some faculty simply see no need for UNM Learn. For faculty teaching classes such as those with low enrollments, infrequent assignments, and minimal document distribution, relatively small benefits (e.g., 7-24 access to documents) accrue only to the students with the costs of learning to use UNM Learn and maintaining course-related content being borne by faculty and support staff. Given the number of faculty avoiding UNM Learn or complaining about ease-of-use, it’s reasonably clear that a significant number of faculty members don’t value the perceived benefits to themselves or student more highly than the costs that they themselves incur.

The faculty survey doesn’t illuminate the student perspective though a separate student survey with LMS-related questions to be completed in May will partly fill that gap. There are some obvious benefits to students from a well-used LMS including accessibility of documents and learning materials, access to grades, online assignment submission and feedback, collaboration with instructors and peers, and more efficient learning. But these benefits are only realized if instructors make good and full use of its features. As the survey results show, many faculty use the LMS as a combination of a static web site, assignment drop box, and Internet-accessible grade book. Features such as calendar integration, online quizzes/exams, modularized and programmed learning, and learning and web site analytics are seldom used. Yet it is these features that LMS-related research points to as having the most significant impact on learning outcomes.

Another student perspective is that of tool overload. To the extent that faculty use different LMSs, varying tool sets in place of an LMS, and use UNM learn in widely different ways, students face a relearning process with each course they take. They must know how to use multiple LMSs/tools and must adjust to the myriad ways in which LMS/tool use varies from class to class. Some students negotiate this maze easily but others struggle. It seems obvious that there must be some negative impact on learning outcomes and grades for some students – though there is no research to back up that assumption.

LMS underutilization has multiple impacts on UNM. These include:

·  Inability to tightly and uniformly integrate other systems with the LMS

·  Loss of learning outcomes that might or could be achieved through better LMS use

·  Complicating support for faculty and students

Modern organizations are expected to use technology to seamlessly connect various organizational functions and systems. At its best, this integration improves quality and efficiency and reduces costs. But achieving those results requires the right technology and widespread or universal use. Lacking those features, the benefits are poorly achieved. Early alerts through LoboAchieve are a case in point. They have yet to deliver their promised benefits due to underuse and poor integration with UNM Learn. Achieving UNM-wide learning improvement through learning analytics is another case in point. Too few faculty use UNM learn in a standardized enough way to provide sufficient learning data for mining, analysis, and downstream improvement efforts.

Ease-Of-Use, Training, and Support – A Gordian Knot?

Many current UNM Learn users cite ease-of-use and training and reasons not to use UNM Learn or as barriers to its effective use. At the same time, support organizations report low utilization of available support and training services. This apparent paradox has left both sides unhappy and struggling for a solution.