Ilearn Student Experience Survey 2014

iLearn Evaluation Report

iLearn Student experience Survey 2014

Produced by Jayde Cahir (Chief Investigator), Helen Carter (Co-Investigator), Cathy Rytmeister (Statistical analysis and advice), Lucy Arthur (Faculty iLearn Support Coordination) and Rebecca Ritchie (Central iLearn Support Coordination).

May 2015

Table of Contents

List of Figures

List of Tables

Results and Discussion

Demographic Details

Technologies used to access iLearn: Student Survey Results

Student Satisfaction with iLearn

Student Satisfaction with iLearn as a support for learning and teaching

Student Experience with iLearn

Statistics on the use of iLearn Functions: data from the iLearn system

iLearn Functions to organise study

iLearn Functions to engage with unit content

iLearn Functions for engagement with learning activities

Student Satisfaction with Services and Support

Student Satisfaction with the Online Unit Guide (iTeach)

Appendices

Appendix 1: iLearn Student Experience Survey Questions

Appendix 2: Survey Advertisements

Appendix 3: Improvements to the iLearn platform

Appendix 4: Statistical Analysis of Student Respondents Demographics

Appendix 5: Statistical Analysis of Technologies Students used to access iLearn

Appendix 6: Statistical Analysis of Student Satisfaction with iLearn

Appendix 7: Statistical Analysis of Student Experience with iLearn

Appendix 8: Statistical Analysis of Student Experience with Online Unit Guide

List of Figures

Figure 1: The way iLearn is used enhances my learning experience

Figure 2: Student recommendation of Macquarie University's use of iLearn

Figure 3: Student Satisfaction with iLearn for access to unit content

Figure 4: Student satisfaction with iLearn as a support for organising study

Figure 5: Student satisfaction with iLearn as a support for interaction and communication with fellow students

Figure 6: Student satisfaction with iLearn as a support for interaction and communication with teaching staff

Figure 7: Student satisfaction with iLearn as a support for engagement in learning activities

Figure 8: Students use of iLearn Functions

Figure 9: Student satisfaction with OneHelp for technical issues with iLearn

Figure 10: Student satisfaction with locating 'self-help' resources

Figure 11: Satisfaction with student 'self-help' resources

List of Tables

Table 1: The iLearn Student Experience Survey Demographic Information for 2012 - 2014

Table 2: Student ownership and access to technology and Internet

Table 3: Technologies used to access iLearn according to the 2014 student survey

Table 4: The percentage of units that used the various iLearn functions

Table 5: Student experience with iLearn Functions in Session 2, 2014

Table 6: Students levels of agreement with the usefulness of iLearn Functions

Table 7: Student satisfaction with the Online Unit Guide

iLearn Student Experience Survey

The 2014 iLearn Student Experience Survey is part of a wider Learning and Teaching Centre (LTC) Quality Enhancement strategy for Macquarie University’s Learning Management System (LMS) and learning technologies more generally. In 2014, iLearn had moved into the second year of its operational phase thus the focus of the 2014 survey was to evaluate student satisfaction with:

the functionality of iLearn from a teaching practice and technical perspective;
the University’s services and support for iLearn; and
to document trends in use to inform future needs in terms of functionality and support.

This focus aligns with the findings and recommendations of the iLearn Implementation 2012: Evaluation Report and the iLearn Evaluation Report 2013 in order to capture the needs and expectations of students and to continue the ongoing quality enhancement cycle.

Results and Discussion

This evaluation study examinedthe second year of the operationalisation phrase of the iLearn platform. Data for this evaluation study was compiled through the iLearn Student Experience Survey (Appendix 1). Invitations to complete the iLearn Student Experience Survey weresent over a three-week period in Session 2, 2014, between the 8th and 29th October. Thepurpose of this survey was to:

monitor engagement with iLearn;
evaluate satisfaction with services and support;and
map changes in learning and teaching across the University.

The surveys were sent to a random sample of 25% of the student population using iLearn, excluding students in the Applied Finance Centre, Bachelor of Business Administration (BBA) in Hong Kong, Exchange Students studying in the Northern Hemisphere and International College of Management Sydney (ICMS) students. The student sample also excluded MGSM, at their request, as they were running their own surveys. This approach to selecting the sample aligned with the creation of student sample groups in 2012 and 2013. As in previous years, the survey was delivered through University evaluation system, Teaching Evaluation for Development Service (TEDS), using EvaSys.

Demographic Details

In Session 2, 2014 invitations were sent to 9,380 and a total of 481 responses were received, which represented 5.6% of the sample.

iLearn Student Experience Survey / Session 2, 2012
482 respondents / Session 2, 2013
902 respondents / Session 2, 2014
481 respondents
Faculty of Arts / 31.4% / 28.6% / 29.9%
Faculty of Science / 18.4% / 17.4% / 16.1%
Faculty of Human Science / 26.6% / 26.4% / 32.8%
Faculty of Business & Economics / 23.4% / 27.6% / 21.3%
Undergraduate / 71% / 74.1% / 80%
Postgraduate / 29% / 25.9% / 20%
Internal / 68% / 77.6% / 79%
External / 26.8% / 21.5% / 25.8%
OUA / 2.9% / 0.3% / 0.4%
International / 16.4% / 15.5% / 11.4%
First Year / 28.5% / 36.7% / 37.2%
Second Year & Beyond / 71.5% / 63.3% / 62.8%

Table 1: The iLearn Student Experience Survey Demographic Information for 2012 - 2014

Further statistical analysis was conducted to ascertain the distribution of student responses from:undergraduate / postgraduate, internal/external, and first year/second year and beyond within each Faculty(see Appendix 4). The results showed that:

Undergraduate students are under-represented (and postgraduate students over-represented) amongst Faculty of Human Sciences respondents;
Postgraduate students are under-represented (and undergraduate students over-represented) amongst Faculty of Science respondents;
Internal (only) students are under-represented (and external only students over-represented) amongst Faculty of Arts respondents;
External (only) students are under-represented (and internal only students over-represented) amongst Faculty of Business and Economics respondents.
First year students are over-represented (and later year students under-represented) amongst Faculty of Business and Economics respondents;
First year students are possibly slightly over-represented (and later year students slightly under-represented) amongst Faculty of Arts respondents; and
Second and later year students are over-represented (and first year students under-represented) amongst Faculty of Human Sciences and Faculty of Science respondents.

Technologies used to access iLearn: Student Survey Results

In the first instance, we asked what technologies the students own or have access to. The rational was that students are not going to access iLearn via Smartphones or iPads etc unless they have access to this technology. The tick-box question was:“I own or have access to the following devices (please indicate ownership or level of access - note that we are assuming that ownership includes regular and reliable access)”, as this would also impact on what technologies they use to access iLearn. The tick-box question was: “My off-campus access to broadband internet is: Regular and reliable; Irregular and/or unreliable; Minimal”. The surveyed students’ responses to these questions are listed below:

Technologies students have access to / owner / Regular, reliable access / Some, unreliable access / Little or no access
Smartphone (e.g. iPhone, Android, Blackberry) (n=477) / 89.9% / 4.0% / 1.3% / 4.8%
Laptop computer (n=477) / 87.8% / 5.0% / 2.9% / 4.2%
Tablet (e.g. iPad, Android or similar) (n=470) / 49.1% / 7.9% / 8.5% / 34.5%
Desktop computer (n=465) / 34.2% / 23.4% / 15.1% / 27.3%
Portable e-book reader (e.g. Kindle) (n=453) / 15.7% / 3.1% / 5.7% / 75.5%
Student Internet Access / Regular and reliable / Irregular and/or unreliable / Minimal
My off-campus access to broadband internet is (n=475) / 91.4% / 7.6% / 1.1%

Table 2: Student ownership and access to technology and Internet

The results show that a high percentage of the surveyed students (89.9%) own a SmartPhone and laptop computer (87.7%), however, these are not necessarily the devices they use to access iLearn. In order to ascertain the technologies used to access iLearn, the survey respondents were asked to:“Please indicate the frequency with which you use the following devices to access iLearn”: University-provided computer on campus (e.g. office, lab, Library); Desktop or laptop computer at other location/s (e.g. home); My own laptop on campus using OneNet (MQ wireless network); Smartphone (iPhone, Android, Blackberry) at MQ using OneNet; Smartphone (iPhone, Android, Blackberry) at other location/s; Tablet (e.g. iPad, Android or similar) at MQ using OneNet; Tablet (e.g. iPad, Android or similar) at other location/s; Other device/network combination - please specify.

Technologies used to access iLearn / At least once a day / A few times a week / A few times a month / A few times a semester / Never or
rarely
Desktop or laptop computer at other locations (n = 480) / 65.2% / 25.4% / 2.9% / 1.9% / 4.6%
Smartphone using another network provider (n = 473) / 24.9% / 20.5% / 10.6% / 9.3% / 34.7%
My own laptop using OneNet (n = 478) / 23.8% / 27.8% / 11.7% / 12.1% / 24.5%
Smartphone at MQ using OneNet (n = 475) / 22.5% / 29.7% / 12.0% / 11.4% / 24.4%
Tablet at MQ using OneNet (n = 473) / 10.4% / 15.2% / 7.2% / 6.3% / 60.9%
Tablet at other locations (n = 473) / 9.3% / 9.3% / 5.7% / 6.4% / 69.3%
University provided computer on campus (n = 473) / 3.8% / 15.9% / 11.2% / 20.9% / 48.2%
Other device/network combination (n = 437) / 3.0% / 2.1% / 0.9% / 1.1% / 92.9%

Table 3: Technologies used to access iLearn according to the 2014 student survey

The table above relates to the use of various technologies to access iLearn, as it is essential to monitor if the system is meeting the needs of new technological devices and if the network is able to support the system and devices in all circumstances. The student responses to the last three surveys have shown a consistent approach to which technologies are relied upon to access iLearn with ‘Desktop or laptop computer at other locations’ being the most popular choice. The use of Smartphones using OneNet has slowly increased. In Session 2, 2012 a total of 9.1% of the students surveyed indicated that they access iLearn via their Smartphone ‘at least once a day’. In Session 2, 2013 15.3% of the students surveyed said they use their Smartphone ‘at least once a day’. In Session 2, 2014 a total of 22.5% of the students surveyed indicated that they are accessing iLearn via their Smartphones ‘at least once a day’. This increase would not only be due to the further proliferation of mobile phones but also the introduction of a mobile friendly iLearn site. In response to ‘other device or network combination’ a total of 92.9% of students surveyed selected ‘never or rarely’. There were fourteen comments in the ‘please specify’ section, for example, Smartwatch and internet networks outside of campus were listed.

According to statistical analysis of the survey results there were some significant variations between frequency of use of different devices and networks to access iLearn. Students in the Faculties of Science and Business and Economics indicate frequent use of ‘University computers on campus’ in greater proportion than students in the other faculties; a greater proportion of Faculty of Arts studentsand Faculty of Human Science students use them "never or rarely". This seems to reflect differences in the extent of on-campus computer laboratory provision in these faculties. Students from the Faculty of Business and Economics are more likely to indicate frequent use of:'Own laptop at MQ using OneNet', 'Smartphone at MQ using OneNet'; 'Smartphone using another network provider'; 'Tablet at MQ using OneNet'; and 'Tablet using another network provider' than students from other faculties. It is important to highlight that there were no statistical differences in student ownership of Smartphones or Tablets. Students in the Faculty of Arts and the Faculty of Science were more likely to indicate infrequent or no use of 'Own laptop at MQ using OneNet', and again, there were no statistical differences in student ownership of laptops. Statistical analysis of the survey results between undergraduate/postgraduate, Internal / External and year of enrolment revealed further significant variations between frequency of use of different devices and networks to access iLearn. Undergraduate students are more likely to access iLearn using 'Own laptop at MQ using OneNet' and 'Smartphone at MQ using OneNet' than postgraduate students. Postgraduate students are more likely to access iLearn ‘at least once a day’ via a 'Tablet using another network provider' than undergraduate students. Internal students are more likely to access iLearn using all devices other than ‘Desktop or laptop computer at other locations’ than external students. Furthermore, external students are more likely to own a laptop than internal students (see Appendix 5).

At the end of the survey were two open-ended questions:

“What aspects of iLearn do you find most valuable for supporting your learning?” (n = 377)
“What improvements would you like to see in iLearn?” (n = 293)

In response to “What aspects of iLearn do you find most valuable for supporting your learning?” There was only one comment from a student saying they liked how iLearn was mobile friendly. In response to “What improvements would you like to see in iLearn?” There were a total of eight student responses, which referred to better smartphone and tablet integration, ease of use on the iPhone including improving access to grades, quizzes and calendar.

Student Satisfaction with iLearn

In order to ascertain student satisfaction with iLearn, students were asked to indicate their level of agreement with each of the following statements about your overall experience with iLearn:

The way iLearn is used enhances my learning experience; and
I would recommend Macquarie University’s use of iLearn as an example of good practice to other universities.

Figure 1: The way iLearn is used enhances my learning experience

Student satisfaction with the way iLearn enhances their learning experience, according to the students surveyed, shows that the level of disagreement with ‘the way iLearn is used enhances my learning experience’ has almost tripled since 2013, which requires further investigation. In Session 2, 2012, 71.3% of students agreed that the way iLearn is used enhances my learning experience, and in Session 2, 2013, 79.5% of students agreed with this statement and in Session 2, 2014 a total of 71.5% of students surveyed agreed with this statement.

Figure 2: Student recommendation of Macquarie University's use of iLearn

Students’ responses to ‘I would recommend Macquarie University’s use of iLearn as good practice’ reveal minor variations from 2012-2013 results. Statistical analysis revealed that postgraduate students are less likely to recommend Macquarie University’s use of iLearn as good practice than undergraduate students (p<0.05) and furthermore, first year students are more likely to recommend Macquarie University’s use of iLearn as good practice than students in their second year and beyond (p<0.01). (seeAppendix 6).

Student Satisfaction with iLearn as a support for learning and teaching

In order to ascertain student satisfaction with iLearn as a support for learning and teaching, the scaled questions focused on five areas. Students were asked to: “Please indicate your level of agreement with each of the following statements about your overall experience with iLearn”.

I am satisfied with the way iLearn enables access to unit information and content.
I am satisfied with iLearn as a support for organising my study.
I am satisfied with the way iLearn supports interaction and communication with my fellow students.
I am satisfied with the way iLearn supports interaction and communication with teaching staff.
I am satisfied with the way iLearn supports my engagement with learning activities.

Student satisfaction with iLearn as a support for learning and teaching, according to the students surveyed in 2012-2014, has shown some variation to the 2013 survey results, due to a rise in the neutral/mixed feelings and disagreement. The only repetitive comments in the open-ended responses that may provide some insight into this were the student complaints regarding the inconsistent use of iLearn, otherwise a majority of the student comments were positive. Again, this requires further investigation.

Figure 3: Student Satisfaction with iLearn for access to unit content

Student responses to this scaled question indicatethe levels of disagreement have tripled since 2013. Statistical analysis revealed that first year students are satisfied to a greater extent with the way iLearn enables access to unit information and content than students in their second year and beyond(p<0.025) (see Appendix 6).

Figure 4: Student satisfaction with iLearn as a support for organising study

Student responses to this scaled question indicatethe levels of disagreement have almost tripled since 2013.

Figure 5: Student satisfaction with iLearn as a support for interaction and communication with fellow students

Student responses to this scaled question indicates some minor variations, considering in Session 2, 2013, 71.4% of students surveyed agreed that they are satisfied with iLearn as a support for interaction and communication with fellow students, and in Session 2, 2014, 64.8% of students agreed with this statement.

Figure 6: Student satisfaction with iLearn as a support for interaction and communication with teaching staff

Student responses to this scaled question indicates, again, some minor variations, considering in Session 2, 2013, 74.2% of students surveyed agreed that they are satisfied with iLearn as a support for interaction and communication with teaching staff, and in Session 2, 2014, 67.2% of students agreed with this statement. Furthermore, statistical analysis revealed that students in Faculty of Human Sciences are more satisfied with the way iLearn supports interaction and communication with teaching staff than students in other Faculties (p<0.025)(see Appendix 6).

Figure 7: Student satisfaction with iLearn as a support for engagement in learning activities

Student responses to this scaled question indicate the levels of disagreement have increased since 2013. However, students in Faculty of Human Sciences are more satisfied with the way iLearn supports their engagement with learning activities than students in other Faculties (p<0.05). Statistical analysis also revealed that first year students are satisfied to a greater extent with the way iLearn supports their engagement with learning activities than students in their second year and beyond(p<0.05) (see Appendix 6).

Student Experience with iLearn

This section focuses on student engagement with iLearn using four forms of data sourced from: the iLearn system; the scaled responses; open ended responses; and statistical analysis by Faculty based on the scaled responses in the survey.

Statistics on the use of iLearn Functions: data from the iLearn system