ITEAM Project

Feedback - EVS Champions meeting 15-11-12

Information from Champions was garnered in two ways; survey and unstructured interview.

Survey

Questions were given to the champions at the beginning of the meeting and completed before the meeting progressed. Words highlighted in red indicate challenges / issues which the Assessment team is in a position to address. We expect to feedback to the group at or before the next meeting.

1. What has gone well during the last 12 months? / School identifiers removed
The first EVS handouts in the school weren’t well planned; staff was informed about using EVS at the last minute and therefore didn’t have the training or planning time to use it effectively. The handout this year has been very successful. Programme tutors now have to request EVS and say how they are going to use it
Staff have now had training, there was continued good engagement between staff and students throughout the dept., colleagues have shown enthusiasm in integrating EVS, have learned to work around challenges for mac uses, EVS embedded in the majority of 1st and 2ndyr modules and a few 3rdyr modules. Most staff who used EVS for the first time last year are continuing with one or two exceptions, working well in large lecture rooms with few technical problems
Issuing handsets at registration, used successfully in induction talks, payment for new handsets very slick, no channel problems
Teachers have used to a good level, improved student experience, useful in large cohort assessment
Used as low threshold assessment, rewarding classroom engagement, 3 modules still using EVS summatively and reporting back positively
Increased use of EVS at all levels of UG programme, debate about how to best use this technology amongst colleagues, school has arranged open meeting about it, LTI has been an invaluable support
2.What challenges have you faced / are still facing?
Database is terrible. We cannot find out which students have not handed back their handsets, and we are going to lose money if we can’t recover them
Some areas were less willing to engage than others, believing that their discipline does not readily adapt to using EVS
Computer screens in law rooms not showing turning point bar, Intermittent problems in lecture rooms not capturing student responses
Quite a few hiccups in making it work, handset failure interrupting class
Not working robustly in all rooms E151
Having to continue to use ‘legacy’ software for mac, need standardised session approach to dealing with session file, have to change channels in each room, technical challenges with individual handsets (technicians report about 20 students bringing in faulty handsets), problems linking handset numbers to Studynet lists, using EVS summatively and still meeting flexibility agenda
Staff confidence in the technology and accurate recording of data, Duplication of handsets with joint hons students
Managing students with study needs agreement, deciding what % would be fair for students
3.What would you like to achieve over the next 6 months?
We have a good system in place within the school but we need to ensure good practice in the use of EVS and not over-use it.
Encouraging staff to use mac advanced features of EVS and do an evaluation of what the students think about EVS use.
More academics using EVS in HBS (local training), Promotion of resource information on LTI
Better understanding of the system, typical uses in eng modules, monitor usefulness and usability amongst staff, have a test to ensure handsets work at time of issuing
To discuss the possibility of integrating session files and combining with class list on Studynet, to set in place training /clearer instructions for issuing handsets, discussion about whether to use formatively or summatively
Sharing of good practice of the summative users
Produce school recommendations in partnership with students and LTI
4.What help do you need?
To take issues with the database to IH.
A roadshow giving staff the opportunity to see excellent practice in the different ways that EVS is used in a variety of disciplines using mac advanced features, Turning technologies to update software with mac office share, Support for evaluation approaches, working with Turning Point to make reports in excel more functional psychology techs are currently being centralised, might a central service be provided
Continued support, LTI encouragement to expand beyond the formative assessment
can’t think of anything
Leaflet / workshops on EVS
Early purchase of EVS and no time to issue to student

Interviews(recorded by the EVS Champions on video)

The Champions were asked to talk about EVS from a School perspective. They had limited time to prepare and no specific questions to answer. Videos were transcribed. Transcription was not wholly verbatim but key phrases and words were captured word for word. Key phrases and words were colour coded into positive and negative statements. Data was cross-checked by LGF and JV for accuracy and completeness.

Positive statement Negative statement ************** name blanked out

***********

Used in School for a couple of years, embedded in 1st and 2nd year modules and some 3rd year. Majority of staff use, a couple don’t. Problems last year with channels but much better this year. Still takes time in lectures making sure students are on the right channel. Academics finding it a lot of work as using it summatively creates angst… even when module information goes up there are lots of email enquiries from students. It’s a lot of work to merge participant / results lists and joint honsstudents have particular problems ending up with two handsets. One of the biggest issues is to do with flexibility … a benefit has been of improving attendance even with the small amount of marks attached (5%). Even on a Friday afternoon! Concerned though that this might conflict with the wider University agenda on flexible learning i.e. if we make students turn up for lecturers are we taking away choice/flexibility. Also described the difficulty of separating out formative and summative tests (referring to the session results).

****************

Most staff received EVS positively. Used for both summative and formative. There have been technological barriers for MAC users e.g. having to flip between PP…

Would like to see better integration between TP and Studynet.Makes the point that there is still a lot of work to do after the class by way of uploading files etc.Overall fairly positive experience but not without challenges. Nearly all of the staff use EVS regularly, mostly for formative. When used for summative they have an invigilator present. They have also developed set of guidelines for running tests and still use paper copies of test papers, so confidence in system is not total.

*****************

Large investment in handsets but not using as widely as hoped this year. Talked

about how they managed tests and raising awareness with students and staff about academic misconduct. Stressed the value of EVS as a tool for engagement.Used formatively for feedback.

*************

No major issues this year. Last year there were issues about channel conflicts which seem to have been resolved. Importing listswas also an issue. Better this year. Has had positive feedback since coming into the role at the beginning of term.

***************

School has used EVS for a number of years. Started with formative and moved to summative. Also use it for attendance (with small number of marks attached). Now extended use into post-graduate. Have also found benefits for international students enabling them to contribute in class. Also particularly good for large cohorts (200) to test understanding, gauge student opinion and do short tests. Helps to target support for students in need by identifying which students need help. Finds it a good tool for engagement.

****************

School has decided to let Programme Tutors decide whether they need them. They will work out at the end of the year what is needed for the next year. They have put processes in place to do this mapping. They have also trained up admin staff to issue the handsets and this year they waited until registration was over as ID cards don’t register straight away on the EVS database (based on last year’s experience), worked much better this year. Admin staffreally effective about handing them out, and Lucy (ed tech) was on call for queries. 718 issues this year. Only two returned as faulty (needed battery change). Most programmes have kept them on, whereas last year they didn’t get used. This year is much better. Heather stressed that there was flow chart / process, very clear system for everyone to use, also lots of resources have gone on school Studynet site to support. Majority using formatively,couple using it summatively (the more advanced users). Looking forward to next year they are hoping to provide some more training and again look at plan for how they are used. Also looking at return/recovery handset process.Key messages communicating between everyone admin, technologists and academic staff. Another thing is to get really clear processes embedded that anyone can follow. Also need to be careful not to hit EVS overload. EVS is tool and has to have purpose, some areas of practice where it works better than others. Lucy mentioned about staff and their confidence, supporting staff or first use (and second and third if necessary). Only major issue is the EVS database which doesn’t give the information they need. Heather added importance of clarity of role, ASL&T is about strategy etc. Loan system in the school (100) handsets, quite a few staff use these, used quite heavily for ad hoc or off campus work.

***********

Had problems last year, not so much with delivery in the classroom but with the operational side and reporting. School has taken a more cautionary approach this year. Most modules are using EVS formatively with three using summatively. Support that has been in place since September this year has been exemplary. Wishes same support was available last year. This year there has been support in class to ensure tests are operational and lots of support with retrieving data to make sure test results are delivered to the student in an effective and timely manner.

**********

This year has seen an increased use of EVS across the School. It is used across the undergraduate programme particularly at level 6. Increased use of EVS for summative has led to discussions about students with study needs agreements and also what would appropriate weighting of results be. With the support of the LTI theSchool has also set up an EVS testing station for students to come and test their handsets to make sure they are working prior to the test.

Negative statements / Positive statements
  • problems last year with channels
  • takes time
  • finding it a lot of work
  • creates angst…
  • lots of email enquiries from students
  • lot of work to merge lists
  • joint hons students have particular problems
  • technological barriers for Mac
  • company support for Mac
  • integration with Studynet
  • lot of work to do after the class
  • not without challenges
  • confidence in system is not total e.g. using paper copies
  • not using as widely as hoped
  • there were issues about channel conflicts last year
  • importing lists was also an issue
  • last year they didn’t get used
  • major issue is the EVS database
  • had problems last year
  • not all staff are keen to use
  • handsets handed out late
/
  • embedded in 1st and 2nd year modules and some 3rd year.
  • much better this year
  • benefit has been of improving attendance
  • received EVS positively
  • fairly positive experience
  • the value of EVS as a tool for engagement.
  • no major issues this year.
  • (channel conflict) seem to have been resolved.
  • better this year
  • had positive feedback (from colleagues)
  • found benefits for international students particularly good for large cohorts
  • use EVS to diagnose maths competency
  • good for testing knowledge
  • extended use to post-graduate
  • helps to target support for students
  • it is a good tool for engagement.
  • worked much better this year
  • admin staff really effective about handing handsets out
  • most programmes have kept them
  • this year is much better.
  • very clear system for everyone to use
  • lots of resources have gone on school Studynet site
  • provide some more training and again look at plan for how they are used.
  • looking at return/recovery handset process.
  • training identified and acted on
  • loan system in the school (100) handsets, quite a few staff use these
  • LTI support has been exemplary
  • lots of support
  • well used by the first year tutors
  • results are delivered to the student in an effective and timely manner.
  • has seen an increased use of EVS across
  • increased use of EVS for summative has led to discussions about SNA
  • With the support of the LTI the School has also set up an EVS testing station
  • formative for feedback
  • most modules use formatively
  • majority use formatively
  • support in class to ensure tests are operational
  • evidence of summative use with low stakes
  • couple using it summatively
  • evidence of summative use with drop quizzes and tutorials

1

The table below shows the problems and challenges identified and what actions will be taken to address.

Problems identified in survey / Actions / Date by which to take initial actions / Contact/lead
  1. EVS database
  1. Handsets handed out late
  1. Joint Hons students have particular problems
  1. Technological barriers for Mac &company support for Mac
  1. Lot of work to do after the class, finding it a lot of work, takes time,lot of work to merge lists, linking handsets to students
  1. creates angst, lots of email enquiries from students
  1. Integration with Studynet
  1. not using as widely as hoped, not all staff are keen to use
  1. EVS handset maintenance service be provided centrally
  2. managing students with disability
  1. room issues
/
  1. Identify user needs and take to LTDU for discussion re: development
  1. Start purchase process planning early in 2013
  1. Meet with Joint Hons PT to discuss and then decide way forward (initial meeting Nov 22nd 2012)
  1. Arrange meeting between Reivo and Mac users. ? Set up Mac users forum.
  1. Continued staff support and training, sharing good practice and top tips
  1. Provide student support via LRC, webpage, at point of use
  1. LTDU in development, clarify deadline at next meeting of elearning group
  1. Support dissemination of EEVS report, share good practice
  1. Discuss with IH
  1. Meet with Disability Services to discuss best practice advice
  1. Follow up with Help Desk
/
  1. 1st December 2012
  1. Meeting with IH and supplier on January 8th 2013
  1. January 31st2013 (before CATs students start in
Semester B)
  1. January 2013
  1. Ongoing
  1. Ongoing
  1. January 2013
  1. December 2012 onward (next EVS workshops)
  1. November 2012
  1. December 2012 
  1. November 2012

1