HE STEM Project:

e-Assessment in Maths & Stats

Report on the First Workshop at University of Nottingham.

February 22nd 2011

The first day of this workshop/conference for all collaborators involved demonstrating current e-assessment implementations and strategies across the HE sector. See Appendix for the timetable.

The talks were split into four sections:

  • Commercial Vendors
  • Local Statistical e-assessment implementations
  • Tried and Tested Implementations in HE
  • New developments and open-source projects

Presentations can be found in the Resources section of the e-Assessment Association

1.Commercial Vendors.

There were three talks in this section on MyMathLab, I-assess, Maple TA.

WileyPlus is discussed under another section.

MyMathLab: aligned with Pearson text books. For example Liz Phillips at UEA reported on her use of the system with the Croft and Davison mathematics textbook for the Science Foundation Year, primarily aimed at mature students.This has been running since 2008 with successful results.

I-assess: (EDI) used heavily at NewcastleUniversityMaths & StatsSchool in first and second years as well as the Foundation Year. It has now been further developed to allow plug-ins which means that CAS such as Maxima and other open source third party software can now be used. Item analysis is now a major feature.

MapleTA (Adept Scientific): uses the Maple CAS and so can ask sophisticated questions. Has many features suitable for Maths & Stats at HE and is the only commercial system designed primarily for Maths at HE. It is used in 12 universities in the UK. There are no plans for looking at interoperability between different systems at present. Most implementation issues reported have been solved in the latest versions.

2.Local Statistical e-assessment implementations

The next two talks were on interesting local developments of statistical packages.

John Horton, University of Nottingham, demonstrated the use of the XerteToolkit to develop an on-lineserver-side system aimed at refreshing statistical knowledge and applications for postgraduate students prior to entry. There are 25 units covering a wide range of statistical techniques with an on-line e-assessment at the end of each unit. Users can practice these assessments and when ready can opt to record their next attempt as a summative assessment.

Karen Ayres of Reading University,with other members of staff, has developed an electronically delivered and marked assignments for students taking statistical modules. The students answer the assignment by using a Word form with entries to be completed, these forms are collected using Blackboard and the answers are later processed into text files and then an Excel file with macros is used to mark and to look for common errors with possible automatic feedback. Allowances are made if these incorrect entries are correctly used in subsequent answers. Also there is some hand-marking looking for allowable errors. After merging the outcomes of this process and eventually mail merging into Word for a list of results there is also available an annotated copy of each student’s submission.

3.Tried and Tested Implementations in HE

E-assessment in various forms has a long history of use in maths & stats and the next four talks demonstrate implementations which are presently firmly embedded in HE institutions.

Douglas Quinney: WileyPlus at KeeleUniversity

Used at Keele and Portsmouth Universities. This implementation has a strong formative element with 3 or 4 assignments given to 250 maths students per week. The students can practice as much as they like, subject to deadlines, and there is immediate feedback. Students can attach their working to their answers if required. Assignments are built from the large database of available questions built into the system; at present there is no authoring facility for local use but this is to be part of the next upgradeas is the querying of the gradebook database. There are benefits for staff as marking is done automatically; students like the system and are happy to use it but there needs to be careful evaluation before stating categorically that there has been pedagogic benefits for the group of students concerned.

Michael McCabe: WileyPlus and MapleTA at PortsmouthUniversity

Successful implementation of WileyPlus at Portsmouth. Used MapleTA as well in the past and latest versions of MapleTA are easy to install and there has been no problems with using the system. However there is to be a major review of teaching and learning at Portsmouth which may impact on the choice of e-assessment systems.

Referenced the JISC report on e-assessment as having the criteria that the present project could adopt. Also the use of commercial systems is to be encouraged, given the infrastructure and support available as well as in the case of WileyPlus a large database of examples to use and share with the community. Open source solutions in general suffer lack of support and there needs to be a solid community of users established.

General maxim: “Open source, no! Open resource OK!”

Martin Greenhow: Mathletics and QuestionMark Perception at BrunelUniversity.

Using Perception 5 to generate randomised MCQ questions suitable for A-level content, C1 to C4, S1 and some of S2 as well as most of D1. Mainly used for service courses at Brunel and not for first year maths degree courses. There is evidence from Brunel that MCQ questions build confidence in students using them and can affect averages on assessment by an extra 5%.

These questions are all available via Martin’s webpage. They feature

  • SVG graphics enabling questions with a strong geometric or diagrammatic content which is dynamically recreated with randomisation of the questions.
  • Very importantly, he has developed malrules designed to trap common student errors with good quality feedback.

Martin has developedmaths e-assessment for economics students in the METAL joint project with several other universities. Another current project is to produce statistics e-assessment for a range of disciplines including economics and geography.

Helen Ashton: SCHOLAR and CALM at Heriot-Watt

Scholar is widely used in the UK, especially in Scotland. It is used in schools and for Highers in Scotland. E-assessments have been produced for a wide range of disciplines as well as mathematics and so there is a very large database of questions. This systematic use has not as yet extended to university level maths or statistics.

The question formats and some other functionality, including steps, simplification and randomisation, are derived from the CALM project based and used in mathematics degree courses at Heriot-Watt since 1985. Since then there has been major developments by the development team at Heriot-Watt in interactive graphics and especially in “follow-through” questions.

4. New developments and open-source projects

The following three projects are aimed at sharing e-assessment solutions in maths & stats across these and other disciplines in HE.

Rhys Gwynllyw : DEWIS at The University of West England (UWE).

The DEWISComputer Aided Assessment (CAA) system has been designed and developed at the Department of Mathematics and Statistics, University of the West of England. The creation of the system was in direct response to problems encountered with commercial CAA systems used previously in the Department over the past 13 years. It is delivered to students in awards in the fields of Business, Computer Science and Software Engineering, Engineering and Mathematics and Statistics and has typically 11,000 student sessions per year. There are plans to make the code open-source and also to import the Mathletics database of questions (see Martin Greenhow above) into DEWIS.

Features:Many question types suitable for HE maths and stats questions; symbolic input, “on the fly” randomisation, verification of maths symbolic student input (will be using MathJax in the near future), follow on marking, malrule analysis and relevant feedback. Also supplies sophisticated feedback on continuation questions. There is an authoring tool aimed at lecturers writing questions for their modules.

Tim Lowe and Phil Butcher: Open University and e-assessment

The first part of the talk by Tim was on the MU123 module, Discovering Mathematics which was launched in October 2010. The aim is to increase retention and to better prepare students for subsequent modules. It is run through Moodle on a calendar basis and includes screen-casts, interactive activities and e-assessments which are run in a practice (PQ) followed by summative (iCMA) mode. There is a variety of Moodle based question types with immediate feedback on practice questions and delayed feedback on the summative tests.It is also possible to bring in questions created by OpenMark. Questions are randomised by caching a set of questions each varied 5 to 10 times and selecting randomly from the set. The student feedback was very positive on the mode of assessment.

Phil Butcher described the open-source developments at the OU, their attempts to get STACK funded and that a good route for open-source developments is via the OpenLearn website and recommended that people put their open-source developments of e-assessment on that site.

The Moodle VLE is open source and is under further development by the OU, however the e-assessments produced by the OU are not open. Moodle 2.1 will be available in the Autumn 2011 with a re-engineered Moodle quiz.

Anthony Youd and Christian Perfect: NewcastleUniversity experience and open-source developments.

Anthony described the use of the i-assess e-assessment system at NewcastleUniversitySchool of Maths & Stats since 2006. This is fully integrated into the curriculum and is used for the majority of formative in-course assessment in the first two years for all modules in the School as well as for large service courses. It uses symbolic input and “on the fly” display of student input in mathematical notation. All other common question types are available.

A full reporting and item-analysis database has been built “in-house” which also recovers all student input.

This has been a very successful implementation with excellent feedback to and from students and will continue.

Christian introduced another assessment tool aimed at both HE and outreach work with schools and colleges. The following attributes were needed: fast, scalable, can be launched from any browser or VLE, can be put on a CD ROM or DVD, can use latex for authoring, renders maths very quickly, no communication with a server required, can include multi-media and is extensible allowing more functionality. Such a system, called Numbas, has been built which allows interoperability with i-assess and has a latex based scripting language for authoring. Numbas has been developed entirely in Javascript to run in a web browser with no plugins or installation An important feature is that symbolic manipulation is performed entirely client-side and has been written from scratch. Numbasis planned to be open-source.

Forum: HE STEM e-assessment in Maths & Stats

22nd February 5.45 – 6.45pm

Chaired by Prof Cliff Beevers

Forum Discussion

The discussion brought out the fact that there were many common features in the workshop presentations of current e-assessment. Also that the introduction of these features was driven to a large partby the following mix of aspirations and requirements:

  • Creating sufficiently challenging questions for HE maths and stats with symbolic input(for example reflective questions).
  • Rich question environments for formative e-assessment and feedback.
  • Adaptive feedback to students
  • Reacting to student input intelligently
  • Monitoring of students progress.
  • Student centred – more in control of their paths through a course?
  • Monitoring of assessment performance and item analysis
  • Efficiency savings in staff resource e.g. standard text-based formative assessment and feedback involves a large-scale marking effort. Using formative e-assessment releases more time for lecturers includingfeedback on higher level skills not at present handled by e-assessment.
  • Good and accessible authoring tools for e-assessment questions and tests useable by lecturer or module-leader and adding to the pool available.
  • Secure, stable systems which are scalable. Possible use in large classes for summative assessments.
  • Good metadata and shared taxonomy towards interoperability and pooling with the HE STEM community.

There has been a consistent story of success in the implementations reported in the workshop, some realising many of the objectives listed above, and student feedback on the use of the systems has in general been excellent. We saw a mixture of local developments where such e-assessment systems have been built “in-house” and also the successful use of commercial systems.

Local Developments

There were several examples of bespoke local development which were driven by the above aspirations and requirements.

Can we pin down the motivation for these local developments?

Examples of continuous local development

  • HW and CALM/SCHOLAR etc.
  • Newcastle and iassess/Numbas also Diagnosys
  • OU and OpenMark/Moodle/OpenLearn
  • UWE and DEWIS
  • Birmingham and STACK
  • Surrey and StoMP
  • Brunel and Mathletics (modified Perception)

Perhaps they were motivated by perceived deficiencies in commercial products

  • Perceived loss of control if such a product used (e.g. text book prescribed and database of questions fixed).
  • Cost?
  • Upgrades leading to reported problems in reinstallation.
  • Upgrades not compatible with previous versions.
  • Scalabilityproblems for large classes.
  • Lack of expected support in particular technical support.
  • Institution or department/School needing more and better functionality which does not fit in with vendor’s development plan.
  • Reflective and higher level question types not generally available
  • Needing major adaption/change of department’s assessment strategy rather than the adaption of the product to the learning strategy.
  • Is HE maths & stats a market that the vendors take notice of and will they react to requests based on some of the above observations?

Problems with local development

  • There is, in many cases, a lack of support.Needed IT systems poorly supported, lack of formal co-operation of local Computer Services. Indifference or sometimes hostility of staff in school or dept.
  • Many such are based on short-term grants/investment and often short-lived local expertise and enthusiasm.
  • This resource input can disappear leaving the systemwithout support – no practical contingency or forward planning in place.

Locally developed e-assessment/e-learning systemswhich are integratedinto the maths or stats curriculum need a policy-based allocation of resourceat the required level which ensures continuity.

If such a system has shown its worth locally there is a natural tendency to share with the HE community especially if it has features developed specially for maths & stats which meet some of the requirements set out above. However, there are many obstacles to getting such a system noticed, demonstrated and used elsewhere.

One possible platform is the OpenLearnwebsite of the Open University.

Also how is such a system to bespread out to the HE maths and stats community?

For example, open-source developments. Once again this will need a support infrastructure depending on the type of license used. There have been open-source JISC funded developments in e-assessment for maths & stats which have not spread.

On the other hand, there have been open-source developments that have worked in the academic community and the reasons why they have succeeded need to be examined.

The Open University is providing infrastructure in terms of delivery via Moodle which could be exploited by a coherent approach to open-source developments in the community. For example, STACK developed at Birmingham can be delivered using Moodle. Once again this delivery potential should be explored.

Commercial e-assessment systems

These have been successfully used in many universities. MapleTA, MyMathLab, WileyPlus, QuestionMark Perception, i-Assess have provided successful e-assessment implementations in HE for application to maths & stats.

Properly supported by the vendor they have many advantages and are often usable straight from the box. All have databases which enable the monitoring of students and classes.

They can be relatively cheap to set-up, for example WileyPlus charges the student the price of a text book for access to the system and that is the only up-front cost. This is seen by some as a disadvantage in that the text is prescribed and the course is dictated by the assessment engine. Should this be an issue?

A major feature in some systems is the existence of large databases of questions in areas such as traditional methods and calculus courses. Tests can be created by simply choosing from these databases. However, it would seem to be an absolute necessity that academic staff can add their own questions by using authoring tools.

Support for the advanced use of e-assessment in maths & stats can be patchy and development of the product can lag what is required by the community. However, MapleTA allows the authoring of new questions and this, coupled with the powerful CAS Maple, allows the creation of sophisticated questions in maths & stats.

Evaluation of symbolic input: Computer Algebra Systems and e-assessment.

There are several ways that symbolic or algebraic student input can beevaluated. I-assess and SCHOLAR use a robust and well testednumerical method of evaluation and comparison; other systems use CAS given their power and ability to short-circuit the writing and checking of some question types. It is clear that systems with CAS checking are more suitable for higher level challenging questions (second or third year) at university. The overheads on using CAS can be quite high and there was some agreement that there could be a choice of which evaluation/checking system to use.

STACK, WileyPlus and MapleTA use Computer Algebra Systems allowing symbolic answers to be checked and also allowing the asking of students for examples of mathematical or statistical objects satisfying given properties. In general they allow the use of higher level questions which challenge students at second or third year levels of their degree courses. STACK is open-source and uses the Maxima CAS whereas the other two use MapleTA.