A Web Based Module Evaluation System

Interim Innovation Project report to the Learning & Teaching Committee

Nadine Foster (Dept of Physiotherapy Studies), Nikki Rochford (Dept of Physiotherapy Studies), Barry Smalley (School of Chemistry & Physics), Stephen Bostock (Staff Development & Training).

Aim of Project

To develop a web-based system to facilitate the rapid analysis of module evaluations. The module evaluation system will be based on one currently in use by the Physics Department.

Background and Purpose

The Department of Physiotherapy Studies has demonstrated a keen determination to improve the already high standard of quality assurance mechanisms used within the department. Whilst the department feels its present mechanism of module evaluation is effective, it’s reliance on paper questionnaires and the resultant time required in the analysis and production of useful data, renders this process rather intensive in terms of staff time and of questionable cost-effectiveness. The relevance of these issues in a quality assurance programme is recognised and the aim of this project was to improve the efficiency and cost-effectiveness of the module evaluation process.

We further proposed to develop this into a portable system using the Physiotherapy Undergraduate course as a pilot for a more general system. This evaluation system will also facilitate the use of information technology by students and will respond to the needs for impartial collection of student views without unnecessary use of staff time.

The Department of Physiotherapy Studies further recognised that the paper module evaluation forms were not fully student-centred. It was felt that they did not consistently investigate students’ attitudes towards their own responsibility for the learning process. It was therefore intended that the modular evaluation form developed for use on the web would investigate the extent to which the student actively engages in the learning process.

We also recognised the need to be able to compare the students’ learning experiences across modules. It was expected, therefore, that the web-based module evaluation system would facilitate comparison in the student experience across modules. The development of this web-based module evaluation system, within the Physiotherapy Department would act as a pilot to evaluate the transferability of the system to other departments within the University. This will fit well with the aims of the Quality Assurance initiatives to enable comparison across departments and Faculties.

It was further expected that this project would contribute to the embedding of IT within the BSc (Hons) Physiotherapy programme, as well as the promotion of the transferable skills of life-long and active learning, via student reflection.

Objectives

  • To develop a standard module evaluation form for the BSc Physiotherapy Studies (Hons) programme, which would facilitate comparability across modules
  • To further develop & enhance the supporting software to analyse and present the evaluation data producing meaningful web-based summary data.
  • To develop a user-friendly tool which does not require a high degree of computer literacy to use
  • To pilot the developed evaluation questionnaire forms within the Physiotherapy programme
  • To investigate how feasible it might be to use this web-based system of module evaluation across the wider University.

Method

In order to meet the above objectives, the following procedure was undertaken:

  1. Current module evaluation tools used within academic environments were reviewed for their appropriateness and pertinence, additionally focus groups and staff discussion were utilised to identify the main areas considered to be important in a module evaluation.
  2. From these a draft questionnaire was then developed.
  3. The draft questionnaire was which was reviewed by all staff and representatives of the student body for relevance and appropriateness and further developed on the feedback.
  4. The developed module evaluation form was then adapted to a web based system, which was then piloted within all first semester physiotherapy modules.

Attached to this report are examples of the form and the outputs.

Results of Pilot

  1. Comparability between web based and paper evaluations.

To ascertain if the computer results were comparable to the results gained from a paper evaluation the student cohorts were randomly assigned to two groups, the first undertook the computer evaluation and the second completed the same evaluation form but on paper.

The following table gives a comparison of the results obtained from these two sources.

Modal scoreModal score
for web versionfor paper version

Module 144

Module 244

Module 344

Module 444

Module 543

Average number of students giving a low score (1,2) for a question,
out of about 38 students (and range of values for any one question)

for web versionfor paper version

Module 12.7 (range 0-22)6.7 (range 0-31)

Module 27.1 (range 0-25)5.5 (range 0-22)

Module 32.5 (range 0-20)3.5 (range 0-18)

Module 48.5 (range 0-22)9.7 (range 3-17)

Module 55.1 (range 0-14)7.7 (range 1-15)

It is clear that the method of delivery did not systematically affect the results obtained from the module evaluation.

  1. Staff Evaluation

A questionnaire was completed by all module leaders to determine their views of the new system:

Staff preference

All 5 staff preferred the new evaluation form

2 staff preferred computer system to the paper based one

3 staff stated no preference between the two systems.

Benefits

All staff stated that the main benefit of the new system was the amount of time saved.

The ease of access to results was also considered to be a benefit.

  1. Student Evaluation

Students also were asked to complete a questionnaire concerning their attitudes towards the new evaluation system. The following graphs illustrate the results of this audit.


Overall, students were happy with the new system. A larger number of students who had completed paper evaluations stated a preference for paper, but they had had no experience of the web based system. Those who used the web form, on balance, preferred it. They were positive about the design of the form. Various improvements were suggested, many of which had already been considered. A large majority of students felt the medium made no difference to their answers, and those that thought it did were not consistent in the effect they thought it would have. A majority thought it would be necessary to book a PC lab to complete the web forms were the computer system adopted.

Further developments

  • The standard web form is now being used for all physiotherapy modules, Years 1, 2 and 3.
  • A programme evaluation questionnaire had additionally been developed and was piloted at the end of May 2002; the results are pending.
  • The web report now provides summary data for inclusion in the course report. The usefulness of this will be determined after summer.
  • The transferability of the system to other departments is now to be determined.

Conclusion

It is expected that, with minor amendments, this method of module evaluation will be embedded within the plans for the new BSc (Hons) Physiotherapy course due to commence October 2002. This is expected to result in a more cost-effective and efficient method of evaluation. Added benefits for Physiotherapy Studies include the increasing use of IT by students and the improvement of information quality gathered as part of module evaluation. The system design will explore long-term issues such as ensuring portability. It is intended that the transferability of this tool to other departments will have been explored and recommendations made for campus-wide availability.

21/09/18