Evaluation of 2Nd Year Undergraduate Laboratory Work

School of Physics

Evaluation of 2nd year Undergraduate Laboratory Work

Summary of key points

This report is written for Dr Alan Usher, School of Physics, to evaluate changes to Stage II laboratories in 2008/9, using analysis of student feedback from 55 questionnaires and 2 focus groups. Key findings were that:

1The benefits of new core and supplementary laboratory design outweighed any disadvantages

2Key strengths of the new arrangements were improved ability to plan and organise time and increased flexibility

3Several suggestions were made about improving laboratory guidance manuscripts.

4Students valued the dialogue fostered through working in pairs with live feedback.

5Suggestions were made about how to improve conflict between partners.

6Assessment issues were most notable amongst student concerns, related to inconsistency of approach and grading.

7There were some examples of very good practice in feedback and its capture, but these were not universally applied.

The reportmakes recommendations about seven key areas:

student guidance about how attendance, linked to poor lab reports, may affect marks
adoption of a system to improve lab manuscripts
raising awareness of the value of teamwork
further training and development to build on good practice
method of assessment and feedback
development of exemplars of elements of worked reports at each grade
development of a more detailed mark scheme

1Introduction

In 2008/9, Stage II laboratories for PHY2017 and PHY2016 were organised in a different manner in two respects:

the use of 3 hour afternoon labs as non-compulsory supplementary labs, following 3 hour compulsory sessions in the morning
'live' marking of reports, in pairs, during feedback sessions.

In November 08, the Education Enhancement link adviser for the School of Physics, Marge Clarke, was invited to evaluate these changes. This evaluation and summary are written for Dr Alan Usher, Reader in Physics, to support consideration of further improvements and may help to inform the annual programme review.

2Method of data collection

The evaluation was conducted through two methods:

The use of student questionnaires, which were distributed to students during the compulsory lab session on 12 February and collected on the same day. The questionnaire was designed in collaboration with Dr Alan Usher, who introduced the changes to the way the lab session are being offered. A blank version of the questionnaire is attached at Appendix 1. The questionnaire drew on a balance of quantitative data, using a Likert scale and qualitative data, inviting comments about strengths and weaknesses. There was also space for ‘any other comments’. Fifty-five questionnaires were returned, of which one was discarded as ‘spoilt’.
Two student focus groups on 19 March 2009, each containing 6 members. Expressions of interest in being a member of the group were invited by the link adviser during a visit to the lab on 15 January, with an additional invitation included on the questionnaire. Ethics approval was given by Christine Heales, Chair of the School of Physics Ethics Committee, with an information and consent form being sent to students who had expressed an interest in advance of the focus groups. A copy of the interview schedule is attached at Appendix 2.

3Findings

3.1Core and supplementary lab design

Evidence from questionnaires and focus groups shows clearly that students consider that the benefits of the combination of taught morning sessions and supplementary afternoons sessions outweighed any weaknesses.

As shown in Table 1, 91% of students agreed or strongly agreed that they had used the afternoon sessions effectively, with 6% disagreeing with the statement. 3% of students did not know.

Comparing the new arrangements with the previous year, students in focus groups mentioned the previous ‘phenomenon of people not really wanting to be there’ with a certain amount of horseplay, and people ‘queuing up to leave at ¼ to 5’. In contrast, the new arrangements gave them the chance to use their own time in a more ‘adult’ way.

In response to a suggestion on the questionnaires that the supplementary lab should be on a different day, all the students in the focus groups disagreed with this, giving reasons such as the break in concentration, getting ‘out of the lab mentally’, risk of 1st years moving the equipment, and the extra travel time that would be involved.

3.1.1Planning and organising time

83% of students agreed or strongly agreed that the combination of taught morning sessions and supplementary afternoon sessions had helped improved their planning skills. 4% disagreed with the statement and 7% ticked ‘don’t know’.

Students made a range of positive comments on questionnaires about their improved ability to plan, for example:

System rewards good planning giving one more free time if experiment is planned properly.
Allows for routine data gathering elements to be accomplished in our time and harder parts to be guided with demonstrator.
Encourages me to sort out problems in the morning.

3.1.2Increased flexibility

50% of questionnaire responses citedincreased flexibilityas a strength, with being able to get other types of work done (33%) and avoiding wasted time (18%) as the most significant groupings. Comments include:

Flexibility is preferable, especially around exam time.
If you have all the results taken, you can work on writing it up in the supplementary session at home if you want.
It’s better for combined honours students as they only do half the credits and so don’t have to stay for longer than necessary.

3.1.3Attendance at supplementary sessions

Quantitative data was collected through the question ‘How many lab sessions have you attended?’All respondents completed this section, with results show in Table 2. 92% of students said they had attended about half or more lab sessions, with 73% saying they had attended ¾ or more of them.

Severalfocus group members volunteered vocal comments about the contrast between the rigorous attendance monitoring in the morning, saying that they might be criticised for being slightly late for those sessions but actually stay all afternoon. They were particularly concerned about taking breaks, citing the fact that they were admonished for taking a 17 minute break rather than the stipulated 15 minutes, and that they might be seeing a member of the Physics staff during that time, with no leeway given about this. Comments were made, such as:

They could be more human about it.
If the demonstrators were knocking about more they’d know if someone was taking the mickey.
You might miss 20 minutes in the morning but then spend 3 hours in the afternoon, and some people just go in the RAM, it’s a little bit unfair to be honest.

A minority of students were concerned about whether they would be ‘marked down’ if they failed to stay for the supplementary lab and felt the guidelines were insufficiently clear.

3.1.4Lack of access to demonstrators

Although 74% of questionnaire respondents commented that lack of demonstrators in the afternoon was a weakness, this was further explored in the focus groups to check whether they had felt ‘duty-bound’ to think up a weakness. Several students felt that this was ‘not much of an issue if you plan properly’, apart from the fact that equipment faults might not be resolved as easily. Some felt it would be advantageous to have one extra person there, whilst acknowledging that ‘Derek’s [lab technician] always there – he knows how to fix things’, in addition to Dr Usher.

Students in the focus groups volunteered their concerns about equipment faults forcibly, but with good humour. Several specific examples were given, including some to suggest that pairs could be unfairly disadvantaged by an older set of equipment than another pair.

We spent the 1st day wondering why it wouldn’t stick together, tightening all the nuts, soldering 12 joints. The other set had a plastic clear box and theirs worked straight away with wonderful results.

3.1.5Lab guidance manuscripts

57% of students agreed or strongly agreed that the lab guidance manuscripts had been useful. 20% disagreed or strongly disagreed with the statement and 23% said they did not know.

The focus group students’ main concern was with inconsistent quality of guidance manuscripts, and had several suggestions about how they could be improved. This was exemplified by comments such as:

Some have loads of depth, and some don’t.
You can see they’ve been written by loads of different people.

When prompted to give the best attributes of lab manuscripts, they made comments such as

They have really good diagrams and some very useful articles from magazines … it’s useful to have extra information, a different sources written in a different way.
The best introduce method points of analysis you should be considering.
They point you in the right direction, give you an example experiment of the sort you will be doing.
Some are very clear, very succinct, just 2 pages and you’ve memorised it.

Some of the least helpful aspects were that the lab manuscripts were vague or confusing, did not refer to current equipment or referred to a different experiment at another university.

Sometimes it’s a wall of maths and just says “Do the experiment”.
The demonstrator says “Oh, that refers to equipment we used a couple of years ago, we’re using this now”, and I’d spent the whole weekend preparing …” or “Oh yeah, that’s not working, that number should be 5 and these are all in the wrong order”.

On being challenged that differences and variety were inevitable, some students said that they could deal with them being ‘just woolly’ and ‘fuzzy’, but that some were simply wrong. Some time was spent discussing the level of access online and whether this could or could not be done through a Virtual Private Network (VPN). Most agreed that it could, but there was some confusion how this could be done and suggestions made that clearer guidelines would be useful. Most thought that WebCT was not a helpful tool.

Comments suggesting improvements included:

A couple of digital photos of the equipment would be good.
Make the lab notes similar.
A certain amount of direction is needed, for example, the aim of the experiment would be nice.
An induction to show how to use some of the equipment would be good.
Label the pages with a link to VPN instructions.

3.2Marking in pairs with live feedback

Evidence from questionnaires and focus groups clearly confirmed that they valued the interactive dialogue that the new arrangements offered. 85% of students agreed or strongly agreed with the statement that receiving feedback and discussing in pairs with their demonstrator had helped them to learn, as shown in Table 3. 11% of students disagreed or strongly disagreed with this statement and 4% said they did not know.

3.2.1Dialogue

Students in focus groups agreed wholeheartedly that having a dialogue with demonstrators was very helpful:

Looking through the script of the last experiment [in pairs with the demonstrator] was really useful.
You’re told each time how to improve by doing this, and this, ready for the next write up – it’s much better to discuss with them, why you did things this way what way to do it, it helps in the long run.

A fewfelt that they could be disadvantaged by not wording their answers in a particular way, or not being able to answer certain questions because they felt ‘blindsided’ by a question that they weren’t expecting.

3.2.2Working in pairs

Students in focus groups identified differences of motivation between partners as an area of concern, but there was some disagreement about the significance of this and potential solutions. There were differences of opinion about whether being ‘naturally less outspoken’ was a disadvantage, with some people feeling that just being in a pair helped people feel less nervous and could ‘pull marks up’ for both.

Most students who had less motivated partners felt considerable annoyance:

It was really difficult, my partner fell asleep because of a hangover.
Last year I was with someone who didn’t have a clue …. They ended up just holding something or just stripping a wire; intelligence wasn’t the issue, it’s the attitude that’s important; if their attitude’s OK you don’t mind going through and explaining it … it helps to improve yourself.
If your partner doesn’t turn up or put anything in, it doesn’t work. You need someone to bounce ideas off.

Some students were aware of guidance suggesting that work should not be shared with an ‘idle’ partner, but none felt they could make this work in practice:

No one’s going to do that, you’re going to create more trouble. It’s not realistic. The only way would be if you go to the demonstrator and show them, so that they can remove the partner … and then you’d never speak to them again.
If a lab partner’s not pulling their weight, and you don’t give them your notes, they’ll hold it against you. It’s easier to give on the small things, then they won’t hold it against you. It’s quite a lot of time to spend with someone who resents you.

However, the comment was also made that:

If you pick your friend or housemate, then you don’t want to let them down, ‘cos you wouldn’t hear the end of it; you’d be much more motivated.

Some felt that this conflict could extend to wider friendship groups and cause more serious rifts outside the lab. Students also debated how far the academic context mirrored a work environment in terms of working with partners, but the consensus was that the two were too different to be comparable.

Several solutions were discussed, in particular:

Being able to choose partners was seriously considered, especially as some students had done experiments where this was possible, and it had worked well.
Having a clearer procedure for being able to change partners ‘if it was really horrific’.
Having pre-determined pairings from similar ability groups, for example those doing Maths 1/Maths 2 in pairs.

The suggestion that pairs draw up a contract together at the start of the relationship was thought to be overcomplicated and unrealistic in terms of policing any breaches.

Whilst acknowledging that there are ‘always going to be some partnerships that work better than others’, the students’ major concerns centred around the impact of working in a pair on the fairness of their individual assessment.

3.2.3Assessment matters

The need for fair, transparent, consistent assessment was the issue which concerned students the most, and which tended to dominate other aspects of the evaluation:

It’s such a big chunk of our year – 20%.
If you’ve go a horrible demonstrator, you could be marked down by 10% [for the extended report].

Students’ perception was that the assessment was a negative process rather than a positive one:

They’re not giving marks for what you’ve done, they’re taking off for what you haven’t done. For the first time, I’ve got a positive comment which was really good.

They expressed a preference for a balance between positive and negative comments being made.

3.2.4Demonstrators’ approaches to assessment and feedback

Inconsistency of approach formed one of the central concerns about assessment. This was expressed in the context of considerable appreciation of their expertise, the value placed on learning simply by talking to them and some very positive comments about individual demonstrators’ kindliness, accessibility, care and attention.

Examples of comments are:

Certain demonstrators are so harsh and we all moan. We say ‘You’re really lucky’ … or we laugh and say ‘Oh, you’ve got that one, you’ll get 10% less’.

Some students felt that one of the causes of inconsistency might be that some demonstrators felt a conflict of interest, because they were ‘helping and assessing’, and that some had personal preconceptions about how demanding a particular experiment might be, and marked more or less favourably as a result. Their perception was that some feedback took an hour, but some never took more than 20 minutes.

One problem identified with paired assessment was that its ‘live’ nature could influence the outcome:

It would be better if all demonstrators went through [the report] beforehand; it’d be easier on them because they feel pressure. They say, ‘OK I’ll give you 5’ if you contest the mark. But some may be bullied into giving a 6. It’d be much fairer if they go through before have an idea, then discuss it afterwards.
When you’re sitting there, the demonstrator is scared to give you good marks – anything more than a ‘very good’. You have to outsmart them to get a good mark and then they’d mark you down anyway.

Concern was expressed that demonstrators differed considerably in the level of prior attention they paid to the lab reports. They felt that paired feedback worked well if it was ‘done properly’.

If I’ve put the effort into writing the report, at least I’d like it read. Some on skim-read it, they’ve only seen the front page and flick through the rest, and say “That’s fine”.

A ‘two-part system' was suggested in which the demonstrators would go through the reports beforehand and decide on a provisional mark for each person. The three-way dialogue was very helpful, but did not need to include details of each individual’s mark.

3.2.5Knowing how to improve

Students were quick to acknowledge the benefits of receiving feedback, linked with good questioning techniques, as a way of learning:

He [Dr Usher] goes through it - he explains why we're doing something. You may already have the answer in your head so he questions you to help get there yourself. Alan talks it through with questions, and suddenly you get it! If you had 10 Alans it would be amazing.

The consensus was that there was an overemphasis by demonstrators on formatting and structure, rather than content. Examples given were referencing styles, diagrams, whether graphs and results tables should be in a report or the appendices, how to draw graphs and label them. They also found that there were inconsistencies between demonstrators about these particular issues, such as drawing graphs, or not, what should and shouldn’t go into appendices and whether or not to use footnotes: