Results of the SET and SEM Survey – February 2014
PGR Results

The following document summarises the findings of the University of Nottingham Students’ Union’s survey on the Student Evaluation of Teaching (SET) and Student Evaluation of Modules (SEM) programmes that are in operation at the University of Nottingham. In particular this document explores the attitudes of Postgraduate Research (PGR) students, particularly PGR students who teach other students, to proposed changes to the SET and SEM programmes in operation. PGR students broadly felt that they should not be exempted from SET and SET evaluations, but that the results should not be made available to students. Respondents also felt that taking part in the SET and SEM programmes would aid their professional development.

Methodology

Respondents were invited to complete a web-based survey, hosted on SurveyMonkey, to measure:

1)The operation of SET and SEM programmes currently in place

2)Student responses to proposals that are forthcoming from the Evaluate project.

3)The level that PGR Teachers would like to be assessed in the SET and SEM programme.

4)The way in which SET and SEM scores could be used to aid the professional development of early career teachers.

5)How valued PGR Teachers feel in their role.

Data collection was completed in early February, with collection closed on the 19th February 2014; collection was publicised through the Students’ Union’s own communication channels including social media and representation structures. Survey response was incentivised by a prize-draw, with respondents able to opt-out of the prize-draw if they wished.

Limitations

It is notable that the sample is small, with a total turn-out of 48; this is not ideal, and means that these data are not necessarily robust when interpolating across the entirety of the student body.

The University of Nottingham has approximately 3000 Postgraduate Research students, and this survey was aimed at them all – it is unknown how many PGR students are actively teaching other students, with conservative estimates between 10-25% - it is therefore disappointing that such a low turn-out was felt in the survey.

Demographics were measured by:

1)The School/DTC that the respondent felt that they were most aligned to

2)Their domicile

3)The level of teaching that they were currently engaged with.

Bar characteristic three, these demographic data will be analysed in the next section, but will only be used for information/validation only – they will not be used for the purposes of wider cross-tabulations.

Demographic data and turn-out

As has been alluded to, the sample size was small, and this leads to the data being not as robust as they could be: this document will assume some general conclusions that should be subject to re-testing to prove their validity. Following cleaning of the data, there were 48 complete responses to the survey.

Chart 1: School/DTC Alignment

A significant minority of responses came from M3 (n=11), with 21 responses from within the Faculty of Engineering; as can be seen, there is a small weighting away from respondents in the Arts and Social Sciences. As the turn-out is so small, however, these demographic data are only presented for information.

Table 1: Domicile of respondents

Count
UK / 26
EU / 4
Overseas / 18

A small majority of respondents were UK students, with 18 International students, and 4 EU students. As has been stated, these data are presented for information only.

Chart 2: Level of Teaching

A significant minority of respondents were currently teaching (n=20), with a majority of respondents either currently teaching or with recent teaching experience at University of Nottingham; 10 respondents were not seeking to teach in future. The average scores for active teachers will be presented below.

Results

Respondents were asked to respond to ten statements around their experiences as PGR students:

1)After completing my postgraduate qualification I intend to go into teaching in the Higher Education sector.

2)I believe that students evaluating my teaching would be beneficial to my professional development

3)I worry that the publication of student evaluation of my teaching may be detrimental to my future career prospects

4)The evaluation of university teaching should only include full or part-time lecturers, and not postgraduates who teach

5)Postgraduates who teach should face different criteria in the student evaluation of their teaching than full or part-time lecturers.

6)If students were to evaluate my teaching, I would prefer that feedback not to be made publicly available

7)I already use student feedback on my teaching to aid my professional development

8)I feel valued as a postgraduate who teaches at the University of Nottingham.

9)I regularly discuss my professional development as a teacher with my supervisor[s]

10)If I had the scores of the student evaluation of my teaching, then I would discuss them with my supervisor[s]

The majority of these statements were based around the experience of PGR students as teachers, although several were more opinion-based, and could be completed by those with no teaching experience; some statements were designed with agreement in mind, and others with disagreement. Respondents were also invited to complete an open-text question to expand on their thoughts.

Respondents were asked to score their responses from 1-10, with 10 being complete agreement, and 1 being no agreement; for the purposes of analysis a score of seven or eight is an apathetic response: as the response rate was small, a large spread of scores affects the mean more significantly, thus meaning that a largely negative or positive response can be skewed to look apathetic – this is especially the case in statements three, four and six.

Chart3: Mean scores – overall and active teachers

Table 2: Responses to statements (modal responses highlighted)

1 (least agree) / 2 / 3 / 4 / 5 / 6 / 7 / 8 / 9 / 10 (most agree)
After completing my postgraduate qualification I intend to go into teaching in the Higher Education sector. / 6 / 3 / 6 / 2 / 5 / 7 / 3 / 3 / 5 / 8
I believe that students evaluating my teaching would be beneficial to my professional development / 3 / 0 / 0 / 0 / 3 / 5 / 9 / 10 / 4 / 14
I worry that the publication of student evaluation of my teaching may be detrimental to my future career prospects / 10 / 2 / 3 / 6 / 8 / 5 / 4 / 4 / 1 / 5
The evaluation of university teaching should only include full or part-time lecturers, and not postgraduates who teach / 9 / 7 / 6 / 5 / 3 / 4 / 4 / 2 / 4 / 4
Postgraduates who teach should face different criteria in the student evaluation of their teaching than full or part-time lecturers. / 4 / 1 / 1 / 0 / 2 / 4 / 6 / 8 / 8 / 14
If students were to evaluate my teaching, I would prefer that feedback not to be made publicly available / 7 / 1 / 3 / 3 / 0 / 4 / 8 / 6 / 3 / 13
I already use student feedback on my teaching to aid my professional development / 17 / 6 / 1 / 4 / 4 / 3 / 3 / 3 / 1 / 5
I feel valued as a postgraduate who teaches at the University of Nottingham. / 6 / 2 / 4 / 7 / 6 / 5 / 6 / 4 / 2 / 6
I regularly discuss my professional development as a teacher with my supervisor[s] / 16 / 6 / 2 / 6 / 6 / 2 / 2 / 1 / 3 / 3
If I had the scores of the student evaluation of my teaching, then I would discuss them with my supervisor[s] / 4 / 2 / 2 / 1 / 5 / 4 / 9 / 9 / 4 / 8

A small majority of respondents (n=19) wanted to go into teaching in the Higher Education sector after finishing their course, although, as is inevitable active teachers were more likely to want to go into teaching afterwards (mean=5.85); a similar amount (n=17) did not want to go into teaching, with 12 undecided.

The majority of respondents aren’t currently using SET and SEM scores to aid their professional development (n=29), although a small minority are definitely currently using them (n=5), and current active teachers slightly more likely to be using them (mean=3.98). Whilst there isn’t current use, the majority of respondents (n=37) would value their inclusion into their professional development – this is the statement with the lowest level of disagreement over the 10 statements.

Few respondents currently discuss their professional development as a teacher with their supervisors (n=31), with very few (n=9) actively discussing their development as a teacher as part of their supervision, most (n=30) would discuss their feedback scores with the supervisors if they received them.

Whilst there is a desire among respondents for the results of their feedback to be available to them as part of their professional development, few would want them to be viewed publicly – these data are expanded in several open-text comments:

“I would be very happy to receive, and perhaps be able to keep, feedback on my own teaching, but I do not believe there is any benefit to making such evaluations publically available. I believe such publication could give an unfair image of PGs who teach, many of whom are teaching for the first time, and prejudice their future career and the way they are treated by students, and perhaps staff.” – PGR, Humanities

“I don't agree with making student feedback of PhDs teaching public. Afterall, this is a learning stage of their career and I don't think it should be judged (publically) on the same criteria as those in post.” – PGR, Humanities

Most respondents would prefer that the evaluation of their teaching not be made public (n=30), and many feel that making their scores public may be detrimental to their future careers (n=21): it is notable that currently active teachers are more worried that publishing their SET and SEM results would be detrimental to their careers. Many would also like to be judged against differing criteria to later-stage teachers (mean=7.52 and 8.20 among active teachers).

PGRs are feeling undervalued in their teaching responsibilities, with a large minority within this survey indicating that they feel this way; this, however, is a very mixed picture with many respondents also feeling equally valued in their roles (mean=3.98).

Conclusions

Due to the very small turn-out of this survey, these results can only be broadly representative of opinion – findings should be subject to a retest before any firm conclusions can be made - there are, however, certain trends coming through in the data that can be seen.

Overall respondents appear to feel that students evaluating their teaching would be good for their professional development; very few want to opt-out of the SET and SEM process, and most feel that they would like to be able to use it more explicitly in their professional development. It is, however, apparent that respondents do have some worries about the public use of these data. It is apparent that feeding these findings more explicitly into the supervision process may be valued by some, and for students be able to access more training before they begin teaching, as one respondent points out:

“Proper training on teaching should be provided to the postgraduates before they start teaching.”

Whilst the relationship between supervisor and student is one that is negotiated on an individual level, there is space for using it to develop professional skills more thoroughly, and the use of SET and SEMs score could underpin this; it is, however, notable that many respondents do not necessarily want this to be the case, and this should only be guidance not policy.

As is understood from the report on the Undergraduate and PGT findings, there is a definite hunger from those students to have a more complex data-set when they select their modules, but it is felt by PGR students that they wish to be exempt from public reporting of results. Whilst some respondents feel that a different methodology should be used, it is unclear how this can be implemented without adding an increased administrative burden on those completing the evaluation – instead, there should an automatic exemption for PGR students from public reporting, with the evaluation only to be used for their professional development. In the development of PGR students as early career academics it is evident that a whole host of skills are required, and teaching skills are just as important as research skills, however, making PGR teachers accountable for the relative satisfaction in a module is not fair on the PGR teacher, or the student being taught.

Samuel R. Nichols AMRS
February 2014

1