9351
Evaluation and accountability in the research process
Barbara Lawrence, Birkbeck College, University of London
Abstract
This paper discusses questions about the meaning of the term evaluation in the research process, arising out of a study of Access to science students. In negotiating access to students attending an Access course in science, problems arose over the issue of confidentiality to the researched, and the usual two-way interaction in evaluation of educational performance. This conflicting mis/understanding of evaluation in research and evaluation in the educational process is explored.
Introduction
This paper arises out of a study of students attending a science Access course in which problems arose over negotiating access to the students. It is thought that some of the problems were due to differing interpretations of the term evaluation and this will be explored, but the issue of accountability and the position of the researcher and the researched versus the gatekeeper to that access is important.
First I (the researcher) will describe the study, its aims and research methods. Second, I will put the study into a political context in relation to the institution in which the main part of the study is situated, and third, I will describe the process of negotiating access and the problems which came out of this. Finally, I will discuss generally the issues and questions involved.
Aims and research methods
The study has been funded for two years (one year at a time) by the UFC. It is investigating factors making for successful access to and completion of undergraduate courses in science by mature students with particular reference to women. It focuses on the access of adult students, particularly women, to part-time undergraduate courses in science, in the context of national discussion about appropriate models for science courses in Higher Education and the entry of mature science graduates into the labour market. Because the two years of the study have been funded separately, each year can be seen as a discrete piece of research in its own right, but with obvious connecting links between them. Both qualitative and quantitative research methods have been used.
The first year of the project had three stages:
1.a national profile of some university courses providing access to undergraduate courses in science for mature students with particular attention to women.
2.a more detailed case study of the characteristics of the current student body on first year undergraduate courses in Science, and Social Science courses at the University of Brenton (the name is fictitious).
3.the recording of the process of setting up the University of Brenton Access course in Science which began in September 1992.
The second year of the project is following closely the students on the Brenton Access course in Science through their year at the University of Brenton. By comparison with students on an established Access course in Science linked with another University, the investigation aims to gain an understanding of: the effectiveness of a part-time Access course in Science as a preparation for a Science degree; and the experience of participating in such a course and its impact on lifestyle, using the following methods:
1.a questionnaire to all course participants at the beginning and end of the course.
2.in-depth interviews with a sample of course members and tutors in order to provide a series of case studies on the courses in the two institutions.
The questionnaire was structured in such a way as to enable respondents to write freely on a number of topics, but the interviews were intended as the main method of gathering data and were modelled on the methods of Glaser and Strauss[1] in a modified form of grounded theory.
The research in context
The Science Access course at the University of Brenton evolved out of a concern, in a University which is committed to providing opportunities for mature students, to do three things. First, it was committed to furthering a developing programme of Access and foundation courses which would enable a broader range of mature students to be prepared for the rigours of part-time evening degree courses. Second, there was a desire to increase the success of a dwindling undergraduate science programme. It was felt that there was a greater need for a basic level of scientific expertise than in other disciplines to enable mature students without previous formal qualifications to participate in the science degrees being run at the University of Brenton. Third, there was a strong commitment to matching a strong scientific research programme with an equivalent degree programme in the University.
Negotiating access, issues of confidentiality and evaluation
An important part of a discussion of the research methods used in this study concern the question of access and confidentiality. It is common practice in sociological research that in order to conduct in-depth interviews the question of anonymity and confidentiality be paramount. Equally, questionnaires are usually completed so as to guarantee the respondents the same confidentiality. Consequently, questionnaires and interview tapes and transcripts (I taped the interviews which were then transcribed by a typist) are coded to preserve anonymity. Names of respondents are only used for contacting participants by post with questionnaires and for arranging interviews. Names, addresses and codes are kept and filed separately and it is considered important that respondents/participants can be absolutely sure of this confidentiality.
Problems may arise, and they did on this particular project, when the research is considered in terms of an evaluation. Having previously interviewed mostly in the area of health and health care, I was well-versed in the issues of confidentiality in the evaluative process. However, when interviewing in the educational field, the term evaluation may take on different connotations. The Chambers dictionary meaning given for the term ‘evaluate’ is ‘to determine the value of’. This gives no indication of how this may have evolved in practice. Consequently, sociological research methods and evaluation have tended to stress the one-way confidentiality of the evaluative process, while evaluation in the educational process has become a two-way interaction between teacher/lecturer/tutor and student in evaluating each other in an open situation. As such, evaluation of educational processes in sociological research has a validity of its own. In this particular example, however, evaluation in both senses was in operation. Not only was the research being undertaken, but a separate course evaluation was being conducted by the course co-ordinator, although this was largely conducted with the use of a questionnaire.
However, classifying the evaluative process as either one- way or two-way is in itself misleading. In the sociological research process, this idea of a ‘one-way’ direction of information from respondent/interviewee to interviewer has been discussed and criticised extensively in the literature. The feminist literature in particular has examined the whole interviewing process in qualitative research. As I have suggested previously[2] understanding the influence of gender as a dynamic in the interviewing process is particularly important, and as such implies that the idea of a one-way process is redundant. While it could be suggested that a woman interviewing women is unproblematic, this has been shown by others not to be the case[3]. And, McKee and O’Brien[4] have shown how the female interviewer may be at some risk in the interview situation with men. While this risk may be minimal in practice it can have a considerable influence over an interviewers interviewing style. And interviewing style is crucial in building up a rapport which encourages the respondent to share personal information. Pettigrew[5] discusses the whole issue of the ‘powerful’ interviewing the ‘powerless’, but fails to recognise the potential power of the interviewee over the interviewer. Oakley[6] and Finch[7] have described the instant rapport which commonly occurs in the interview situation between women, but other factors can upset this ‘balance’, such as gender, ethnicity, class or professional status.
There is another way in which the whole process of interviewing can be viewed as instrumental in understanding the evaluative process as either one-way or two-way. Oakley has defined the interview as a one-way process which ‘objectifies’ women. As she comments:
...when a feminist interviews women: (1) use of prescribed interviewing practice is morally indefensible; (2) general and irreconcilable contradictions at the heart of the textbook paradigm are exposed; and (3) it becomes clear that, in most cases, the goal of finding out about people through interviewing is best achieved when the relationship of interviewer and interviewee is non-hierarchical and when the interviewer is prepared to invest his or her own personal identity in the relationship.[8]
This all suggests the importance of the interviewer/ researcher in the whole interviewing process. Cicourel raised the issue of the part played by the researcher:
Field research...is a method in which the activities of the investigator play a crucial role in the data obtained.[9]
However, as I have suggested elsewhere, the ‘dynamics’ of the research process continue into the analysis and writing up of research and beyond. This is an issue which has been taken up by Opie[10] where she discusses the ‘empowerment of participants on a personal and broadly therapeutic plane’. She suggests that some participants are able to reflect and re-evaluate their experience as part of the process of being interviewed. However, she also identifies the problem of appropriation in which the researcher’s dominant view may subsume the participant’s comments and interpret and analyse them according to their own particular representation or view. As she says:
This is achieved through a filtering of the new (information) through the old (knowledge) so that the new is suppressed in favour of the old; and the potentially destabilising imprecate of new knowledge is subverted by a process which permits the established view to retain its dominance.
It is of course important to be aware of this potential when embarking on an analysis of any data but in particular in that of qualitative research because of its implied less ‘objective’ nature. Opie attempts to show how a deconstruction of textual practice may bring about a reduction of appropriation. As she says:
A deconstructive analysis requires the detailed assessing of the participant’s world...The researcher... is engaged in a fluid process of identifying and questioning ideology (her own, not merely the other’s), her location within the literature, the nature of her textual practice and the personal and political implications of methodology for the participants in the study.
The essence of this discussion has been to show how the idea of evaluation as either a two-way interaction in evaluation of educational performance, or a one-way research evaluation process is a false one. An extensive literature demonstrates the attempts made in the research situation to limit the one-way transmission of information into a dynamic situation between researcher and participant. And an understanding of this dynamic must be carried through into the analysis. As discussed above, the feminist literature has been particularly illuminating in examining and understanding what may be happening in the interview situation and in the evaluative/research process.
The main problems which I found arose over my obtaining access to the students and this seemed to stem from this so-called differing interpretation of evaluation by myself (the researcher) and the course co-ordinator. This occurred despite similarities in practice over the actual procedures on this occasion. It was these apparent differing interpretations of the term evaluation which led me to review and reconsider my own research practice. In reflecting on what I was doing I am able to suggest how there were differences in the two ‘evaluations’. First, the evaluation of the course in the research process is one part of a broader piece of research. In essence, the research is wider than the course in that it was also discussing with students things which are not directly related to the course, but rather are about what it is like doing a course. Crucial to this are the discussions about participants’ home lives and domestic situations and the impact of studying/attending a course on its equilibrium. This aspect of the research is particularly relevant to this study because the comparisons between the courses at the two institutions (one a part-time evening course, the other a full-time day course) could be expected to affect participants’ domestic situations in different ways. An analysis of these differences is particularly useful for course planning and development.
This leads to questions about the need for confidentiality in the research process. While the guarantee of confidentiality to the respondents seems in opposition to evaluation in the educational process, it is apparent that for the research project overall, confidentiality is crucial in ensuring the most ‘open’ participation from respondents. To use the example of the discussion of their domestic situations again, experience has shown me that people are loath to expose this aspect of their lives to general view. For instance, even with a guarantee of confidentiality, it would appear that far more households have an equal or near equal domestic division of labour than is in fact the case. On probing a breakdown of who does which domestic chores it is apparent that tasks are divided for the most part along traditional gender lines[11]. This is the sort of information which would not be forthcoming without that guarantee of confidentiality.
Finally, discussions of the part of confidentiality in the research process have led me to review who it is that I am accountable to in conducting such research. The two-way evaluation in the educational process ensures that all participants in such an exercise (i.e. students and teachers/tutors) are accountable to each other. In the research situation it is the responsibility of the researcher to ensure that the data and analysis are treated in such a way as to be absolutely accountable to all who have participated in the research process (ie students and teachers/tutors/course co-ordinator). This can be done by ensuring that the analysis is built upon a fair representation of all the views expressed, and to bring about a reduction of appropriation[12].
To sum up
This paper demonstrates a number of issues about accountability in the evaluative and research process. It opens up for discussion the need to reflect on our accepted research practice. I found that I was unprepared for the reactions of the course co-ordinator to what I considered were tried and tested methods, widely used and rarely questioned. It led me to reflect on my practice in a number of ways.
The paper has discussed a number of issues and attempted to clarify an understanding of differing interpretations of evaluation in the educational and research process. The discussion has centred around the feminist literature for an analysis of the interview as a central feature of the research process. This has led to questioning the way in which ensuring confidentiality for participants in the research is effective in enabling accountability to those participants.
[1] B Glaser and A Strauss (1968), The discovery of grounded theory. London: Weidenfeld and Nicolson
[2] B Lawrence (1987), Gender and