Journal of the Market Research Society, Jan 1993 v35 n1 p63(14)
Participantobservation and the pursuit of truth: methodological and ethical considerations. (includes reply) Marla Royne Stafford; Thomas F. Stafford.
Abstract: The methodological and moral issues concerning empirical data generated from participantobservation methods are analyzed to determine how accuracy and reliability of results can be improved. Participantobservation accounts for the activity taking place plus systematic and random interference which would vary depending on whether observation is covert or overt. As social scientists have previosly indicated, observation affects the activity, thus covert operations may yield the more accurate results. Covert operations can be used together with video recording devices to record the process to enhance the accuracy of observation processes.
Full Text: COPYRIGHT Market Research Society (UK) 1993
Introduction
As qualitative research continues to gain popularity among consumer researchers, participantobservation emerges as a method of interest. Participantobservation is ideally suited for gaining an insider's perspective, while allowing the researcher to observe surroundings and behaviour. As with any qualitative method, one must be concerned with ensuring that the observations are truly representative of the individual or group being studied. Further, are there 'better' ways of coming closer to that exact representation of the specific phenomenon which, for lack of a better word, is herein called truth? Of equal concern, in attempting to employ the most appropriate tactics, what ethical considerations might arise? In short, how does one balance maintaining methodological precision with ethical standards?
While this paper specifically investigates one particular method of qualitative research, the viewpoint advanced is that a mixture of traditional quantitative measurement philosophy with qualitative data-gathering methodology might result in more rigorous qualitative research approaches which can still provide the richness of observation without giving up the quest toward underlying 'truth.' The specific concern here is for the value of information gathered in participantobservation studies.
In recognising that one cannot, as a practical matter, separate the researcher's subjective influences and interpretations from the attempt at objective interpretation of observations, some have proposed philosophies that admit the subjectivity of the scientific process (Hirschman 1986; Peter & Olson 1983; Lincoln & Guba 1985). However, according to Kirk & Miller (1986), objectivity (as partitioned into its component attributes of reliability and validity) is the essential basis of all good science, be it quantitative or qualitative. Without objectivity, the only basis for accepting research findings would be on the authority of the individual author of the research. Thus, it seems important to find a way to introduce subjective agreement between participant observers into qualitative research approaches. Doing so would introduce greater degrees of reliability to a research paradigm that has traditionally favoured richness at the expense of reliability (Deshpande 1983).
Qualitatively-orientated philosophers suggest that objectivity has been sought but not achieved. Qualitative research, in general, (Calder 1977) and, specifically, participantobservation (Bouchard 1976), are subject to inherent threats to validity. On the one hand, since reliability and validity are so interrelated, increasing the former might also improve the latter (Peter 1981). On the other hand, it is suggested that the inherent characteristics of qualitative research are not conducive to reliability. While one might be able to replicate data, interpretations of those data might differ. Sykes (1990) proposes an exacting documentation process for observational naturalistic research, reasoning that detailed documentation would enable other researchers to 'replicate' the findings of particular studies, thereby solving, if only in part, the reliability problem.
It seems that there might be some advantage to be gained by integrating certain approaches to 'truth' with some quantitative measurement philosophies for increasing rigour in research. The goal here would be to increase the understanding of the observations made in qualitative research, especially participantobservation. To that end, the purposes of this paper are to discuss: (1) the empirical concerns for the assessment of 'truth' in participantobservation; (2) the various trade-offs, in terms of ethics and 'truth value,' involved in the differing levels of participantobservation; and (3) the use of recording devices in participantobservation as an aid to more accurately assessing 'truth' and the ethical issues that arise from their use.
Participantobservation and the 'truth' of findings
Whilst traditional quantitative, research perspectives tend to favour measurement quality, or what Deshpande (1983) terms reliability, and qualitative approaches tend to favour richness of data, which is analogous to validity (Deshpande 1983), the norms for good science also dictate that social science research should exhibit both qualities (Kirk & Miller 1986; Hunt 1984).
However, if one accepts the suggestion to mix methods from both qualitative and quantitative paradigms (Bouchard 1976; Deshpande 1983), then it appears that one of the most useful concepts that could be brought to qualitative methodology is that of measurement theory, as outlined by Churchill (1979). In relating his paradigm for better measures of marketing constructs, Churchill offers the following conceptualisation of the components of observation:
!X.sub.O^ = !X.sub.T^ + !X.sub.S^ + !X.sub.R^
The meaning of this model is simple: what we observe is a combination of the 'true' phenomenon actually taking place, plus a combination of systematic and random interference (ie, error) with our attempts to observe the phenomenon. The significance of this basic conceptualisation of the process of measurement is in the deliberate and specific recognition of interfering factors. This view is not incompatible with the qualitative approach. While those who advocate qualitative methods might argue with the ultimate nature of 'truth,' they, too, concede that errors are inherent in observations (Hunt 1984).
By the recognition of inherent error, one is forced specifically to recognise the factors which intrude upon observational accuracy, rather than ignoring them and hoping that they will simply go away. Some qualitative scientists, in proposing that truth is situational and contextual, could fall into the trap of thinking that one can never begin to understand a specific phenomenon at a more basic, 'true' level. In addition, it is possible that by relying on subjective interpretation alone, and ignoring the inherent nature of error, qualitative scientists could be greatly increasing the !X.sub.s^ (systematic error or bias) component of measurement in their studies. This could happen when researchers allow personal beliefs and individual biases to intrude upon what should rightly be a purely objective process of data gathering and interpretation.
Very simply, with qualitative methods such as participantobservation, it is impossible on a practical level to separate the gathering of data from their interpretation. As such, the danger of bias exists without sufficient checks on interpretation. What value does biased observation have if scientists fail to recognise and attempt to account for that bias? Is this 'truth'? Probably not, even if consensus can be built around the biased observations.
As previously noted, reliability is a weak element in qualitative research; it is a trade-off for the richness of data obtained. In participantobservation, the observer becomes the instrument. The ability of the observer consistently to observe and interpret phenomena, or to agree with other observers about those phenomena, is the key to better levels of reliability in participantobservation, and through this, a better understanding of the phenomena being studied. Truth may depend on the situation or context, but this does not necessarily mean that a better understanding or a more accurate approximation of 'the truth' cannot be shared by several similarly trained observers of the same specific phenomenon. Jorgensen (1989) indicates that the use of multiple observers together with the ability to independently replicate the study and verify procedures can increase 'dependability' and 'trustworthiness' of participantobservation. Similarly, Sykes (1990) observes that 'consultative validity' can be achieved by involving other investigators in the interpretation of observational findings. Similarly trained scientists should arrive at equivalent interpretations and thereby increase one's confidence in the 'truth' of what they have seen.
It is worth considering the possibility that methods which can aid in the 'replication' of qualitative observations and interpretations of phenomena can contribute greatly to the 'truth value' of research findings in participantobservation. With the introduction of greater levels of reliability through the process of independent verification, one component in the Churchill (1979) model for better measures is bolstered: reliability in measurement decreases random error, subsequently increasing the accuracy of the observation. An increase in the reliability of observational interpretations also increases validity, bringing one closer to an understanding of the 'true' phenomena of study.
There might also be specific methods one can consider for control of Churchill's other error component. Systematic variation relates to bias factors introduced either by the subjects' reactions to the presence of the researcher or the researcher's interpretation of the phenomenon under study. These issues are equally important and will be addressed below.
Overt vs covert participantobservation
Overall, participantobservation allows for two major categories of research: overt and covert. When the researcher openly requests permission to observe a situation, and plans to make his or her identity, objectives, and intentions known, the research is considered overt. Overt research is generally considered ethically acceptable, but it is also considered obtrusive, leading to two problems: (1) the researcher may be refused access, causing subsequent research attempts to be denied; or (2) even if access is granted, subjects may react to the researcher's presence. As the latter is of most concern here, discussion will centre on this issue.
The effects of observation on subjects
Numerous social scientists have recognised that a primary problem in overt research is 'reactivity,' or subjects' reactions to the knowledge that they are being observed (Bouchard 1976; Kidder, Judd & Smith 1986; Neale & Liebert 1980; Schuler 1982; Kazdin 1982). Neale & Liebert (1980) assert that the act of observation itself may produce changes in the phenomenon being studied and that this problem is hardly unique in social science research. They state:
"the presence of such reactions may render the obtained results unrepresentative of the natural situation in which the investigator is ultimately interested....any such situation may be referred to as a reactive arrangement." (Neale & Liebert 1980, p 44)
The effects of reaction to observation are particularly well-illustrated by the Hawthorne Studies. During the late 1920s and early 1930s, employees at the Hawthorne plant of the Western Electric Company were observed in a study examining worker productivity. Various treatments were introduced into working conditions (eg lighting changes, work hour changes). No matter which treatment was employed, worker productivity increased (Neale & Liebert 1980). Even the control groups, who received no treatment at all, experienced an increase in productivity (Griffin 1987). This increase in performance due to one's knowledge of being observed became known as the Hawthorne Effect.
This reaction effect can occur when any aspect of the investigation causes the subject to act upon what he or she believes is expected or desired of him or her. In short, bias may result if the true purpose of the research is discovered or if it becomes known that researchers are making observations. This bias corresponds to the systematic error identified in Churchill's (1979) model of the components of observation.
Kazdin (1982) states that direct observation has its own source of bias, and Robson & Wardle (1988) report that a comparison of two groups -- one observed and one not observed -- resulted in the observed group responding with answers that were shallower, more compromised, more moderate and more superficial. In short, less 'truthful' answers were obtained.
Neale & Liebert (1980) suggest that the researcher create a situation where subjects are involved in everyday activities and are not aware that they are participants in a scientific investigation. Schuler (1982) proposes complete participation, where the researcher assumes a fictitious identity.
Such concerns with overt research techniques build a persuasive argument for the utilisation of covert techniques, where the researcher becomes an 'insider.' In this case, the true identity of the researcher remains concealed, and subjects believe the individual to be a new member of their group. Since group members are unaware of being observed, data generated by such methods are not contaminated by respondent reaction (Bouchard 1976). As a result, the truth value of the observations is greatly increased.
Ethical considerations
While covert techniques may lead to greater precision and greater truth value in qualitative research, there are ethical concerns, particularly over the deceptive nature of such approaches. As a result, many researchers question their use.
Deception in the use of covert techniques may take two forms: (1) the subjects being studied are not informed of the research; and (2) because they are unaware of the research, subjects are not asked for the traditionally required 'informed consent.' For these reasons, some researchers maintain that covert strategies violate the rights of human subjects, particularly the right to informed consent. However, since the notion of informed consent was developed after the Nuremberg trials of war criminals charged with carrying out extreme medical experiments during World War II, some researchers argue that such consent is not applicable in observational research where the intent of the researcher is not harmful (Bulmer 1982).
On the other hand, if deception is finally disclosed, future research in that setting or environment may not be practical. Once the trust has been violated, it is usually impossible to reinstate it. Furthermore, the reputation of colleagues in the field may be placed at risk when such deceptive tactics are revealed. According to Homan & Bulmer (1982), concerns over deception, violation of trust, and the possibility of damage to the reputation of the discipline as a result of the use of these tactics is not limited to covert methodology; they are probable hazards in overt investigations as well.
In advocating covert techniques, some researchers indicate that their use is essential to acquiring truthful information (cf, Jorgensen 1989). Douglas (1976) believes that most human research would be virtually meaningless if participant observers were required to reveal their intentions and argues that covert research is a necessary, useful and revealing method. Jorgensen (1989) notes that the limited access which accompanies more overt tactics generally results in less valid and less reliable findings. For this reason, some participant observers defend covert strategies, reasoning that there are important aspects of human existence that cannot be known except from the inside. Denzin (Denzin & Erikson 1982) argues that revealing the purpose of the investigation is unnecessary, and informed consent need not be sought. He states that social scientists:
"have the right to make observations of anyone in any setting to the extent that he (sic) does so with scientific intent and purpose. Any method that moves us toward advancement of knowledge in science without unnecessary harm to subjects is justifiable. The method must not damage credibility or reputation of the subject, and the researcher must take pains to maintain the integrity and anonymity of the subjects." (Denzin & Erikson 1982, p143)
Hodges (1988) outlines the conditions under which deception is acceptable. Though his work was directed toward journalists, one can easily liken the work of participant observers with that of reporters, since both are trying to explain a set of events. The first condition is that the information sought must be of overriding public importance. Secondly, there should be no reasonable likelihood that comparably accurate and reliable information could be obtained as efficiently through conventional investigative techniques. And thirdly, the proposed deception must not place innocent people at serious risk.
The question of whether a particular research project is of overriding public importance is admittedly subjective, but most researchers would certainly maintain the (subjective) importance of their own work. With regard to the second requirement, the additional use of covert documentation provides for more accurate (thus, more reliable and ultimately more valid) information that can be obtained through participantobservation, alone. Thirdly, most ethical social scientists do not wish to harm subjects (Bulmer 1982; Jorgensen 1989).
In marketing research, additional protection is provided by at least two professional codes of ethics which strictly prohibit researchers from placing any individual at risk for purposes of advancing the discipline; these ethical codes are published by the American Marketing Association (1991) and The Market Research Society (1992). Of the two organisations, only The Market Research Society has something substantial to say specifically on participantobservation. For example, covert participantobservation should only take place without participants' permission when they are in a situation where they might reasonably expect to be seen or heard (Code of Conduct, Rule A14, The Market Research Society, 1992). Standard and more general prohibitions against the unauthorised release or misuse of data are provided by the American Marketing Association and are echoed by The Market Research Society in Rule A2, which specifies that outside access to research data be limited to cases of data processing and validity checks.
Levels of participantobservation: methodological give and take
While the two general types of participantobservation include the polar views of overt versus covert, one can look further and examine the various types or levels of observation. These four levels range along a continuum of complete participant, participant-as-observer, observer-as-participant and complete observer (Babbie 1986; Bouchard 1976; Jorgensen 1989). Of these four, three are considered overt techniques, and one, complete participation, is covert.
As mentioned above, a researcher employs complete participation by surreptitiously becoming a member of the group under study. Although complete participation is potentially the most deceptive level of participation, it has already been noted that such research is generally pursued based on the belief that the data will be of greater value. If a researcher remains a member of the group for an extended period of time, more accurate and valuable information can be obtained (Bouchard 1976; Hirschman 1986). Not only will subjects continue to exist and interact normally, but the researcher will be able to achieve a deeper understanding of the meaning of the phenomena being studied. For example, a consumer researcher studying buyer/seller relationships in a retail setting may accept a position as a salesperson in order to become integrated into the group. Through actual participation, the researcher can become part of the process and consequently, achieve a deeper understanding of the buyer/seller relationship. In a similar vein, an advertising agency developing a beer campaign might send an agency employee out to a local pub to become friends with the patrons. By becoming a member of the group, the employee can learn about aspects of beer consumption that he, otherwise, might never be exposed to.