Hackett, R., Gilsdorf, W., & Savage, P. (1992, January 1). News Balance Rhetoric: The Fraser Institute's Political Appropriation of Content Analysis. Canadian Journal of Communication [Online], 17(1). Available:
News Balance Rhetoric: The Fraser Institute's Political Appropriation of Content Analysis
Robert A. Hackett (Simon Fraser University), William O. Gilsdorf (Concordia University), Philip Savage (Canadian Broadcasting Corporation)
Abstract: The Fraser Institute's bulletin On Balance, which offers monthly content analyses of news, is critically examined. Methodological, conceptual and epistemological problems are discussed, as well as a rhetorical mode of reporting results which implies a politicized interpretation of news.
Résumé: On fait la critique du bulletin de l'Institut Fraser, On Balance, qui offre des analyses mensuelles du contenu des nouvelles. On discute les problémes de méthodologie, de conceptualisation, et d'épistémologie, et même que le mode rhétorique de rapporter les résultats qui implique une interprétation politique des nouvelles.
Introduction
Increasingly, the mass media are recognized not simply as observers and reflectors of political life, but as themselves political players and definers of reality. It is not surprising then, that political advocacy groups are devoting increasingly more attention and resources not only to conveying their messages through the media, but to scrutinizing and pressuring the media directly. Such organizations hope thereby to gain more access and public profile, delegitimize opponents, displace rivals, and/or shift the very terms of debate. Indeed, in the U.S. at least, some advocacy groups (such as the right-wing Accuracy in Media, or AIM, and the left-liberal Fairness and Accuracy in Reporting, or FAIR) have arisen specifically for this purpose. Advocacy groups attempting to influence U.S. network television programming have had differing degrees of success, varying with the goals and strategies of each group in relation to the imperatives of the network TV system, as well as external factors beyond each group's control -- changes in the political climate, shifts in television industry practices and priorities. Advocacy groups have had at least some impact on prime-time programming (Montgomery, 1989, p. 217). In that respect, it is unfortunate that media scholars have paid relatively little attention to the claims, strategies and effectiveness of pressure groups which make the media a primary focus of their activities.
The tactics employed by such groups have ranged from letter-writing campaigns and public criticism through regular newsletters and other publications, to more ``hard ball'' tactics, such as advertiser boycotts, law suits, appeals to government authorities including broadcasting regulatory agencies and even (in the case of wealthy pressure groups) attempted ownership takeovers (see, e.g., Montgomery, 1989, p. 10). Amongst the repertoire of such tactics is one less drastic and probably more typical -- the ongoing monitoring, analyzing and publicizing of media performance. This article concerns one such monitoring effort -- the monthly On Balance newsletter (hereafter cited as OB) produced by the National Media Archive, a division of the Fraser Institute.
Our analysis is based upon a reading of the first fourteen issues of OB, published between October 1988 and January 1990. Our critique focusses first on OB's understanding and application of its preferred methodological technique, content analysis. We argue not only that there are specific problems in application, but more importantly, OB makes claims for the straightforward objectivity of content analysis which cannot in principle be sustained; that is because the technique is dependent upon the construction of categories, which is necessarily an interpretive process. We then identify some fundamental conceptual and epistemological problems of the OB research. In particular, OB seems to share with some other media research an uncritical acceptance of the problematic notion of ``balance,'' a positivist assumption that the news can straightforwardly reflect the ``real,'' and a tendency to analyze the meanings and ``content'' of news without examining or theorizing the conditions of its production and reception. Finally, we examine OB's rhetorical mode of reporting results, a mode which sometimes appears to lay an inferential basis for a politicized interpretation of the news, one which often accords with the Fraser Institute's own ideological perspective.
Background: The Origins of On Balance
OB owes its existence to the Fraser Institute, a ``think tank'' founded in 1975 by right-wing economist Michael Walker, with backing from the giant forestry corporation, MacMillan Bloedel. Headquartered in Vancouver, the Institute had by the mid-1980s developed an annual budget of over $1 million, and a staff of eighteen, thanks to the backing of over 400 corporations and prominent Canadian conservatives like Peter Pocklington and Conrad Black (Fraser Institute, 1989). The Institute is openly pro-free enterprise and generally critical of trade unions and government economic intervention. In the words of its own annual report (1989, p. 8), its role is to ``redirect attention to the use of competitive markets as the best mechanism for responding to change.''
OB itself is produced by the National Media Archive (NMA), a division of the Fraser Institute founded in 1987 with a guaranteed five years of funding. With an annual budget of $200,000 from the Institute and from clients who subscribe to the newsletter, the NMA videorecords and transcribes major news and public affairs programs from the two main national English-language TV networks, CBC and CTV. In that respect, the NMA offers a valuable research service, not readily available elsewhere in Canada. On the other hand, unlike public archives, the NMA does not attempt to maintain professional archival standards of indexing and preservation. Instead, the NMA has made a name for itself primarily by conducting its own in-house analyses, and publicizing them through OB, which employs a popularly accessible writing style and is widely distributed to members of Canada's journalistic and business elites. In the first issue of OB (October 1988, p. 7), the NMA described the newsletter's role:
Each issue of On Balance discusses a different public policy issue and analyses it in terms of the positions represented, the people interviewed, the affiliation of the interviewees and the objectivity of the reporters involved in the story construction. This analysis is done using a standard communication research tool known as content analysis.
The first issue of OB, released during the 1988 federal election campaign, focussed on the highly topical subject of the free trade debate. OB claimed that media coverage, particularly on CBC television, was imbalanced against the free trade agreement. Highlighting the OB study, the Fraser Institute itself issued a press release (October 13, 1989) entitled ``CBC provides unbalanced coverage of free trade,'' stating that ``the news and public affairs programming of the [CBC] network has taken a position against the trade deal.''
In the midst of an election campaign revolving around the free trade issue, such a claim was particularly newsworthy. The OB study was publicized, largely uncritically, by a number of columnists, particularly those predisposed to accept the NMA's analysis. Supportive press commentary was penned, for example, by Dave Smith (Vancouver Sun, October 18, 1988), Douglas Fisher (Toronto Sun, October 19, 1988), and right-wing Alberta newsmagazine publisher Ted Byfield (Financial Post, November 7, 1988), amongst others. Over two years later, in a harshly critical presentation to the Canadian Radio-Television and Telecommunications Commission accusing the CBC of being anti-free trade, soon-to-be-appointed CBC director John Crispo called the OB study ``definitive'' (Michael Valpy, Globe and Mail, April 4, 1991).
Other organizations have likewise uncritically accepted and amplified the OB research. Thus, the Business Council of British Columbia (BCBC) released a special bulletin in January 1989, asking ``how balanced is reporting of industrial relations issues?'' It then answered this question on the basis of the December 1988 edition of OB:
All parties got bad press but some parties got more bad press than others. The [OB] report concludes that as a result of the enhanced attention paid to labour sources, management got the most unfavourable coverage. In the Globe and Mail, corporations received well over twice as much negative as positive coverage. CBC provided critical coverage of management nine times out of ten. (Business Council of British Columbia, 1989, p. 2)
The BCBC concluded its bulletin with a warning to management to be more pro-active in its relations with the media, based on the NMA's ``proof'' of inadequate media treatment of management.
The contentious and politically charged nature of OB's conclusions, combined with their apparent resonance within business and media circles, make it especially important to subject the OB research program to critical scrutiny. We begin with a basic underpinning of OB's credibility -- the claim that OB makes for the objectivity of its content analyses of news. There are two problematic aspects of this claim: first, OB's particular application of content analysis; and second, its general understanding of, and its claims for, the technique itself.
Content Analysis: Misapplying the Technique?
To be clear on our own position, two of the authors of this paper have themselves used the technique of content analysis extensively in previous research (e.g., Gilsdorf, 1979, 1980, and 1981, p. 55; Hackett, 1983, and 1991, Chap. 9). Employed carefully and self-consciously, it can be a useful means of summarizing or describing selected aspects of a large body of material. Especially when combined with other methodologies and analytical frameworks, it can also be an important tool in interpreting the potential meanings of media texts.
On the other hand, we must recognize that content analysis is limited to and by the choices researchers make about categories, and by the degree to which they select and treat the material systematically. Like other efforts to quantify and ``scientize'' human behavioural and communication processes, content analysis tends to mask the degree of interpretation present and to disguise the inevitable influence created by the intervention of the researcher. That is why it is important to apply the technique in a self-reflexive way, and to indicate how the researcher's assumptions might influence the results. Moreover, done in isolation, content analysis provides only limited knowledge about the coding and decoding process -- that is, about the preferred meanings embedded in the text, and the interpretive process of perceivers. As in all research that considers the coding process or analyzes text, great care must be taken with respect to inferences about audience interpretations or ``effects'': these cannot simply be ``read off'' from quantifiable descriptions of the ``manifest content.''
From our reading of the first year's editions of OB, there appear to be both interesting results and some serious problems with the use of the content analysis method. These problems include certain apparent contradictions in reported findings, a lack of clarity in key areas, an unwillingness to recognize the degree of interpretation present at several levels, and a failure to utilize content analysis in association with other analytical frameworks and methodologies.
Some of the problems can be summarized simply as a lack of rigour and consistency in conducting the research and reporting its findings. Take, for example, the question of sampling. In the edition on free trade, OB excluded both the Globe and Mail's Report on Business, and its editorials, arguing in the latter case that ``no equivalent could be found on television programmes'' (OB, October 1988, p. 3). Such exclusions almost certainly skewed the results away from finding media coverage supportive of free trade: The Report on Business emphasizes economic concerns and speaks to a business constituency, and the Globe had editorially endorsed free trade. Later OB studies, however, do include the Report on Business, with a significant impact on the reported results, and they make no mention that newspaper editorials are excluded from the sample. Possibly OB researchers have been simply inconsistent in their sampling decisions; alternatively, they may have sought to maximize the probability of finding ``imbalanced'' coverage.
A similar inconsistency in the sampled universe was evident with regard to the exclusion or inclusion of CBC programmes, and in the definition of the scope of political issues, which shapes the inclusion or exclusion of stories from samples. Thus, the OB study of health care coverage (February 1989, p. 3) excluded stories on specific illnesses or ethical issues from the sample, and included only those reports relating to ``the health care system in general.'' The reported finding that CBC network news does not give health care ``prominent attention'' (p. 1) in the news may well be in part an artifact of that sampling decision.
Moreover, from the information supplied in the OB reports, it is difficult for the reader to verify the accuracy and reliability of the coding process, even though OB (January 1989) has itself criticized media coverage of opinion polls for vagueness and insufficient methodological explanation. The reader was assured in the first issue of OB that a ``high level of inter-coder reliability was obtained,'' yet no percentage figure was supplied, nor was the type of reliability test identified. Moreover, there was no clear indication in the report itself of the consistent application of coding rules.
Similarly, the discrete coding unit, which appears to be the ``statement,'' is not clearly explained. For instance, to take the example of a reporter's narrative quoted in OB's health care issue (February 1989, p. 4), are five consecutive sentences from the same speaker elaborating a single theme coded as one statement or five? The quantitative results presented by OB, and their implications for perceived ``balance'' in news reporting, are highly dependent on the (inadequately explained) definition of a unit of analysis. Indeed, the selection of a ``statement'' extracted from a verbal transcript is far from being an obvious choice as the basic unit in the editorial decision-making process of television news, or in its construction of meanings. Other potential units of analysis which have been employed in other research have ranged from individual words, up to the entire news item or even newscast (Lichty and Bailey, 1978, p. 116), as well as the theme, and the visual shot or scene -- surely an essential dimension in television news. Yet, without further explanation or theorization to justify the ``statement'' as a unit of analysis, OB (October 1988, p. 1) makes this astonishing claim:
To ensure objectivity each news story is broken down to its constituent statements...By analysing each news item in terms of its integral statements, it is possible to perform completely objective analysis of the positions presented in the sense that any group of randomly selected researchers would come to the same conclusions if exposed to the statements that had been analyzed.
In addition to the inadequate explanation or justification of key methodological choices, the presentation of findings (however interesting they may sometimes be) is sometimes inaccurate, vague or misleading. Sometimes there are apparent contradictions. In one case, a pie chart contradicts the percentage reported in the text (OB, February 1989, p. 3). The OB edition on labour (December 1988, p. 4) decried the preponderance of union over management representatives as interviewees, yet in the specific instance of coverage of the nurses' strike (OB, February 1989, p. 5) doctors and government officials were interviewed far more often than were nurses or union representatives, especially on CBC. Surely this finding ought explicitly to qualify the implication from the earlier study that news favours unionized workers. Sometimes, reported results are simply confusing. In the same OB issue on health care (p. 4), OB first states that the CBC only twice mentioned user pay as a revenue option, then suggests that the ``CBC provided 9 in 10 statements in support of government funded'' health care, and finally states that ``What is significant is that CBC simply did not address the means by which health care in Canada is to be paid for.''
Other flaws in OB's reporting of results are evident. The proportion of statements which are ``neutral'' rather than ``favourable'' or ``unfavourable'' on an issue is often excluded or underemphasized, particularly when results are visually highlighted in charts and graphs, and especially in the early issues of OB. Thus, the finding that the vast majority of statements made by reporters on free trade were neutral statements of fact (in the case of CBC, over three-quarters) did not preclude OB (June 1989, p. 3) from asserting that ``Story content was two to one against the deal.'' Moreover, percentages are often reported without indicating the actual total of statements on which they are based (e.g., OB, November 1988, p. 2). If the N's are quite low, then conclusions about ``balance'' in coverage ought to be appropriately qualified. Percentage-based conclusions are especially troubling in light of OB's tendency (November 1988, p. 2) to report conclusions based on assertions like, ``Of all statements which could be classified...'' How many statements were thus excluded from categorization, and on what basis? Finally, findings are over-generalized to characterize whole media organizations on the basis of the particular programme studied; thus, the Fraser Institute press release of October 13, 1988, concerning the OB study of free trade reportage by the National and Journal, was entitled ``CBC provides unbalanced coverage of free trade.''
OB's application and interpretation of content analysis lacks rigour in other ways as well. There are attributions of the effects of media coverage that cannot be supported by the method employed, namely, content analysis. Thus, OB asserts in one issue (November 1988) that ``The way in which Free Trade has been reported by the media is having a serious impact on the way Canadians view the deal, political parties, and the party leaders.'' Likewise, OB sometimes apparently imputes motives on the basis of content analysis; for instance, OB (February 1989, p. 4) claimed that CBC was ``provoked'' to cover health issues by the Alberta nurses' strike. Such claims should at least be qualified. The determinants and impact of news cannot simply be ``read off'' from its apparent content, and indeed no communication research method enables us to understand the very complex process of news production and reception as easily as the OB reports sometimes imply.
A further dual difficulty in OB's application of content analysis is the failure either to formulate theoretically relevant hypotheses for testing, or to position its work in relation to other research on news production and content. These omissions lead OB to repeat commonplace observations or well-established findings in a tone of revelation -- that news privileges events over issues, the negative over the positive, entertainment over information, and so on. More crucially, they also allow OB to avoid dealing with previous research which would contradict or qualify its previous conclusions. One example concerns patterns of regional coverage in Canadian news. Thus OB (June 1982, p. 2) asserted that ``news from Ontario dominates the coverage, with nearly two-thirds of CBC coverage focusing on Ontario.'' This conclusion was based, however, on an assumption that news from the National Capital Region should be categorized as Ontarian. To consider all news stories originating from Parliament, the Supreme Court, or other federal institutions as ``focusing on Ontario'' is debatable, especially when other published studies specifically address this question. Lorimer's 1984 research on CBC radio found that a ``considerable proportion of Ontario sources were accounted for by the presence of the nation's capital in Ontario,'' and that when these were factored out, CBC coverage ``almost exactly follow[s] the relative populations of the provinces'' (Lorimer and McNulty, 1987, pp. 265-271). Other research (e.g., Kiefl, 1978) also qualifies the OB conclusions about the regional profile of news.