29
CFSC Research
Tracking the Impact of an Advocacy Paper
Sponsored by the International Development Research Centre, with support from the Communication Initiative, this study by graduate students at Canada’s University of Guelph lays out a methodology for evaluating the impact, effectiveness, dissemination of, and potential applications for, an advocacy paper. It also suggests how to improve the document itself. The authors are Cassie Barker, Heidi Braun, Marshall Gallardo Castaneda, Franklin Kutuadu, Richard Marfo, Pete Sykanda and Rosana Vallejos
1.0 Introduction
This report documents the activities and outcomes of three months of collaboration and evaluation surrounding the advocacy document Communication for Development: A Medium for Innovation in Natural Resource Management. This has been an important exercise for all involved, as it served to reinforce the importance of such publications, and the efforts made by communication advocates in the field of Natural Resource Management (NRM) to emphasize its role and necessity. The experience also stressed some of the limitations of print and electronic media, and the communication of these messages, when measuring effectiveness and utility.
The report structure follows the process of this undertaking, from the background of the document and its evaluation; the activities completed in disseminating and generating feedback on the document; the findings and analysis of both our activities and the feedback; and our recommendations on future dissemination and feedback. This section examines the background and purpose of both the document and this project.
1.1 Booklet Background
The document Communication for Development: A Medium for Innovation in Natural Resource Management was written by Ricardo Ramírez and Wendy Quarry, and published in early 2004 by the International Development Research Centre (IDRC) and the Food and Agriculture Organization (FAO) of the United Nations.
Ramírez and Quarry have been collaborating with the Communication for Development Group at FAO and IDRC to update a useful but outdated FAO communication advocacy publication, such as Communication: A key to human development (1994). Wendy Quarry explains:
Given that the document is at least 10 years old and that there have been many changes in thinking around communication since its production, we felt we needed something more up to date for the same purpose. That thinking and a lot of discussion with IDRC and FAO has led to the new version.
Communication for Development is intended to be the first of two updated publications focused on the function and planning of communication in Natural Resource Management (NRM). Its design started with an audience research phase, followed by inputs from several communication specialists, a peer review, a “field test”, and many rounds of editing and revising.
1.2 Booklet Purpose
This publication is an introductory resource for decision-makers in NRM, intended to be used by communication advocates as a support to a face-to-face discussion with NRM managers and decision-makers. As stated in the Communication for Development Foreword:
This document presents, through stories and examples, the experience of many people and projects worldwide where communication methods and approaches have been applied to address natural resource management problems. It is a joint effort between IDRC and FAO to help decision-makers, planners and practitioners understand why and how communication for development activities can support sustainable natural resource management and rural development efforts.
1.3 Initial Dissemination and Feedback
At the time of its publication, no strategy was in place for dissemination to key Communication for Development advocates, nor a feedback mechanism on its effectiveness as an advocacy tool. Initially, 2000 copies were printed; over 40 hard copies were sent to contributors. Hard copies were available at the United Nations Roundtable on Communication for Development, September 6-9, 2004, in Rome, and at the Snowden Symposium on Communication for Social and Environmental Change at the University of Guelph, October 5-6, 2004.
The need for dissemination and follow-up was identified as important by the authors and by Guy Bessette of IDRC during the Rome Roundtable. They all shared concerns as to its limited distribution and usage in the field. Guy Bessette, Senior Program Specialist at IDRC, and Ricardo Ramírez identified listservs of NRM decision-makers for dissemination, and explained the possibility of facilitating a discussion about the document through an electronic forum (e-forum) with the Communication Initiative (see Text Box 2).
1.4 Project Justification
The School of Environmental Design and Rural Development and IRDC signed a contract with an understanding that a group of graduate students in Ricardo Ramírez’s REXT*6311, Extension Theory and Methods course would take on the task, which is elaborated below.
2.0 Project Activities
The following section outlines our group’s activities for the dissemination of the IDRC/FAO publication and the operation and management of the e-forum discussion.
The activities for this project were a direct result of the Terms of Reference outlined in our contract (No. 109295):
a) Make the document known through a number of listservs and facilitate a debate around it.
b) Design and implement a communication study to ascertain several questions such as:
· How is this document being used/likely to be used?
· Is it reaching/likely to reach its intended audience?
· Is it having an impact/likely to have an impact in the form of decision-makers actively seeking more information on issues or allocating resources to communication efforts?
· Is it being used/likely to be used in ways that were not foreseen?
· What are the lessons we can derive in terms of the design, publication and distribution of advocacy papers?
c) Propose a communication strategy to complete the actual distribution list for the publication and other publications in the same Devcom-ENRM field.
2.1 Dissemination
Our dissemination strategy for the publication consisted of the following steps:
· Identification of the listservs to target for dissemination. The listservs we targeted were those that served the interests of communication specialists and NRM decision-makers, the intended audience of the publication (see Appendix 1 for information regarding the listserv specialization).
· Contacting the Listservs. Once a list of listservs had been assembled, they were contacted to gain consent.
· Dissemination to the Listservs. Our goal with dissemination was to make the publication widely accessible in both the Communication for Development and NRM sector. Though the message was sent to several listservs, we were unable to obtain confirmation of how many actually distributed the message to their members.
We placed a great deal of emphasis and time on the wording and format of the dissemination letter in order to entice as many people as possible. In addition to dissemination, we also took the opportunity to begin the process of advertising for the upcoming e-forum. The dissemination letter included a link to the publication from the FAO server, as well as directions to the e-forum registration.
The first dissemination message was sent January 28, 2005 to the World Bank e-forum on Communication for Development. A message to the listservs and publication contributors followed on February 18, 2005. (Refer to sections 3.2 and 4.2.1 for further information regarding the e-forum.)
2.2 The Process and Management of the e-forum
The process and management of our e-forum was as follows:
· Invitations: An invitation to participate in our e-forum was sent out to at the same time as the document was distributed (see Appendix 2). A personalized invitation was also sent out to those people who had received a hard copy of the publication at the time of printing (see Appendix 3). (See Appendix 4 for a list of those contacted with a personal invitation to participate in the e-forum.) Unfortunately, a number of these people could not be contacted as their e-mail addresses were no longer valid.
· E-forum Timeline: The e-forum began on February 28, 2005, and continued until March 16, 2005; a total of 17 days.
· Generating Discussion: Throughout the duration of the e-forum, we worked to stimulate discussion regarding this publication; we asked questions aimed at encouraging debate and discussion among NRM practitioners regarding the impact and use of communication in NRM. The questions asked of the participants in the e-forum were directed by those found in the terms of reference of our contract. These questions were designed to be direct and focused in order to solicit responses that were specific to this publication. The questions were posted in a particular sequence in order to solicit focused responses and to create a logical flow for the discussion.
Our group met regularly to discuss the progress of the e-forum, discuss recent postings, develop additional questions and decide on any other actions that needed to be taken. Summaries of the main points that arose from e-forum contributions were provided for participants at the end of week one and at the closing of the e-forum. Two of our group members served as moderators throughout the discussion, posting introduction messages (see Appendix 5) and summaries, presenting our questions, and facilitating the debate. As we approached the original closing date for the e-forum on March 11, 2005, we made the decision to extend the discussion for an extra five days to allow for further discussion and contributions.
3.0 Findings
The following are the results of our efforts to disseminate the document and invite participation in an e-forum discussion. A summary of the contributions and suggestions that were put forth during the on-line discussion is also included (analysis of these findings follow in section 4.0).
3.1 Dissemination of the publication
The booklet was widely publicized through selected listservs[i], the Communication Initiative Web site, the Drum Beat newsletter and the World Bank e-forum on communication.
Although it is difficult to conclude exactly how many people may have accessed the document as a result of the dissemination activities, the hit counter on the FAO site where the document is posted indicates that the number of visitors increased dramatically. Prior to dissemination, a total of 29 views and 22 visits were logged by the FAO server. (View is the number of times the page has been accessed. A visit is different from a view in that the visit counts only unique IP addresses that access the page. One might view the page three times, which would be tracked as 1 visit, 3 views.)
In February, following the dissemination and invitation efforts, 320 views and 198 visits were recorded. This 11-fold increase in views and 9-fold increase in the number of visits demonstrates a clear increase in interest, which can be attributed to the promotional activities carried out by our group.
3.2 Participation in the e-forum
At the start of the e-forum on February 28, 37 participants had registered. Participation grew minimally throughout the 17-day discussion and the final count of participants reached 42 on March 16, the closing date of the e-forum (see appendix 6 for a list of e-forum participants).
Overall, registration numbers were lower than we had hoped for considering the number of listservs contacted and their extensive distribution networks, and certainly low in comparison to the World Bank e-forum on Communication for Development, with 711 registered participants. This consequently had an impact on the number of contributions made to the e-forum. Throughout the 17-day on-line discussion, a total of 23 posts were made to the e-forum site. The posts included:
· Thirteen responses and comments relating to the e-forum questions, made by nine participants.
· Two informational posts related to another resource material of interest to participants.
· Eight contributions from the e-forum moderators including a welcome message, questions, summaries and concluding remarks.
These results indicate:
· A 21 percent participation rate (nine participants out of 42 contributing responses to the questions).
· Out of the nine respondents, three posted more than one contribution.
Based on an evaluation of e-forum participation by Piotr Mazurkiewicz (from the Development Communication Division of the World Bank) who has organized over ten e-discussions, he affirmed that a typical e-forum sees a 15-20 percent participation rate. Participation in our e-forum was therefore on par with this trend.
It was observed that the majority of participants who made contributions to the e-forum are specialists in the field of Communication for Development, though many also have a background or experience in the domain of NRM. The implication of this is further discussed in section 4.2.1.
Overall, we felt that the e-forum attracted a small group of participants and as a result, the number of contributions was few, relative to our expectations and in comparison to the World Bank e-forum. In spite of this, the quality of responses was quite encouraging and several insightful comments were generated.
3.3 Contributions to the e-forum
As outlined in section 2.2, the format of the e-forum worked to stimulate debate and discussion by posting specific questions to be addressed. The questions asked (see box 4 in section 2.2) addressed the following themes:
· Usefulness of the document for reaching policy makers.
· Impact of the document.
· Limitations of the publication.
· Suggestions for improvement and alternative strategies.
What follows is a synopsis of the main ideas and common issues expressed in the participants’ contributions. A selection of direct quotes taken from the e-forum responses has also been included to allow the participants’ words to underscore and add depth to the summarized interpretation of these findings. The full postings and archives from the e-forum can be viewed at: http://www.comminit.com/discussion.html?action=viewtopic&topicid
3.3.1 Usefulness of the document for reaching policy makers
The respondents felt that the document presents valuable information, experiences and best practices relevant to a discussion on communication for development. The issues are presented in a clear and simple format, and key concepts that are useful to development workers, researchers and communication practitioners are highlighted throughout. Most of our respondents felt that the content provides excellent references, stories and perspectives that are essential for development communication. The document constitutes an important tool to complement or perhaps replace the old FAO document Communication: a key to human development by Colin Fraser and Jonathan Villet.