A Joint WBI/CIPE Evaluation on a Think Tank Conference

A Joint Evaluation of a Conference on Think Tanks Sponsored by

WBI and CIPE

Harare, Zimbabwe

March 8-10, 1999

______

Laurence Colinet

Geoffrey Geurts (CIPE)

Adrian Hadorn

Erik C. Johnson (CIPE)

WBI Evaluation Studies

Number ES99-35

World Bank Institute

The World Bank

Washington, D.C.

Copyright © 1999

The International Bank for Reconstruction

and Development/The World Bank

1818 H Street, N.W.

Washington, D.C. 20433, U.S.A.

The World Bank enjoys copyright under protocol 2 of the Universal Copyright Convention. This material may nonetheless be copied for research, educational, or scholarly purposes only in the member countries of The World Bank. Material in this series is subject to revision. The findings, interpretations, and conclusions expressed in this document are entirely those of the author(s) and should not be attributed in any manner to The World Bank, to its affiliated organizations, or the members of its Board of Directors or the countries they represent. If this is reproduced or translated, WBI would appreciate a copy.

Table of Contents

Executive Summary i

Background and Context 1

Evaluation Design and Methods 3

Conference Objectives 4

Participants’ Pre-Conference Expectations 5

Table 2. Participants’ Levels of Interest at the Beginning of the Conference 6

Overall Achievement of Conference Objectives 8

Usefulness of the Conference as a Learning Event 13

Conclusions, Analysis, and Recommendations 14

[ANNEX 1] 19

[ANNEX 2] 21

Executive Summary

Introduction

Since November 1997, the World Bank Institute (WBI) and the Center for International Private Enterprise (CIPE) have co-sponsored four regional capacity building workshops for policy institutes, or think tanks. Events have been held in Cairo, Egypt (November 1997); Moscow, Russia (September 1998); Beirut, Lebanon (February 1999); and Harare, Zimbabwe (March 1999). A worldwide conference of think tanks is currently being planned for December 1999. Regional events are also being considered in Latin America and South Asia.

WBI and CIPE have also jointly evaluated the workshops, in order to derive lessons learned that could help improve future programs. The preliminary findings of the Beirut evaluation, for example, were communicated to the program manager responsible for the Harare event, held one month later.

The evaluation design for the Harare conference was tailored to fit the specific context. The evaluation was based on the results of one pre- and one post-conference questionnaire. Given the interest of WBI and CIPE in forging long-term partnerships with think tanks, the pre-conference questionnaire targeted the single issue of the expectations of the participants.

Workshop Objectives

The agenda of the Harare workshop was organized by WBI and CIPE around six objectives and eleven presentations. The lectures presented in Harare were different from those presented in Beirut. For example, institutional themes were more developed in Harare. However, the two conferences had some similar objectives and topics. For purposes of comparison, the objectives and presentations in Harare were combined into the same three categories used to analyze the Beirut event:

a)  Three of the workshop objectives related to information sharing among think tanks, in order to:

Ø  help think tanks in the region benefit from each other’s experience;

Ø  guide the World Bank in its efforts to build partnerships with knowledge institutions in Africa; and

Ø  stimulate the development of new programs and products.

These objectives were to be reached through face-to-face networking, as none of the formal presentations addressed information issues.

b)  One objective related to increasing the institutional capacity of think tanks in key areas of their activities. There were seven presentations on this issue: the state of African think tanks; the pros and cons of different think tank structures; management and staffing decisions; ways to popularize policy debate; designing media strategies; influencing legislative and executive bodies; and strategic options for financial sustainability.

c)  Two objectives related to policy issues: increase understanding of the role of policy institutes in civil society, and help think tanks contribute to the policy process. There were presentations on these four issues: the main obstacles to African growth; enhancing government effectiveness through budget reform; holding government accountable through fiscal transparency; and linking democracy with good governance.

At the beginning of the event, participants were asked to assess the level of interest of their organizations in the conference objectives and lectures.

Like participants in Beirut, the think tanks in Harare showed a keen interest in the objective of increasing institutional capacity. The most popular lecture in both conferences was strategic options for financial sustainability.

Among the objectives related to information sharing, the sharing of experience was most attractive to the Harare participants. This objective had also sparked the interest of participants of the Beirut conference. Unlike Beirut, however, the development of new products and programs ranked low among Harare participants.

Three policy topics were addressed in both conferences: regional constraints to economic growth, budget reform, and fiscal transparency. Participants in Harare were more interested in these topics than were those in Beirut.

The findings from both conferences also show that the practical interests of participants sometimes differ from their intellectual interests. For example, participants in Beirut were interested in lectures on the development of a worldwide network, but did not identify expansion of the regional network as a key objective for their organizations. Similarly, participants in Harare were most interested in the presentations on policy issues, but expected to take away from the conference mainly capacity building and information sharing skills.

At the end of the conference, participants were asked to assess the extent to which each conference objective was met. Overall, they perceived the conference as useful. As in Beirut, Harare participants reported that the conference was successful in terms of capacity building, information sharing, and face-to-face networking. Also as in Beirut, time management and the agenda design were identified as the weakest points of the event.

Participants were also asked to self-report their increase in knowledge in regard to the topics addressed in the conference. In general, the ratings were higher for Harare than for Beirut. As the population attending the two conferences and the agenda of the meetings differed, it is not possible to conclude that the Harare conference was of a higher quality than the Beirut conference. However, the figures in the report could serve as a benchmark for future events of the Global Development Network in these two regions.

As was the case for the Beirut conference, participants in Harare perceived that they gained more knowledge of institutional capacity than of policy issues. The self-reported increase in knowledge for every policy topic was below the WBI benchmark of 20 percent, while it exceeded the WBI objective for each capacity building theme. As in Beirut, participants at the beginning of the conference considered themselves more knowledgeable about policy topics than about institutional capacity building themes. This finding might explain why participants felt they had learned less about policy issues than about capacity building. WBI might improve this score in future conferences by addressing policy topics with which participants are less familiar, and/or explore more appropriate ways to present policy issues.

Recommendations

The lessons from the Beirut and the Harare conferences can aid in the preparation of future regional conferences and the worldwide conference planned for December 1999. The evaluation team makes the following recommendations:

a)  Make use of pre-conference surveys to learn about participants’ training needs: Use the survey undertaken for the worldwide conference to ask participants which capacity building or policy topics are of the greatest interest to them. The same could be done for future regional events. Similarly, the call for papers for the worldwide conference could be used to find out not only what papers participants would like to present, but what topics they would like to hear presented. If participants express a strong desire for a particular topic, there is a high likelihood that they perceive themselves as needing more knowledge in that area.

b)  Form presentation format working groups for the worldwide and regional conferences to discuss the optimal format for maximizing the effectiveness of capacity building and policy issue presentations. The objectives underlying presentation of policy issues versus capacity building issues are quite different. In light of the fact that the Beirut and Harare participants perceived themselves as more knowledgeable in policy issues, perhaps a different format of communicating this information should be utilized, such as a structured case study or best practice format. For the capacity building topics, a more rigorous training format with visual aids, working groups, and interactive discussions could be used to try to achieve even better participant satisfaction scores. The working group could also look for ways to address time management issues at the conferences.

c)  Develop a plan to help African think tanks take advantage of their network of contacts. Some 85 percent of Harare participants reported that the workshop helped to create or strengthen their network of contacts. However, several participants suggested some kind of follow-up of the conference recommendations. WBI should think of strategies to build on this face-to-face networking in order to promote the sharing of information on policy issues of mutual concern, to exchange information and experiences on the development of new programs and products, and to build partnerships. For example, the idea of creating a virtual network such as the one that exists in the MENA region could be explored.

d)  Consideration should be given to repeating the set of evaluation questions (as shown in Table 1 below) at future Global Development Network events. The results of each conference could serve as a benchmark for the next regional or worldwide event.

Table 1. Question 3 of the Post-conference Questionnaire:

Results from Harare and Beirut

1 Arithmetic average of all respondents to the question on a scale where “1”=”minimum and “5” = maximum.

2 Percentage of respondents who answered with a “1” or a “2” out of all respondents to the question.

3 Percentage of respondents who answered with a ‘4” or a “5” out of all respondents to the question.

21

Harare, Zimbabwe

Background and Context

Since November 1997, the World Bank Institute (WBI) and the Center for International Private Enterprise have co-sponsored four regional capacity building workshops for policy institutes (think tanks). Events have been held in Cairo, Egypt (November 1997); Moscow, Russia (September 1998); Beirut, Lebanon (February 1999); and Harare, Zimbabwe (March 1999). A worldwide conference of think tanks is currently being planned for December 1999. Regional events are also being considered in Latin America and South Asia.

The decision by CIPE and the World Bank to jointly sponsor these workshops was based on a shared interest in strengthening the institutional capacities of policy institutes so that they may better fulfill their role in civil society.

In Africa, WBI and CIPE were also able to enlist the support of two additional organizations as workshop sponsors, each of which works in cooperation with its own network of think tanks. The African Capacity Building Foundation (ACBF) in Harare was established in 1991 to build sustainable human and institutional capacity in economic policy analysis and development management in Sub-Saharan Africa. The Secretariat for Institutional Support for Economic Research in Africa (SISERA) is a multi-donor initiative dedicated to strengthening the institutional framework for economic research in Africa, by focusing on synergies between research and training and on the provision of advice on economic policymaking.

One significant element of the partnership between WBI and CIPE has been the joint evaluation of the workshops. This collaborative effort was initiated as a means of establishing a process by which the lessons learned from each workshop could be used to improve future programs. Findings are communicated through the dissemination of evaluation reports and through meetings between evaluators and program managers from WBI and CIPE. For example, the preliminary findings of the Beirut evaluation, held one month earlier, were communicated to the program manager responsible for the Harare event.

A number of steps were taken during the design and planning stages of the Harare event to ensure that the weaknesses of the Beirut event would not be repeated. The following list of findings from Beirut were communicated to the Harare organizers:

a)  Conference objectives were too ambitious. In response to this information, the objectives presented to the Harare participants were condensed. However, that event came too soon after the Beirut event for the agenda to be redesigned.

b)  Conference discussants did not act as discussants, and moderators did not effectively control the discussion. In response to this information, a paper describing their respective roles was distributed to each speaker, moderator, and discussant.

c)  Speakers did not utilize a diversity of presentation formats, and the use of visual aids was especially weak. In response to this information, speakers were asked to diversify their presentation styles by using flip charts, PowerPoint software, and overhead transparencies.

d)  Participants could have been better prepared to contribute to discussions if they had received conference papers in advance. In response to this information, two of the main presentations were sent to participants in advance.

Evaluation Design and Methods

The evaluation for the Harare workshop was based on the results of one pre- and one post-workshop questionnaire. The questionnaires offered a mix of closed quantitative and open qualitative questions. A five-point Likert scale, ranging from 1 = minimum to 5 = maximum, was used for answering quantitative questions. Both questionnaires are attached as annexes to this report.