RBME System Introduction and Conducting a Readiness Assessment
Thanks to COP-MFDR management for organising this discussion forum and also Prof. Ray C. Rist for allowing his publication “Ten Steps to a Results-Based Monitoring and Evaluation System” to be used in this discussion.
While reading Prof. Rist’sbook and other discussion contributors comments over the first few weeks, I have picked up some points which I would like add and contribute my comments to the discussion.
I would welcome a feed back from Prof. Rist and other discussion group members.
“Learning by doing” and "Just do it”
Some assessment is essential at the initial stage however degree of detail may vary. Assessment not only at the beginning of developing the M&E system should be adopted as an on-going process since as environment and situation change, objective, scope and plan must be adjusted.
Please note, as on-going assessment is essential for the life of M&E system, the technique of assessment should also be as “learning by doing” as a better assessment will contribute to the success of M&E System. However, all care should be taken as too scrupulous assessment can become very distractive and discouraging.
"Just do it” is also vital as the stakeholders, particularly the top level management, especially in developing countries, are very impatient and are lookingfor the outcome very quickly. This particularly applies to politicians who are in power for only 4 years. The department directors general, who are under political pressure, could easily be rotated within 6 to 12months. With new top management, the whole assessment processcould start from the beginning.
A combination of"assessment", "Just do it" and "learning by doing" approaches shouldbe utilisedin developing countries, as this will satisfy requirements ofdonor organisation and country stakeholders. The ending result may not be as expected in developed country but would definitely achievean improvement and hence, provide a platform for M&E Systemsupport for continuing progress.
In early 1990 we were asked by the Planning Director of Livestock Development Department in Thailand to assist them in building M&E System where committee members were appointed from various stakeholders in the department, who had no practical experience in developing M&E System (or Information system).The committee was assessing and meeting for nearly two years, when the planning director decided to ask us to directly involve in the process. We proposed the funding required, based on our experience in the developed country where my firm and I have extensive experience in developing management information system.Our estimates were much below the actual requirement we had encountered, as capacity of the department was very limited, unlike in developed country.The department was trying to do their best to assist us by providing extra funding butstill it was limited and nowhere near enough. We took a very hard decision to fund our resource to push ahead with the project that commenced in 1992 and completed in 2000 at which point we felt that the department has sufficient capacity to sustain the system into the future.With a proper assessment, this situation would be much easier to handle but in developing country that was unsuitable at the time.
The experience gained was used by our firm to develop the standard Result-Based Monitoring and Evaluation software and process for the subsequent government organisations in developing countries, together withvarious capacity building programs.
Concluding, a clear expected outcome of any plan/project is a normal requirement of stakeholders in private and public sectors as all funding allocation must be accountable. However, discretion should be used to balance the approach between proper assessments, learning by doing and ‘just do it’.
…if a country or organization or ministry is actually ready to start moving towards MFDR.
Recognising who is ready or who is notcan bevery complex as we have encountered when presentingour M&E systemto over 50 government organisations in Thailand, to various levels of stakeholders in the organisation. It is important that the interest is from all levels of the organisation or ministry. Our presentations and workshops were conducted to various levels of management or stakeholders. Each of the stakeholders had different interestand required differentincentive in pursing the M&E System.For example, Policy/Planning department was interested in themeasurement of activity, output and outcome. Financial department was interested in the financial input or the budget of the policy/project. Local politicians were interested in the policy/project that is affecting their electorate. Personnel and Human Resources department was interested in organisation’s and staff performance, etc.
Jump start to the outcome oriented M&E System
Since input and output system is providing the feedback to the expected outcome, even if wesuccessfully implement the outcome oriented M&E System, butthe input-output system is not established, the outcome information may not be much used, especially if we are trying to use the outcome to improve the subsequent year policy/development plan. Some government organisations would start with input and output and simple measurement but emphasise all the stakeholder requirements,then progressed to outcome system and, eventually, delivering the organisations impact nationally.The entire process could take around 5-6 years. This appeared to work well as the organisation had minimal resources and limited capacity. Later, with larger organisation, with larger budget and support of political system, we have attempted the input-output but also back-tracked the entire development plan of the past three years. This process took over a year, when the government officials realised that simply input and output alone have contributed to a lot of progress already.
It is worth noting thatoutcome oriented M&E System is amodel from developed countries where all the essential input and output components have already been established over the years.
The modular approach should be used for the M&E System where modules could be implemented as needed by organisation and situation and slowly progress to cover the entire organisation’s result-based monitoring and evaluation system.
It is important to note that the M&E system is the on-going progress measurement in the organisation. The original design of M&E system should allow for this modular approach where better approach and new technologies can be modularised and added on at later stage.
Difficult to implement at first thought
All implementation activity should allow for unforseen circumstances - especially in the environment previously not exposed to information system development and reporting - i.e.: stakeholder consensus, power failure. In well developed information system environment in developed countries 10-20% of oversight would be added to plan implementation activity but for developing countries this percentage is much higher.
Building a results-based Monitoring and Evaluation System
In building M&E System fordeveloping countrieslessons from similar undertakings in other developing country could be closely related as the country development approaches are similar. This could also be compoundedby closely related cultural, political and civil society systems, particularly in the same geographic location.
New Challenges in Public Sector Management
In most emerging countries, measurable budget utilisation is still the only possible means of monitoring and evaluation at the national level for most government departments due to lack of capacity and systems experience. We may find some champions in some departments or local governments who are more advanced than othersin using system to assist in monitoring and evaluation, but there are not many of them.
Some government departments may have pursued M&E System since early 1990 and over the years, after various lessons learned, a system has been developed suitable to their capacity and environment, but national level Monitoring and Evaluation has not reaped the benefit as evident from the inability to pass on the experience to others that can be used for policy/plan adjustment.
The challenge is to have the M&E system implemented at the same time when the development plan is implemented, where all indicators (i.e. input, activity, output and outcome) can be measured. In real life often this is not the case. Bangkok Metropolitan Administration works on 5 years development plan. When the M&E system was implemented, the development plan has already been in its third year. To allow the development plan to be fully measured, indicator results needed to be translated into the M&E system format. This task proved to be very difficult as the result reports from the first 3 years were in descriptive format and inconsistent, and needed to be converted into a format measurable by the system. External resources needed to be hired for analysis, formatting and entering policy/project information. The project/district officers weren't available to provide an advice on the accuracy of the information. Eventually all results and target indicator information was entered but was unable to provide meaningful measurable result as monitoring and evaluation process for the first 3 years wasmanagedwithvery limitedcapacity.
Entered information needed to be selected (certain sector) to be manually corrected to provide some meaningful measurement to stakeholders to allow demonstration for comparison with previously set up measurement benchmark for remainder of the development plan.
Back-tracking of measurement information could be useful, but simply to create benchmark, and continued monitoring and evaluation of selected project or sector should be used for entering target and actual result only. However, if possible, an attempt should be made for entering the entire plan to obtain a guide for the future measurements.
There were numerous discussions and capacity building training programs for setting up indicators and M&E process for various stakeholders. However, to train all stakeholders to start real result measurement is an overwhelming task and suitable system tools and process to assist are not easily found. In monitoring and evaluation, a number of stakeholders (i.e.: Ministry and local government) need to be involved and clear continuing leadership is greatly required at the early stage.
International and External Initiatives and Forces for Change
It is a challenge for developing countries to start at the even most elementary level of measurement as this requires innovative approach because simply copying from the developed country will not provide expected outcome as developing countries’ environment and culture are different. An expert who has built experience in the developed country may find directly applying this experience will become a long term – on-going process without much progress.
MDG targets that have been translated into a set of indicators, where this set of indicators may be relevant to policy and/or decision maker at macro level, may not be shared or be understood by other stakeholders, at the middle- and operational or grass-root level. Some form of indicator target should be setup at the initial stage of M&E System that is most relevant to each stake holder. Innovative M&E System should have the facility to allow stakeholders to harmonise their differences while results-based approach is in progress to establish common indicators of organisation, or enable establishing a link from one group of indicators to another. Eventually, as a result of support from different stake holders and facilities in the M&E System, an organisation will derive common workable indicators.
As an example - At Bangkok Metropolitan Administration as part of transformation to the result based M&E management approach, the external consultant was commissioned to establish the indicators at the operational (project/district) level. However, these indicators were in disagreement with policy maker at macro level. Ministry of Interior supervising the Bangkok Metropolitan Administration was also establishing the policy level indicators from the high level, however these indicators were not aligned with the operational level (Project/District officers). M&E System should have the facility to allow stakeholders to harmonise their differences.
Choosing Indicators
All stakeholders may not share the same thought on indicator. Standardising their thought may not be easily concluded because usually discussions are mainly either based on theory or on existing work process. The M&E System, as well providing various monitoring and evaluation reporting, must facilitate the process of continuing harmonising of all stakeholder perspective indicators in conjunction with experience gained from using the system (e.g. : Providing report to allow each indicator and result to be compared.) The approach of learning by doing should be used as otherwise this process of choosing indicators could take a very long time.
Most government departments in Thailand have gone through a lengthy discussion to establish indicators but the real measurement has not yet been realised.
Poverty Reductionthrough debt relief program – as a prerequisitefor this M&E capacity an innovative M&E System approach should be required to be usedor applied to qualify for this program.
The government of Thailand(not taking part in this program) has implemented poverty reduction policy where public funds were distributed directly to the grass root level with emphasis to improve local economy. However, those funds missed the originally intended recipients due to inconsistencies with supervision and lack of M&E System. And in cases where they did reach the originally intended recipients,funds provided were misused by the recipients instead ofinvesting intolocal community self businessscheme. Establishing M&E System has become urgent to allow furtherassistance accurately by providing information which can be used as a guide / road map.
Government downsizes and Reform
Recently, through elected politicians, there is a major push for the government downsizing, reform and decentralisation while, at the same time, maintaining and improving service delivery. The political objective and will by the government are quite obvious but lack of capacity and M&E system hamper the progress of this reform. However, continuing reform and search for M&E tools to assist this reform could result in drastic improvement in the public sector system. Incentive should be given to civil servants; the suitable result based M&E system would be needed more than ever to allow the momentum to continue as effective tools provide feedback to the government.
Traditional Implementation-Focuses and Results-Based M&E systems
For thedeveloping countrynot fully familiar with the traditional implementation-focus practice, pursuing the results-based M&E Systemmaycause a lot of confusion.The implementation-focus forms the foundation to progress at the results-based M&E system.
The implementation-focus could be at the project unit level and leave the results-based M&E system to be handled by different processes by means of feedback information to form part of theoutcome information for impact assessment.
Outcomeinformation survey and feedback system could be implemented and, once implementation-focus has stabilised, then the outcome and result-based monitoring could be introduced.
On a yearly basis, Livestock department, using information system to record, conducted survey outcome information amongst6 million farmers on the number of animal stock. This information would be used as baseline informationin various locations (i.e.: Regions,Provinces)to determine theeffect ofthe implemented policy/plan and projects that were earlier monitored and evaluated by the information system.
Building Reporting system
For the stakeholder to quickly get familiar with the system reports and speed up the process of building institutional capacity, the M&E system should be able to provide system reports during the design, implementation and on-going usage of the M&E System using the system monitoring and evaluation informationof the existing stakeholder ‘s policy/projects.
Measuring Inputs, Activities, Outputs, Outcome andImpact
It isessential for small unit (project/district office) within organisation to be able tocommence measuring themselveson the input, activities and output. This self measurement wouldin turn,make them accountable for their own actions.Once input, activities and output are measured, theoutcome measurement could also addvalue to the result-based monitoring andevaluation. It is important that thesmaller unit measurement isperformed on regularbasis (i.e. monthly), the policy and organisation’s measurement are performed in less frequent intervals (i.e. quarterly or half yearly).These frequentmeasurements would allowcorrection and adjustment to ensure expectedresults target are achieved.
All self measurement informationusing M&E system must immediately be passed on to the policy and organisation level to allow stake holders’ information sharing.
Government sharing information
It is normal for government departments in developing countries not willing to share information but use it only for their own gain (e.g.: to prove their performance success - as there achievement incentives were offered).However, M&E Information system should have the facility for each government department to hold their information within their department (i.e.: project/district organisations) environment. But when entering their performance information this information would be transferred to the central head office as part of the M&E system process (i.e.: Planning or Finance Department/Ministry) allowing it to be used for the entire organisation’s performance measurement and policy monitoring and adjustment. The central information could be allowed to be shared with some sensitivity control without creating a sense of intrusion from the government department. Sense of intrusion would create resistance to share.
Please note in developing countries, sharing knowledge and information is not well established within the culture, unlike in developed countries where practice of sharing information had been developed over many years while protecting and facilitating all stakeholders concerns (i.e.: intellectual rights, information act, etc)
M&E Data Management
M&E Systemshould provide the facility for the government to gradually collect measurement information (i.e.: Input and Activity) and gradually increase as skill and experience are gained. All collectedinformationshould becategorised and grouped so it could beretrieved and summarised as needed.Institutional buildingin the reporting informationstructure and usageof the M&E System must be established toallow continuing usage and improvement of information structure and reporting.