1
DRAFT
DRAFT
Good Practice Guidelines Package
for
Workshop
10 - 11 May 2005
Wellington
SPEaR Working Party (D Archer/D Brown, R Good, L Hoather, P Honeybone, J Tauri, S Walker (Convenor). 3 May 2005
CONTENTS
Page
1.0OUTLINE3
2.0CONTEXT3
2.1Role of SPEaR3
2.2Operating Environment3
2.3Social Context4
2.4Good Practice: An Introduction4
2.5Good Practice: Statement of Purpose 5
2.6Policy Cycle5
2.7Project Cycle6
3.0DEVELOPMENT PROCESS8
3.1Process to April 20058
3.2May 2005 Workshop Objectives9
3.3Process Proposed Post May Workshop10
3.4Workshop Programme10
4.0THE DRAFT GUIDELINES12
4.1Why Choose a Principles Framework?12
4.2The Layers/Tiered of information Approach12
4.3The Principles13
4.3.1Respect13
4.3.2Integrity14
4.3.3Responsiveness15
4.3.4Competency15
4.3.5Reciprocity16
4.4Draft Guidelines Development to Date17
4.4.1Research and Evaluation Contracting18
4.4.2Research and Evaluation Ethics23
4.4.3Research and Evaluation involving Maori26
4.4.4Research and Evaluation involving Pasifika29
4.5Areas Identified for Development33
AAPPENDICES (separate document)
A1SPEaR Working Example 1 & 2
A2Level Three Information Samples
A2.1Sample of a Letter: Protection of Contractee Information
A2.2Sample Acceptance Letter – Agency to Potential Contractee
A2.3Sample Letter of Agreement For [NAME OF RESEARCH CONTRACT]
A2.4Sample CONTRACT PERFORMANCE STANDARDS
A2.5Agreement for the Purchase and Provision of ?????Research
A2.6Discussion Paper on Intellectual Property
(Carl Davidson, NoDoubt Research 2004)
A2.7Pre-Design Phase Funding Bid Example
GOOD PRACTICE GUIDELINES
PACKAGE for 10 - 11 May 2005 Workshop
1.0OUTLINE
This workshop is designed to test and further develop a good practice[1]guideline ‘product’ to assist in setting standards for good practice in New Zealand social policy research and evaluation. The ‘product’ will be web-based so it is open and accessible to all parties. It should provide advice relevant to all stakeholders in social policy research and evaluation: public servants in management, policy, research and evaluation functions, academic researchers, private sector and third sector researchers and evaluators and practioners in all sectors.
2.0CONTEXT
2.1Role of SPEaR
The Social Policy Evaluation and Research Committee (SPEaR) is one of the initiatives arising from the Improving the Knowledge Base (IKB) project in 2001. SPEaR has been established by government, with a mandate to “oversee the governments social policy research and evaluation purchase”[2]. In undertaking this role, SPEaR has a number of specified functions, including
- to act as a vehicle for gathering agency views on sectoral research capacity issues;
- to sponsor initiatives aimed an improving social policy research capability (including agency research practice); and
- to promote the utilisation of “best practice” approaches, tools and techniques through development (where necessary) and/or dissemination.
The ‘good practice’ programme aims to develop the base information that will enable the promotion listed in the SPEaR Terms of Reference.
2.2Operating Environment
The New Zealand social policy research and evaluation environment provides a unique context for setting standards to provide a framework for quality practice. While researchers and evaluators often have disciplinary Codes of Ethics, and public servants have Codes of Conduct, there is no specific guidance available to help achieve consistent quality research and evaluation practice given institutional arrangements, cultural practices and social conditions.
The institutional setting includes constitutional factors such as recognising the centrality of the Treaty of Waitangi. The legal setting includes recognising the provisions of legislation such as the Official Information Act, the Privacy Act and the Public Finance Act, and contracting and case law. These statutes can be viewed on the NZ Legislation site at:
Institutional arrangements see over $30 million worth of social research and evaluation occurring in the public sector per annum, through a range of mechanisms from central Government departments. Resources primarily flow through the Votes of Health, Education, Social Development, Labour, Justice and Research, Science & Technology (via HRC $2-4m and FRST $12-15m) and to a lesser degree from the full range of departments. A very small amount is available from philanthropic trusts and private donors. The Marsden Fund (administered through the Royal Society of New Zealand) adds about $1-2m per year at the ‘blue-sky’s’ end of the social research spectrum and further funds flow to tertiary sector social science via the Tertiary Education Commission.
2.3Social Context
The population of this country is small by international standards and is ethnically and culturally diverse. The major population ‘groupings’ are usually labelled as European/Paheka, Maori, Pasifika and Asian. These labels disguise considerable diversity in relation to values, attitudes and practices. Country of origin, first language, gender, education, age, socio-economic status and locale introduce further key variables.
Effective policy and practice often needs to be able to draw on research and evaluation that is grounded in the diverse realities experienced by the full range of the population or in particular the range of realities within a particular set of groupings. This often means that researchers and evaluators are required to commission and/or gather information from groupings with protocols and values somewhat different from their own. With relatively small numbers in particular groupings, and a relatively small overall population, there is a high risk of ‘research fatigue’ among some sections of the population.
There is also a relatively small research and evaluation community, this means that bad practice and/or ‘corner cutting’ tend to become well known. Capability building and learning from each other is critical to building our capacity to produce credible information that will be used. There are ways in which the pitfalls can be minimised for the unwary, and experience and learning can be shared. These guidelines attempt to gather some of these ways into a readily accessible, practical and relevant resource.
2.4Good Practice: An Introduction
It is generally recognised that good research and evaluation (R&E) is of a quality that enables it to be used in policy advice, service design and delivery, and decision making. Expectations, particularly relevant to Government-commissioned R&E, are that, for social policy R&E to be useful, it is:
- Relevant – understanding of the policy environment, addressing appropriate issues, applicable and is cumulatively building knowledge (synthesizing and consolidating);
- Timely – commissioned and delivered in a timely manner so as to meet Government’s policy formation and decision making requirements;
- Credible – of an appropriate quality, produced by skilled practitioners and considered methodologically sound, objective and non-partisan;
- Accessible – produces engaging and clear policy-relevant messages, recognising that data and methodological issues are secondary to the intended audience.
To achieve these desired outcomes, good practice is a cornerstone for ensuring credible and relevant R&E is conducted and well managed for timely and accessible results that translate into evidence informed action throughout the policy-practice cycle.
2.5Good Practice: Statement of Purpose
The purposes of the SPEaR good practice guidelines are to enhance the standard of research and evaluation practice across the social sector as a whole. The guidelines are designed to provide practice-based advice aimed at improving social sector research and evaluation systems and processes, enhancing the professional conduct of government officials and external research contractors, improve the generation of information that informs policy development and service delivery, and encourage the development of respectful and ethical working relationships between all participants in social sector research and evaluation.2.6Policy Cycle
The following diagrams were developed for the 2004 workshops to illustrate the context for social policy R & E. ‘The Policy Cycle Ideal’ is the theoretical ideal (the textbook model) while ‘The Policy Cycle in Practice’ is based on experience of the reality.
It is important to note that social R & E is only a part of the evidence spectrum used by policy advisors and decision makers. Other forms of evidence include information analysis from administrative data, cost-effectiveness analysis, forecasting and modelling, policy consultations, organisational feedback and expressions of popular opinion.
It is also important to note that the existence of social R & E does not guarantee its uptake,
2.7Project Cycle
The guidelines and associated principles can be applied to the process (or systems) summary diagram developed to outline the R & E process in government. The flow diagram represents the typical steps undertaken in the initiation, developmental, doing and communications/dissemination phases of projects.
The SPEaR approach to good practice is holistic in that the principles and guideline content cover all the stages of the R & E process, unlike almost all material detected elsewhere to date. For example, material on contracting seems to start at the point a contract is beginning to be negotiated or exists through to the contractor producing a report and being paid, rather than offering guidance about the stages and processes that precede or follow, which can be crucial to the success of the actual work and to the uptake of the report. Most Codes of Ethics seem to relate to the ‘undertaking of the project’ stage (see above), and sometimes to feedback to research participants, and do not offer guidance about pre-project design. Material about R & E involving Maori or Pasifika peoples is largely concerned with the project stages or is concentrated on research which is not conducted for social policy purposes.
However, the practical realities of R & E in the social policy context can mean that while we should always strive for good practice, there will be situations where time, budget, deadlines and agency prescriptions constrain people and hence their ability to achieve the guideline ideals. A principles-based approach enables people to ensure they cover basic standards and, where possible, they follow the principles through the layers of advice. Seeking advice from peers with more experience of the choices and trade-offs involved in social policy R & E will assist in the navigation of the ideals and realities involved.
If officials cannot achieve all the aspects of good practice, they need to be very clear why particular decisions are made.There are likely to be other projects where such issues have been faced and people with experience about what was done and why and what the consequences were. It is that sort of experience that we are trying to gather in the good practice programme and to bring it together in an accessible way.
3.0DEVELOPMENT PROCESS
3.1Process to April 2005
The need for the development of guidance for R & E practice was agreed to by SPEaR at its first meeting when discussing the implementation of the Terms of Reference. Four key areas for guidance were initially agreed: Contracting, Ethics, Maori and Pasifika. A diversity of practice, by agencies and by researchers and evaluators, was evident.
Initial information gathering (in NZ and in relevant overseas countries) and informal testing of ideas with a range of peers occurred during 2002 and 2003. Some relevant material existed – usually only covering part of the project cycle - or was written from the perspective of only some of the stakeholders. Useful material included cross-disciplinary Codes of Ethics, the Health Research Guidelines for Pasifika Research, several agency guidelines for consulting with Maori, and with Pasifika, and a UK paper on Contracting. Parts of some material were useful[3]. Basically, however, much of the material has had to be developed from scratch utilising accumulated experience and involving a range of people.
A series of Background papers were therefore produced by the SPEaR Secretariat and discussed at workshops held in early June 2004[4]. The Background papers were also made available through the SPEaR website and some feedback was received. Around 45 people with expertise in the four key areas participated. The workshop participants generally confirmed the need for sector guidance, identified issues which needed attention and debated various approaches which could be taken. A short summary report was presented to SPEaR at the June 2004 committee meeting.
Notes of Key Points: Best Practice Workshops (RG/PH to June 04 SPEaR)
Guidelines for R & E involving Pacific Peoples:
Recommended that an Advisory function/body is needed to get critical mass and quality enhancement, to enable knowledge building and minimise the one-off myriad of ad hoc ‘advisory’ gatherings now. This would need to be accessible by a range of agencies for a range of projects. Could be hosted by SPEaR and resourced partly by SPEaR (and perhaps partly by agencies). At present a few people are being stretched over and over again and agencies need to work smarter.
Guidelines for R & E involving Maori
Balance between conceptual/principles and vision/pragmatism. Important. Need to provide a framework for case studies and lessons. Some further development and resourcing required to achieve this.
Ethics
Debate occurred about need for a social agency ethics committee or whether existing mechanisms (ASSR, AES, on job supervision/training, approval processes) were sufficient and able to be strengthened into existing project practice. Needs some further distribution/consultation (including range of current practice). Good to utilise ASSR/AES material. Experts keen for engagement.
Contracting
Clear need to improve practice. Range of viewpoints expressed and there is a clear need for improved understanding between commissioners and contractors. R & E/Legal/Public Finance Act & audit , needs closer weaving. Competition v collaboration – dialogue needed. Modular contracting practice across agencies not effective in the long term as it leads to duplication. Need to focus on meta learning and net accumulation of knowledge.
Overall
Endorsement of process/appreciation for inclusion. Willingness for ongoing engagement. Importance of enduring relationships. Vision of evidence informed policy/quality R & E supported. Lifting the game needed. Capacity and capability building vital. Communication improvement and sharing of knowledge a good start. Breadth/teeth/traction needed in SPEaR Departments through this programme and game would lift through the wider social R & E sector.
SPEaR decided to establish a Working Group to oversee the development of the next stage of the guidelines. Progress was reported back to the sector both informally and more formally through various presentations such as the Pacific Research Fono (Lower Hutt 24 November 2004) and the Social Policy Research and Evaluation Conference (25/26 November 2004)[5].
By early 2005, the principles had been developed and the content of the guidelines sections was drafted. By this stage, the extent of the inter-linkages and connections between the four key areas was very evident, and given that web-based publication is the most accessible, it seemed sensible to have one overall guidelines document with sections. If a layered/tiered approach to information arrangement was taken, then detail could sit behind and connections could be handled through the various layers in a reasonably accessible way for the end user (see section 4.2). This approach would enable a user to both gain an overview and to delve into detail that was relevant and of interest rather than be crowded by detail they did not need at that time. It also allows for the continued evolution of the guidelines as further sections are developed, working examples become available, legislation changes, feedback on their application comes in, etc.
3.2May 2005 Workshop Objectives
The aim for the workshop is:
To bring together a group of R & E practioners and specialist knowledge holders, in relation to key areas of guideline development to shape, inform and extend the content of a good practice guideline “product”for draft release on the SPEaR website.
To achieve this objective, we aim to:
- Achieve a working degree of consensus among the Practitioner community (both Govt/non-Govt.) through the workshop process on a draft guideline structure and content
- Develop a draft guideline for public release on the SPEaR website to encourage to wider feedback;
- Have an evolving web-based product, which is embedded in the operating environment, that is useful for all social R&E community operators.
Workshop participation draws primarily from those who participated in the 2004 workshops, plus those who could not attend and provided comment. Invitations were extended further to extend and deepen input. SPEaR member agencies were again invited to nominate staff to participate.
3.3Process Proposed Post May Workshop
Objectives post-workshop include:
- SPEaR Committee sign-off on a draft product for communication via the SPEaR website
- obtaining a mandate for, and identifying key expertise to enable further development e.g. developing a section for R & E involving recent Migrants and Refugees, and a section for R & E on sensitive subject matter (e.g. R & E involving mental health service consumers, trauma survivors, etc.) where there are particular challenges involved in gathering reliable information
- obtaining a mandate for further promotion of the draft guidelines and application testing in a range of agencies;
- considering effective ways of implementing and monitoring the use of the guidelines building on the recommendations of the 2004 workshops and this 2005 workshop.
In other words, we intend to incorporate your input at this workshop and report back to the full SPEaR committee. We will use the SPEAR website as a key mechanism for sharing the draft Guidelines and enabling access, feedback and updating. We also signalled in the 2004 workshop report back that there were organisational aspects that needed further consideration (see section 3.1). The 2005 workshop participants’ comment on these aspects is welcomed.
3.4 Workshop Programme
The workshop programme has been designed to include the testing of the draft principles and key area guidelines through specialist and more generic eyes – using plenary and break-out groupings and formal and informal interaction. Each break-out group has a designated facilitator with experience in the area. Working party members will also be available as resource people for the break-out groups as appropriate.
/ Good Practice GuidelinesDraft Workshop Programme
10-11 May 2005
Brentwood Hotel, Kilbirnie
Timing / Activity / Rationale
Day One
9.30 am / Gather & Morning Tea / Airline timetables, Wgtn offices
10.30 am / Introductions
Summary of activity since 2004 workshops, the principles and draft application, agenda for workshop and work to do, to ensure links, overlaps, particular content is there, etc. discussion / Catchup’s and focussing
12.30 - 1.30 pm /
Lunch & informal discussion
1.30 - 3.30 pm3.30 - 4.00 pm / Breakout groups - discuss issues
- Maori
- Pasifika
- Ethics
- Contracting
Groups use Secretariat material as base for discussion and identification of any additional material needed, links with other groups, resource material etc
They have flexibility re timing, have meal support and can fit life needs in as needed.
4.00 - 5.00 pm / Come back together for brief progress report, including areas where there are inputs sought from other groups, etc / Progress milestone, opportunity for cross-group dialogue and linkages to be identified so further work can occur; arrangements made for input etc prior to 10.30 am on Day Two as possible
7.00 pm /
Dinner & informal discussion
/ Groups may have issues & across break-out group connections to be discussed and some people can go, if they need to.Day Two
Up to 10.00 am / Each break-out group re-convenes at times they agreed on Day 1, and complete material as needed, for report back / Flexible re break-out group needs. Can meet over breakfast if need to, etc
10.00 - 10.30 am / Morning Tea / Butcher’s paper or OHP report back finalised so all can access
10.30 - 12.30 pm / Report backs written up drafts;
discussion of issues aspects; overlaps and interweaves identified;
where to next and closing / Working through linkages, gaps and realistic identification of further work needed post-workshop and potential resources for that.
12.30 pm / Lunch for those who wish to stay / Travel and other life needs can be addressed.
4.0THE DRAFT GUIDELINES