The Canadian Evaluation Society (CES)

Experience in Developing

Standards for Evaluation & Ethical Issues

Paper presented in The 5th European Conference on the

Evaluation of Structural Funds in Budapest, Hungary

Workshop 6: Developing STANDARDS for

Evaluation & Ethical Issues

26 June 2003

Gwen Keith

Canadian Evaluation Society (CES) President

B.ED., M.ED., M.ED.

Table of Contents:

Page

1.Objectives of the Paper 3

2.Developing Evaluation Standards 4

in North America

3.Developing Ethical Conduct Guidelines 10

in Canada

4.Application of Joint Evaluation Standards 14

& CES Ethical Guidelines in an Enlarged Europe

References 16

Appendices:

A.The Program Evaluation Standards 17

B.Fundamental Ethical Principles 21

C.Canadian Evaluation Society (CES)

Guidelines for Ethical Conduct 24

The Canadian Evaluation Society (CES) Experience in

Developing STANDARDS for Evaluation & Ethical Issues

The Joint Evaluation STANDARDS and CES Ethical Guidelines have helped to positively define the practice of evaluation in Canada. The story of the process of development and renewal of these resources under the existing North American structures will be highlighted with considered applications to program evaluation in Europe.

Objectives of the Paper:

The Canadian experience of developing evaluation standards on a international level and ethics for evaluation at a national level will be presented. In particular, the processes for developing the Joint Evaluation STANDARDS and the Canadian Evaluation Society Guidelines for Ethical Conduct will be presented with implications for use in Structural Fund evaluation.

1.Program Evaluation STANDARDS

  • What is a standard?
  • What is the Joint Committee of STANDARDS for Educational Evaluation and it’s operation?
  • What is the role of The Canadian Evaluation Society related to this?
  • What are the Program Evaluation STANDARDS and advantages?

2.CES Ethical Guidelines

  • What are ethical guidelines?
  • What is Canadian Evaluation Society (CES) and how does it operate?
  • What is the process for development of the Guidelines?

3.Application of Program Evaluation STANDARDS and CES Ethical Guidelines to an enlarged Europe and for Structural Fund evaluatoions.

The STANDARDS and Guidelines presented here represent work since 1981 in the USA with the launching of the Standards of Educational Programs, Projects and Materials for program evaluation and Guidelines for Ethical Conduct in Canada published in 1996. While both of these initiatives have had sustainable use in evaluation practice in North America, their development and renewal involved several challenges. These included working through established organizational structures, creating agreement on common ground, dependency on the work of evaluators and research grants, strategies to market the material, and the challenge of continual renewal. Canadian Evaluation Society wishes to contribute to the global evaluation community and the story of the development and use of these resources could be of value to Structural Fund Evaluators.

Developing Evaluation STANDARDS in North America

Defining the Area:

The Program Evaluation STANDARDS (STANDARDS) were developed and are sustained by the Joint Committee of STANDARDS for Educational Evaluation (Joint Committee). A standard as defined by the Joint Committee is “a principle mutually agreed to by people engaged in a professional practice of which, if met, will enhance the quality of fairness of the professional practice.” The Joint Committee has evaluation standards development in program, personnel, and student evaluation.

The Joint Committee of STANDARDS for Educational Evaluation has been in operation since 1975. It is a voluntary standards setting body accredited by the American National Standard Institute (ANSI). Standards approved by ANSI become the American national standards in worldwide communications and require renewal on a regular basis. The Joint Committee is a non-profit organization of at least sixteen professional associations dedicated to the use of evaluation standards to improve evaluation practice. The Joint Committee provides training and technical assistance to evaluators and users of evaluation results. It maintains a balance of perspectives between practitioners and evaluation specialists. The Joint Committee meets in the Evaluation Centre at WesternMichiganUniversity in Kalamazoo, Michigan, USA. Its work is supported by grants, foundations and from contributions from sponsoring organizations.

Primary objectives and responsibilities of the Joint Committee are:

  • serving as the governing body, articulating policies by which the Joint Committee sets standards;
  • systematically obtaining feedback and advice from all directly and materially affected persons during the process of standard setting;
  • articulating standards involving assessment of needs for standards, drafting trial standards, obtaining reactions and advice from the field, studying relevant literature and finalizing standards;
  • promoting the sound use of the standards through researching their use, issuing interpretations in response to persuasive issues and specific problem areas, and promoting and assisting training;
  • ensuring that the standards are appropriately evaluated;
  • informing sponsoring and cooperating organizations, involving them in the work of the Joint Committee, and helping them use and disseminate the standards;
  • facilitating feedback on the standards and is dedicated to the revision of standards as required.

The Joint Committee published resources that serve as standards for evaluation practice based on research, theory and professional practice. It published The Program Evaluation STANDARDS in 1994, The Personnel Evaluation STANDARDS in 1998 and The Student Evaluation STANDARDS in 2002 with revisions to the publications taking place on a regular established basis. It promotes sound evaluation practices through meeting existing and emerging needs in the field of evaluation through such standard setting.

The Canadian Evaluation Society (CES) became a sponsor in 1994 and was accepted with voting membership on the Joint Committee with the appointment of Gwen Keith, a CES National Council member, as the CES representative. CES has been involved in the process of revising program evaluation standards and personnel standards and developing student evaluation standards ever since. Revised evaluation products are the result of an extensive review process including submissions by written panels, reviews by national and international panels of evaluation experts, series of field tests, hearings across North America and the involvement of external validation panels.

The Program Evaluation STANDARDS purpose and use:

The goal of The Program Evaluation STANDARDS (STANDARDS) is the development of evaluation standards to help ensure useful, feasible, ethical, and sound evaluation of educational programs, projects and materials. Taken as a set, thirty STANDARDS provide a working philosophy for evaluation. They define the Joint Committees’ conception of the principles that can guide and govern program evaluation efforts. They are intended for both users of evaluation and for evaluators.

These STANDARDS for sound program evaluation practice are organized by the characteristics of evaluation including STANDARDS for utility, feasibility, propriety, and accuracy. Each standard is accompanied by a rationale, guidelines for meeting the STANDARDS, common errors associated with the STANDARDS, illustrative cases and references (see Appendix A for a summary of the Standards for Program Evaluation).

The STANDARDS have been developed for use by teachers, administrators, school boards, training evaluators, curriculum specialists, legislators, personnel administrators, counsellors, community leaders, business and education associations. They guide the decisions, employment and assessment of evaluation of educational programs, projects and materials but have applicability other areas. People who commission or conduct evaluation or who use evaluation results to improve education and training in schools, universities, medical and health care fields, the military, human resources, the government and law would find these STANDARDS useful.

There is significant support of the use of the STANDARDS. Some examples are included here. The opinions of Stufflebeam (1992), Fournier (1994), and Patton (1994) are highlighted regarding the support for and use of the STANDARDS.

Stufflebeam (1992) describes sets of STANDARDS, such as The Program Evaluation STANDARDS, as noteworthy because they provide:

  • an operational definition of … program evaluation;
  • evidence about the extent of agreement concerning the meaning and appropriate methods of educational evaluation;
  • general principles for dealing with a variety of evaluation problems;
  • practical guidelines for planning evaluation;
  • widely accepted criteria for judging evaluation plans and reports;
  • conceptual frameworks by which to study evaluation;
  • evidence of progress ... to professionalize evaluation;
  • content for evaluation training;
  • descriptions of “best evaluation practices” (p.1).

In a review of research using the STANDARDS, Stufflebeam (1992) concluded that …“evidence supports the position that The Program Evaluation STANDARDS are needed, have been carefully developed, are credible in the United States, and have been put to practical use.” (p.26)

Fournier (1994) recommends that “The Program Evaluation STANDARDS is a “must have” book for anyone responsible for reviewing evaluation proposals, planning and conducting evaluations, managing evaluation projects, or judging the merit and worth of evaluations once completed. For experienced practitioners, it provides a set of values and principles by which to guide successful practice… For newcomers and the less experienced who may be responsible for commissioning and using evaluations, the STANDARDS supply a useful framework for generating a list of questions to raise about any evaluation plan… an invaluable “how to” resource for graduate students venturing out into the field, and it instills a sense of what it means to be a responsible evaluator… For clients or consumers, the book offers advice on what they should expect of an evaluation...”

Fournier indicated that the STANDARDS show much variation and diversity: “variation in terms of the evaluation field (e.g. health, law), evaluation settings (e.g. business and industry, federal government, training and development), evaluation roles (e.g. internal, external), and evaluation clients (management team, human resource director, medical student)…”

Patton (1994), states that, “Certainly no contemporary student of evaluation should come through a training program without studying the STANDARDS.” (p.195) “There can be no question that these are the evaluation profession’s definitive statement of STANDARDS. I use knowledge of the STANDARDS as an indicator for knowledge of evaluation… the STANDARDS represent much more than a set of professional guidelines; they constitute a philosophy of evaluation that emphasizes and values utility, feasibility, propriety, and accuracy.” (p.198)

The Canadian Evaluation Society sees the Program Evaluation STANDARDS as useful as a stimulant for research and best practice in evaluation.

  • the STANDARDS must sustain a high excellence to continue to be ANSI approved STANDARDS for professional principles and conduct. The public STANDARDS setting process offers a systematic and ongoing process for applying STANDARDS for improving evaluation services.
  • the STANDARDS continue to offer new thinking in program evaluation i.e. sensitivity to culture and minority issues.
  • the STANDARDS are congruent with successful training in program evaluation as evidenced by school conference services; training documents, review in projects and dissertations. The use of illustrative cases is particularly noteworthy.
  • the STANDARDS have expanded to several countries and involve electronic communications.
  • legislative agendas use the STANDARDS to evaluate programs, use of STANDARDS as prerequisite for a consultant to evaluate programs.
  • those who attempt to integrate the STANDARDS into their mandates are encouraged to generate cases for review to help make program evaluation more successful.
  • the Program Evaluation STANDARDS are aligned to personal and student evaluation. These act as a reference for stimulating discussion about the need for professional evaluation and for reference about the complexity of setting STANDARDS.
  • the journey with the Joint Committee process gives one a successful, process for developing STANDARDS, updating STANDARDS, marketing STANDARDS and ensuring that there can be a strategy with the intention prospectively.
  • implementation of the STANDARDS can enhance competencies by helping one ensure practice and demonstrate commitment to evaluation.
  • since the STANDARDS recommend evaluation be formally and systematically evaluated against pertinent STANDARDS they can be helpful in the certification of evaluation practices by being a readily available set of guidelines. They can give validity to a process service as they are highly reputed by evaluation participants.
  • These STANDARDS inform discussion on key issues including
  • selecting among schools of choice
  • determining how to improve schools
  • asking good questions
  • The Joint Committee serves in a brokerage role to help evaluators to come in contact with the STANDARDS. It serves an information role by being a depository for papers, articles, and reports written about use of STANDARDS in educational evaluation. There is also a role of research and development in terms of taking on projects that will further the application of STANDARDS that have been developed or will be developed. The technical assistance role is evident in providing testimony in court. Committee members provide workshops to help people make use of the STANDARDS.
  • The Joint Committee continues to invite groups to be involved in validation panels and reviews.

Developing Ethical Conduct Guidelines in Canada

The Guidelines for Ethical Conduct (see Appendix B) were developed by the Canadian Evaluation Society (CES) and published in 1996. These guidelines focus on ethical conduct in competence, integrity and accountability for evaluation.

Defining the Area:

The Canadian Evaluation Society (CES) is a national organization with over 1,600 individual members and 14 Chapters across Canada. It was incorporated in 1981 as a Canada-wide, non-profit, bilingual corporation. It is dedicated to the advancement of evaluation theory and practice. It’s vision is to be a leader for evaluation in Canada and a major contributor in the global evaluation community. The mandate of the CES is to:
(1) provide leadership to individuals and organizations in support of evaluation theory and practice in Canada and in the global community; (2) improve the state of evaluation theory and practice; (3) promote the importance of an evaluation culture; (4) promote and facilitate the enhancement of evaluation capacity for members and non-members.

Framst (2003) describes the CES structure and operation. For administrative and representational purposes, the Society’s by-laws define 14 chapters. Members may form chapters of the Society for their province or territory to further the objectives of the Society. Once recognized by Council, a chapter may act on behalf of the members of CES in the province or territory. A chapter will be recognized if it has at least 20 members, its by-laws are compatible with those of the Society, and there is no other recognized chapter in the province/territory.

A board of Councillors (Council) governs the Society. Council is made up of officers (President, Vice-President, Secretary, Treasurer) and Councillors. Each recognized chapter is entitled to name one representation (Councillor) to the Council. Each chapter is governed by its executive, in accordance with its own by-laws.

CES National Council, with representation from each active chapter, is responsible for carrying out five basic functions.

1.Providing strategic direction to CES at the national level.

2.Establishing and implementing national standards and policies. CES Council has endorsed the STANDARDS of Utility, Feasibility, Propriety and Accuracy development by the Joint Committee on STANDARDS for Educational Evaluation, and in 1996, established three guidelines for ethical conduct of evaluators: Competence, Integrity and Accountability.

3.Designing and delivering services and programs that are of nation-wide interest. The following list illustrates some of the programs and services delivered by CES on a nation-wide basis:

  • A national conference that is rotated around Canada.
  • The Canadian Journal of Program Evaluation.
  • The CES Newsletter that is published electronically and posted to the CES web-site.
  • Essential Skills Series Curriculum and Facilitator’s Manual (ESS). This series of four introductory level on-day workshops was developed with funding from CES National.

4.Administering the day-to day affairs of CES. Through a contract arrangement with The Willow Group in Ottawa, CES provides for administration of the membership database, membership fees, accounts payable, Council and Annual General Meeting, etc.

5.Oversight of chapters. The CES President and Council have an obligation to monitor the activities and financial status of chapters as indicated above.

CES Provincial Chapter Boards/Executives have similar responsibilities to those of CES Council. While the primary focus is regional, chapters have a responsibility to participate in governance and program design at the national level.

1.Providing strategic direction to the chapter, based on both provincial/territorial and national needs and priorities, and accountability of policies, programs, and services of public and private organizations.

2.Establishing and implementing STANDARDS and polices. Chapter directors and executive members have a responsibility to be aware of standards and policies established at the national level, and to promote awareness among their members. They are also responsible for establishing policies related to operation of their chapter. ES-Ontario has developed a policy manual to help ensure corporate memory about Chapter policies at the Board level.

3.Designing and delivering regional services and programs. This is the largest area of responsibility for Chapters from a resources perspective.

4.Administering the day-to-day affairs of chapters. In addition to administrative responsibilities associated with programs and services, chapter Boards and Executives are responsible for ensuring compliance with CES and chapter by-laws, financial viability of the chapter and accountability to members of the chapter.

5.National leadership and support. National Council, the primary governing body of CES, is a collection of individuals from each of the provinces and territories with approved and active chapters. It is the responsibility of Chapter Representatives to bring regional needs and perspectives to Council, and to work with their Council colleagues to design polices and programs that best serve the collective needs of members and the greater evaluation community.

It is within this structure that the CES Ethical Guidelines were developed. The CES has had an active history of exploring guidelines. In 1988 CES National struck a committee and Chapters were consulted between 1988 and 1989. In general, there was support received from Chapters for voluntary applications and non prescriptive suggestions coming from National CES as opposed to regulations and sanctions. The particular issue of ethical guidelines was revisited by the 1993 National Council with National Member Marthe Hurteau providing a discussion paper – Reflections on a Code of Ethics. There was a particular need for incorporating ethics into the Essential Skills Series, the training modules offered through local chapters across Canada. The Hurteau document was discussed at Chapter levels. A working group was then established to refine the guidelines and structure a session held at the Annual National Conference in Quebec in 1994. Based on the panel discussion held and follow up assistance from the National Council Professional Development Committee, a condensed version of the guiding principles was produced along with a questionnaire to solicit input from the wider CES membership (Appendix B). In 1996 the final set of Guidelines for Ethical Conduct were published and are sent to all new members of CES. Ethical Guidelines were addressed in an updated Essential Skills Series (see Appendix C).