Effectiveness draft v.1
Objectives of this document
The objective of this document is to guide and support Competent Authorities (CA) and audit bodies in verifying the effectiveness of the official controls, by proposing general principles.
The aim is to give the principles for the evaluation of the effectiveness of the official controls with a view of being applicable to audits of official control systems.
This document is intended to assist in the implementation of section 6.1 (Guiding Principles) of the Annex to Commission Decision 2006/677/EC.
Scope and intended audience
This guidance applies to the evaluation of the effective implementation of official controls and CA's control systems and their suitability in achieving objectives, as required by Articles 4(6) and 8(3) of Regulation 882/2004.
It is intended for use by CA / audit bodies that carry out audits on official control (systems) according to the requirements of Article 4(6) of Regulation 882/2004.
It supports the development of good practice in the evaluation of effectiveness of the control systems.
The first stage of the document will focus on point 6.1b of 2006/677. At a later stage ….(develop more)
Legal framework
REGULATION (EC) No 882/2004
4.6. Competent authorities shall carry out internal audits or may have external audits carried out, and shall take appropriate measures in the light of their results, to ensure that they are achieving the objectives of this Regulation. These audits shall be subject to independent scrutiny and shall be carried out in a transparent manner.
8.3. . Competent authorities shall have procedures in place:
(a) to verify the effectiveness of official controls that they carry out;
and
(b) to ensure that corrective action is taken when needed and that the documentation referred to in paragraph 1 is updated as appropriate.
Commission Decision 2006/677/EC
6. Implementation of the Audit Process
6.1. Guiding Principles: (a) Compliance with planned arrangements; (b) Effective implementation; (c) Suitability to achieve objectives
To comply with the requirements of Article 4(6) of Regulation (EC) No 882/2004, the audit system should cover the following three points set out in Article 2(6):
(a) Verification of compliance with planned arrangements in order to provide assurances that official controls are carried out as intended and that any instructions or guidelines given to staff carrying out the controls are followed. This may largely be addressed by document review, but will also require on-site verification. The audit team will require good generic audit knowledge and skills to address this audit objective.
(b) Verification of the effective implementation of planned arrangements. In order to assess effectiveness, that is the extent to which planned results are achieved, on-site operational implementation must be included. This should include an assessment of the quality and consistency of the controls and should involve on-site audit activities. The audit team will require the relevant technical expertise in order to address this audit objective.
(c) The audit system should also seek to assess whether the planned arrangements are suitable to achieve the objectives of Regulation (EC) No 882/2004, and in particular the single integrated multi-annual national control plan. This should include assessing the suitability of official controls, with regard, for example, to their frequency and the methods applied, having regard to the structure of the production chain(s) and to production practices and volume. The audit team should have substantial knowledge and understanding of system auditing, together with relevant technical input to address this audit objective.
178/2002 Art 17.2
Definitions
This document should be read in conjunction with the definitions contained in Regulation 882/2004 and Commission Decision 677/2006 bearing in mind that the definitions from those documents apply.
Effectiveness: is generally speaking the capability of producing an (intended) effect or of achieving an objective. In this particular context it is the capability of a Competent Authority in achieving the objectives of Regulation 882/2004. It is not to be confused with efficiency, which is normally used when we want to refer to input-output ratio i.e. cost and/or resources required to produce an output.
Objectives of evaluating effective implementation of the official controls
(6.1.b) The main objective of verifying the effectiveness of official controls is to contribute to consumer safety, animal health and welfare, plant health and increase stakeholder confidence in effective and efficient use of resources.
Objectives of evaluating suitability of planned arrangements
6.1.c will be covered after 6.1.b
Principles
Discussion on the need to spell out the pre-requisites: Is 6.1.a a pre-requisite? This point requires further elaboration.
Key elements for consideration
· Has the official control system embedded the concept of evaluation of effectiveness?
· The aim of the verification of effectiveness: does the system detect non-compliances? And is the system able to solve the non-compliances?
· Are the objectives clearly identified? This would include both, quantitative and qualitative? (maybe a need for developing the different levels: strategic, operational? Depending of the scope of the audit or the control level)
· Do they have indicators? If so, are those capable of measuring effectiveness? Do they analyse/interpret the results and do they draw conclusions upon them? Does it lead to corrective actions? Does the review of effectiveness lead to corrective actions as part of continuous improvement? (PDCA cycle)
· The audits will focus not only what is done but also how it is done.
· Methods: quantitative and qualitative methods. A document (IE) has been circulated for consideration, describing qualitative approach to the evaluation.
The document from F1 on Art 8.3 of 882/2004 has a list of questions that will be considered.
The processes are outlined in the diagram below.
The Process – explanations to the diagram
- Define input
1.1. Results of the previous assessments of effectiveness
1.2. Any other information/data relevant for the purpose, e.g. legislation, resources (human resources, budget, infrastructures), MA(N)CP, Strategy, Objectives, Risk process
- Define throughput
2.1. Processes of controls implemented by the CA based on planned arrangements
2.2. Contingency controls?
- Define output
3.1.
- Define outcome
4.1.
- Define impact
5.1.
- Feedback/Review: the purpose is to increase, where possible, the effectiveness of the control system.
ANNEX: Useful Sources of information/Standards
- ISO 9004 (benchmarking)
- IIA
http://audit-network.wikispaces.com/Effectiveness