Prompt Evaluation of the IRRS Mission to the Czech Republic

Prompt Evaluation of the IRRS mission to the Czech Republic Page | 1

Table of Contents

1.Introduction

2.Basic Mission Data

3.Feedback from the Team Members on the Advance Reference Material of the IRRS Mission

Characteristic additional comments

Further specific remarks

General conclusions

4.Feedback from the Team Members on the Effectiveness of the Mission

Characteristic additional comments

Further specific remarks

General conclusions

5.Feedback from the Host Country on the Effectiveness of the Mission

General conclusions

6.Overall Evaluation of the Effectiveness and Efficiency of the Mission

Discussion and conclusions

7.Summary

Appendix 1: Module-wise Coverage in the Mission Report

Appendix 2: Module-wise Coverage of the Action Plan

1.Introduction

The Contribution Agreement ENER/11/NUCL/SI2.588650 between the European Atomic Energy Community (represented by the European Commission) and the IAEA among others foresees as an expected result a performance monitoring based on the evaluation of some key performance indicators of the IRRS missions.

The Nuclear Safety Action Plan of the IAEA has as one of its main objectives the requirement to “Strengthen IAEA peer reviews in order to maximize the benefits to Member States” and in specific, it calls the IAEA Secretariat to review the effectiveness of the IRRS peer reviews.

In reply to these requirements a system of performance indicators have been elaborated in order to measure the efficiency and effectiveness of the IRRS missions. Some of the performance indicators are based on direct feedbacks from the IRRS team members and from the representatives of the country hosting the IRRS mission.

Results of and conclusions from the feedbacks and efficiency and effectiveness evaluations are given in the presentPrompt Evaluation Report.

2.Basic Mission Data

Host country: Czech Republic

Host organization: State Office for Nuclear Safety

Mission date: 18 to 29 November 2013

Team Leader: Derek Lacey (UK)

Deputy Team: Peter Johnston (AUS)

Team Coordinator: Adriana Nicic (IAEA-NSNI)

Deputy Team Coordinator: Ibrahim Shadad (IAEA-NSRW)

Number of external experts: 19

Number of IAEA staff: 6

Number of observers: 1

Mission type: full scope

Scope of the mission:

Core modules (No. 1 through 10)
Facilities and activities: NPP, research reactors, fuel cycle facilities, radioactive waste facilities, radioactive sources facilities, decommissioning activities, transport activities
Additional areas (Module 11): control of medical exposures;occupational radiation protection;control of discharges, materials for clearance and chronic exposures, environmental monitoring for public radiation protection
Fukushima module
Policy issues: (1)Governance issues related to the establishment of a Commission, (2) Transparency

3.Feedback from the Team Members on the Advance Reference Material of the IRRS Mission

The Advance Reference Material (ARM) normally includes the results of the self-assessment of the host country (usually performed with the aid of the Self-Assessment Tool – SARIS); the Action Plan for improvement in issues found in the self-assessment, a module-wise summary of the status and activity of the regulatory body reviewed and a number of other documents needed for an objective and well informed peer review.

The team members are requested to offer their opinions on the quality of the ARM by rating from 1 to 5 (5 reflecting the highest satisfaction) three questions and also expanding in free text their comments on the questions. The questions and the average values of the team members’ ratings are given below:

No. / Question / Average mark
Qr1 / How complete do you consider the ARM? / 3.7
Qr2 / How realistic picture could you obtain on the area you will be reviewing from the ARM? / 3.7
Qr3 / What is your overall evaluation on the quality of the ARM? / 3.6
Total average / 3.7

Characteristic additional comments

Qr1:

The stated compliance with requirements were not always supported by evidence
The regulatory guide and the management system related topics were missing from the English text

Qr2:

Some parts of the ARM were available too late
The document often described what the host country wished to reach not what the situation was
Certain areas were described quite realistically, in other cases the general picture was not supported by specific examples

Qr3:

The ARM was burdened by translation errors
The general quality was good, however, certain questions were answered in very generic manner
The number of Q/A’s were very different in different modules

Further specific remarks

The IRRS module structure is not always followed in the SARIS Questionnaires
The draft of the new Atomic Act would have been useful to see
The ARM was changing for too long and it is not user friendly
Notification of the team members on the change in the ARM would be necessary
The ARM and the mission reporttemplate should be consistent with each other
The ARM should contain specific examples of compliance

General conclusions

1)The ARM contained the necessary generic information, yet specific examples supporting stated compliance with requirements were missing

2)Structure and management of the ARM need revision to turn it more user friendly

3)The SARIS Questionnaires and the Report Template need to be revised for consistency

4.Feedback from the Team Members on the Effectiveness of the Mission

The team members are requested to offer their opinions on the effectiveness of the mission by rating from 1 to 5 (5 reflecting the highest satisfaction) five questions and also expanding in free text their comments on the questions. The questions and the average values of the team members’ ratings are given below:

No. / Question / Average mark
Qt1 / How effective do you consider the activity of the expert team during the mission? / 4.2
Qt2 / How effective do you consider the activity of the IAEA staff in the team during the mission / 4.2
Qt3 / How effective do you consider the activity of the Team Leader? / 4.6
Qt4 / How effective do you consider the activity of the Deputy Team Leader? / 4.5
Qt5 / How satisfied are you with the preparations of the mission (enough time for preparing yourself, information provided by the IAEA, etc.)? / 4.1
Total average / 4.3

Characteristic additional comments

Qt1:

The team was very effective, had good spirit and possessed ample experience
More time should have been available for interviews at the beginning of the mission.

Qt2:

The IAEA staff was very helpful, committed and provided practical guidance and assistance
The administrative support was highly appreciated

Qt3:

The team leader was effective, and has shown strong leadership
He was clear in his expectations, focused on relevant topics, good time management, excellent coordination
The team was guided too fast through the findings, more discussion would have been necessary

Qt4:

He did his job well, demonstrated excellent individual expertise
He was supportive, active and constructive
The role of the DTL is not fully clear. He should have spent more time on the general consistency of the report

Qt5:

The mission was prepared very well, all information provided by IAEA was timely, yet some parts of the ARM was updated late
There was not enough guidance for a first time participant
Information and preparations from IAEA were perfect, the Nucleus based access was a good idea
The team administrator provided good support

Further specific remarks

Cross-reading should be extended
More time would be neededfor interviews and discussions with the counterparts and also for group-discussions
Use of examples is recommended
Working in groups is very effective, this should also be followed when discussing with counterparts

General conclusions

1)All members and leaders of the team were rather effective and did a great job

2)Support from IAEA staff was appreciated

3)The mission schedule needs refinement to provide more time for interviews, group discussions and for good balance in discussing findings and the report

5.Feedback from the Host Country on the Effectiveness of the Mission

The Liaison Officer of the host country is requested to offer the opinion of the host on the effectiveness of the mission by rating from 1 to 5 (5 reflecting the highest satisfaction) seven questions and also expanding in free text their comments on the questions. The questions, the host country’s ratings and the associated comments are given below:

No. / Question / Mark
Qh1 / How effective do you consider the mission in assisting the continuous improvement of nuclear safety in your country?
Comment:The IRRS mission can be a significant stimulus to further improvement of national nuclear safety policy. Exchange of viewsand knowledgealways helpsto improvebecause theviewfrom the outsidebrings aslightlydifferent perspective oneachissue. / 4.0
Qh2 / How objective was the peer review?
Comment:The peer review was very objective, yet some members of the team IRRS tried to apply their experience from their own country as the good way. / 4.4
Qh3 / How has the mission helped the exchange of information, experience and good practice with other countries?
Comment:The IRRS mission is useful for the international exchange of information. On the other hand reviewers should avoid favouring their national approach. / 4.0
Qh4 / How consistent was the use of the IAEA safety requirements and guides in the mission?
Comment:Using the IAEA safety requirements was not very consistent. / 3.9
Qh5 / How justified are the findings of the peer review?
Comment:The findings are fully justified. Conclusionstimedifferencescorrespond tothe period covered bythe legislationof the Czech Republicand thecurrent recommendationsofthe IAEA. / 4.2
Qh6 / How relevant are the findings of the peer review for the future development of your regulatory body?
Comment:The findings are clearly defined and relevant. Findingswill be used inthe creationof new implementingregulationsfor the newAtomic Act. / 4.4
Qh7 / How competent were the reviewers in their review and findings?
Comment:The IRRS team was generally competent but some reviewers should come better prepared. / 4.5
Average / 4.1

Further specific remarks

The IAEA self-assessment tool (SARIS) should be modified. The current version does not reflect a graded approach. E.g., there are hundreds of questions in the modules for research reactors, transport, radwaste and much less focus is given to the modules on NPPs. The Module 10 questionnaire is very robust, very detailed, many items are repeated.
Interpretation of IAEA Safety Requirements and guidance ontheir possible implementation in the relevant Czech legislation would have been welcome.
Certain findings were somewhat formal and will not add improvements to the otherwise working system.
Discussion of the mission findings on a plenary session was missed, although there has not always been full consensus between the reviewers and their counterparts.

General conclusions

1)The purpose of the mission was acknowledged by the hosts

2)The review was considered sufficiently objective, the findings were accepted as realistic, the reviewers were judged generally competent

3)Reviewers should not take their national practice as basis for comparison in peer review

4)Plenary discussion of the final report is necessary

5)SARIS questionnaires need further revision

6.Overall Evaluation of the Effectiveness and Efficiency of the Mission

The Performance Indicators developed for the measurement of the effectiveness and efficiency of an IRRS mission were evaluated in the extent as it was made possible by the dataavailable at the time of the present evaluation.

In the next figure the values of the performance indicators as they follow from the evaluation of the mission data[1], as well as the overall effectiveness of the mission are presented. The rightmost columns (EFF. INDICATION) present the ranges where the particular PIs fall (green – optimum, yellow – acceptable, red – needing attention), whereas the frame in the right lower part summarizes the overall effectiveness of the mission (green – optimum, white – effective, yellow – acceptable, red – to analyse).

The bar diagram in the figure shows the number ratio of issues covered by the mission report to those foreseen by the Standard Mission Report Template. Details on this coverage are given in Appendix 1.

The coverage of the Action Plan items (defined by the host country) by Recommendations and Suggestions by the reviewers of the various Modules is shown in the figure in Appendix 2.

Prompt Evaluation of the IRRS mission to the Czech Republic Page | 1

Discussion and conclusions

The values of the effectiveness and efficiency Performance Indicators (upper right table in the figure headed by EFF. INDICATION) suggest the following conclusions:

1)Almost all Performance Indicators of the mission are in the optimum range

2)Feedback of the team on the ARM suggestsslight non-conformances as discussed in section 3.

3)Conciseness of the mission report is rather good, partly due to the fact that coverage of the required topics by the report was checked already during the mission. Details of the coverage are given in Appendix 1.

4)Details on the coverage of the Action Plan by findings (which is also in the optimum range) are given in Appendix 2.

5)The overall effectiveness of the mission is in the lower region of white (effective) range and its value (0.005) is one of the bestin recent missions, indicating a rather efficient mission.

7.Summary

The data of and feedback from the full scope initialIRRS mission to the Czech Republic have been analysed. The following conclusions are drawn:

The ARM contained the necessary generic information, yet specific examples supporting stated compliance with requirements were missing
Structure and management of the ARM need revision to turn it more user friendly
The SARIS Questionnaires and the Report Template need to be revised for consistency
All members and leaders of the team were rather effective and did a great job
Support from IAEA staff was appreciated
The mission schedule needs refinement to provide more time for interviews, group discussions and for good balance in discussing findings and the report
The purpose of the mission was acknowledged by the hosts
The review was considered by the host sufficiently objective, the findings were accepted as realistic, the reviewers were judged generally competent
The host noted that reviewers should not take their national practice as basis for comparison in peer review
Plenary discussion of the final report is necessary
SARIS questionnaires need further revision
Almost all components of the mission that are measured by Performance Indicators were rated optimum
The mission was rated highly effective (nearly absolute optimum) by the evaluated Performance Indicators

Appendix 1: Module-wise Coverage in the Mission Report

Prompt Evaluation of the IRRS mission to the Czech Republic Page | 1

Appendix 2: Module-wise Coverage of the Action Plan

As seen from the figure, the coverage of the Action Plan (AP) items by findings is rather good (the optimum range is above 0.5) in most Modules. Poor coverage is reached in Module 1 (responsibilities and functions of the government) where the AP included a single item and it was not covered; in Module 5 (authorization), where a multiple AP item related to radioactive sources has not been covered and in Module 7 (inspection), where three AP items related to inspection programme, inspectors’ responsibilities and witnessing activities in transport have not been observed by the team.

[1] On the Efficiency and Effectiveness of IRRS Missions, Draft v6, Working Material, IAEA, Vienna, 2013