Prompt Evaluation of the IRRS Mission to Finland

Prompt Evaluation of the IRRS mission to FinlandPage | 1

Table of Contents

1.Introduction

2.Basic Mission Data

3.Feedback from the Team Members on the Advance Reference Material of the IRRS Mission

Characteristic additional comments

General conclusions

4.Feedback from the Team Members on the Effectiveness of the Mission

Characteristic additional comments

Further specific remarks

General conclusion

5.Feedback from the Host Country on the Effectiveness of the Mission

6.Overall Evaluation of the Effectiveness and Efficiency of the Mission

Discussion and conclusions

7.Summary

1.Introduction

The Contribution Agreement ENER/11/NUCL/SI2.588650 between the European Atomic Energy Community (represented by the European Commission) and the IAEA among others foresees as an expected result a performance monitoring based on the evaluation of some key performance indicators of the IRRS missions.

The Nuclear Safety Action Plan of the IAEA has as one of its main objectives the requirement to “Strengthen IAEA peer reviews in order to maximize the benefits to Member States” and in specific, it calls the IAEA Secretariat to review the effectiveness of the IRRS peer reviews.

In reply to these requirements a system of performance indicators have been elaborated in order to measure the efficiency and effectiveness of the IRRS missions. Some of the performance indicators are based on direct feedbacks from the IRRS team members and from the representatives of the country hosting the IRRS mission.

Results of and conclusions from the feedbacks and efficiency and effectiveness evaluations are given in the presentPrompt Evaluation Report.

2.Basic Mission Data

Host country: Finland

Host organization: STUK

Mission date: 15 – 26 October 2012

Team Leader: Philippe JAMET (F)

Deputy Team Leader: John LOY (UAE)

Team Coordinator: AdrianaNICIC (IAEA NSNI)

Deputy Team Coordinator: Hilaire MANSOUX (IAEA NSRW)

Number of external experts: 18

Number of IAEA staff: 6

Number of observers: 0

Mission type: Reduced Scope First Mission

Scope of the mission:

Core modules (No. 1 through 10)
Facilities and activities: NPP, fuel cycle facilities, radioactive waste facilities, radioactive sources facilities, decommissioning activities, transport activities
Additional areas (Module 11): control of medical exposures, occupational radiation protection, public and environmental exposure control
Fukushima module
Policy issues: (1) Uranium mining and milling, NORM and associated waste; (2) interface between nuclear safety and nuclear security; (3) the possible conflicts of interest in the advisory bodies concerned with STUK

3.Feedback from the Team Members on the Advance Reference Material of the IRRS Mission

The Advance Reference Material (ARM) normally includes the results of the self-assessment of the host country (usually performed with the aid of the Self-Assessment Tool – SAT); the Action Plan for improvement in issues found in the self-assessment, a module-wise summary of the status and activity of the regulatory body reviewed and a number of other documents needed for an objective and well informed peer review.

The team members are requested to offer their opinions on the quality of the ARM by rating from 1 to 5 (5 reflecting the highest satisfaction) three questions and also expanding in free text their comments on the questions. The questions and the average values of the team members’ ratings are given below:

No. / Question / Average mark
Qr1 / How complete do you consider the ARM? / 4.25
Qr2 / How realistic picture could you obtain on the area you will be reviewing from the ARM? / 3.81
Qr3 / What is your overall evaluation on the quality of the ARM? / 4.09
Total average / 4.05

Characteristic additional comments

Qr1:

The ARM was comprehensive, of good quality, needed considerable effort to compile
Certain guides referred to were not available

Qr2:

It contained the most important information, yet practical examples were missing
Good overview, sometimes not clear enough
Gave realistic picture

Qr3:

Good quality, well thought through product
Repetitive at some places

General conclusions

1)The picture obtained from the ARMwas realistic, the quality was generally good, although sometimes it was not clear enough or was repetitive.

2)Description of the real processes would have been more enlightening than references to the YVL Guides.

4.Feedback from the Team Members on the Effectiveness of the Mission

The team members are requested to offer their opinions on the effectiveness of the mission by rating from 1 to 5 (5 reflecting the highest satisfaction) five questions and also expanding in free text their comments on the questions. The questions and the average values of the team members’ ratings are given below:

No. / Question / Average mark
Qt1 / How effective do you consider the activity of the expert team during the mission? / 4.40
Qt2 / How effective do you consider the activity of the IAEA staff in the team during the mission / 4.53
Qt3 / How effective do you consider the activity of the Team Leader? / 4.80
Qt4 / How effective do you consider the activity of the Deputy Team Leader? / 3.90
Qt5 / How satisfied are you with the preparations of the mission (enough time for preparing yourself, information provided by the IAEA, etc.)? / 3.68
Total average / 4.26

Characteristic additional comments

Qt1:

The team worked well and provided results very fast
Team members were well selected

Qt2:

The IAEA staffwas very helpful, efficient and committed
Some more support by printed material would have been useful
The experience of the IAEA staff was valuable

Qt3:

The team leader provided good leadership, was patient and effective
Very creative and active
High concern for achieving consistency and agreement

Qt4:

The deputy team leader provided good support to the team leader
His role was not clear
Provided very reasonable comments
Might have been more active

Qt5:

The ARM was provided too late
The information provided by IAEA was clear
It should be clarified what applies to the specific modules
During a longer opening meeting better explanation on the expectations would be needed

Further specific remarks

The report template was useful; the questionnaires should be updated to be in line with it.
More instructions on the expected details in the report are needed
Expectations related to the Fukushima module should be clarified, the modules needs revision and further guidance
Team building would be beneficial

General conclusion

1)The ARM was provided rather late leaving little time for the reviewers

2)In spite of the late availability of the ARM the team worked efficiently

3)The activity of the IAEA team was generally appreciated

4)The activity of the Team Leader was highly appreciated by the team members

5)The report template was well accepted

6)The Fukushima module needs reconsideration

5.Feedback from the Host Country on the Effectiveness of the Mission

The Liaison Officer of the host country is requested to offer the opinion of the host on the effectiveness of the mission by rating from 1 to 5 (5 reflecting the highest satisfaction) seven questions and also expanding in free text their comments on the questions. The questions, the host country’s ratings and the associated comments are given below:

No. / Question / Mark
Qh1 / How effective do you consider the mission in assisting the continuous improvement of nuclear safety in your country?
Comment:Recommendations and suggestions are very helpful for the improvement of the Finnish regulatory control framework / 4
Qh2 / How objective was the peer review?
Comment:The background and personal interest of the reviewers were reflected in the interviews in some modules. These were, however, mostly avoided in the final report / 3
Qh3 / How has the mission helped the exchange of information, experience and good practice with other countries?
Comment:The variations concerning the exchange of information, experience and good practices were significant between different modules and thematic topics / 3
Qh4 / How consistent was the use of the IAEA safety requirements and guides in the mission?
Comment:The variations concerning the use of the IAEA safety requirements and guides were considerable between different modules and thematic topics. However, the conclusions of the report were in general based on IAEA requirements / 4
Qh5 / How justified are the findings of the peer review?
Comment:The findings of the mission were in general well justified / 4
Qh6 / How relevant are the findings of the peer review for the future development of your regulatory body?
Comment:Recommendations and suggestions are very relevant for the improvement of the Finnish regulatory control framework / 4
Qh7 / How competent were the reviewers in their review and findings?
Comment:In general the reviewers were competent. However, all reviewers did not systematically follow the IRRS guidelines in the interviews. IAEA should arrange training at least to the EU-experts on the IRRS guidelines and procedures / 4
Average / 4.14

Further comments:

Self-assessment questionnairesare too detailed and very time-consuming to answer
Almost similar questions are repeated within the same module and also in other modules
Better approach would be to remain at the primary question level and to include description of the topics which should be included later

6.Overall Evaluation of the Effectiveness and Efficiency of the Mission

The Performance Indicators developed for the measurement of the effectiveness and efficiency of an IRRS mission were evaluated in the extent as it was made possible by the dataavailable at the time of the present evaluation.

In the next figure the values of the performance indicators as they follow from the evaluation of the mission data[1], as well as the overall effectiveness of the mission are presented. The rightmost columns (EFF. INDICATION) present the ranges where the particular PIs fall (green – optimum, yellow – acceptable, red – needing attention), whereas the frame in the right lower part summarizes the overall effectiveness of the mission (green – optimum, white – effective, yellow – acceptable, red – to analyse).

Prompt Evaluation of the IRRS mission to FinlandPage | 1

Discussion and conclusions

The values of the effectiveness and efficiency Performance Indicators (upper right table in the figure headed by EFF. INDICATION) suggest the following conclusions:

1)The size of the team,its IRRS experience and the length of the report it produced were in the optimum range.

2)As also discommended by several experts, the ARM was issued too late, just on the merge of the optimum range. This might be the reason for the lower than optimum number of advance comments.

3)According to the marks offered in its feedback the host was not fully satisfied with the mission (although the textual comments above reflect slightly more satisfaction then the marks).

4)The host provided a detailed and honest Action Plan with 49 actions therein. The mission has realized a considerably lower number of issues (recommendations and suggestions) this is why the action plan coverage and beyond action plan coverage Performance Indicators are far in the red range. These two insufficiencies of the mission result in the major part of the overall deviation from optimum. (Note here on the other hand, that inadequacy of the action plan coverage is partly due to the utterly fair approach of STUK in collecting all possible actions to be done.)

5)The findings are somewhat unbalanced in the sense that the expected negative correlation between the number of recommendations and the number of good practices in absolute value is lower than optimum. In other words one of these numbers (either the number of recommendations or that of the good practices) is too low. This is in agreement with the conclusion above that in an optimum case the number of findings would be higher. (It is to be noted here that the relatively low number of recommendations is partly the consequence of the approach chosen by the team leader in which the reviewers were encouraged to concentrate on major issues in their recommendations.)

6)The report conciseness, i.e. the coverage rate of the topics listed in the report template is acceptable. Going into more details, the bar graph in the figure shows that this coverage is fairly low in the authorization and enforcement modules and is lower than optimum in Module I (Governmental responsibilities and functions) and in Module IX (Regulations and guides), the other modules are very near or within the optimum range.

7)The overall effectiveness of the mission (0.07) is in the white (effective range).

7.Summary

The data of and feedback from the reduced scope first IRRS mission to Finland have been analysed. The following conclusions are drawn:

The team members were generally satisfied with the Advanced Reference Material (ARM)
The team considered that the picture obtained from the ARM was realistic, although concerns were expressed as for the clarity of some parts and more practical examples were missed
The ARM was providedtoo late
The team members were well selected and worked effective
The activity of the IAEA staff was generally appreciated
The activity of the Team Leader was highly appreciated by the team members
The report template was well received
The Fukushima module needs revision
The representatives of the host country were generally satisfied although in their ratings they expressed some concerns regarding the effectiveness, and usefulness of the peer review, objectivity of some reviewers was questioned, the usefulness of information exchange was uneven among the modules and reviewers
The findings of the mission did not cover all issues brought up in the Action Plan. The number of findings and that of the good practices are somewhat unbalanced
Coverage of the topics by the mission report was not optimum in some modules
The mission was rated effective by the evaluated Performance Indicators

[1] On the Efficiency and Effectiveness of IRRS Missions, Draft v5, Working Material, IAEA, Vienna, 2012