International Actuarial Association Actuarial Standards Committee

ISAP 1A – Governance of Models
Report on Treatment of Comments on the Final Review

Submission

The Actuarial Standards Committee (ASC) is pleased to submit this report. It was drafted by the ISAP 1A Task Force of the ASC and reviewed and edited by the ASC. Throughout the document “we” means the ASC.

This report documents the comments we received on the proposed final ISAP 1A in the final review process and how we addressed those comments.

We would like to record our thanks to the commenters on the final review. They have helped us make slight adjustments which have improved the ISAP.

Process

In accordance with the IAA’s due process for the development of ISAPs, the proposed final ISAP 1A along with the amendments to the Glossary for the proposed final ISAP 1A were published for final review on 18 August 2016 with a comment deadline of 18 September 2016. The proposed final ISAP 1A and the associated Glossary were provided to all member associations and commenters on the Exposure Draft to enable them (a) to draw attention to any unintended consequences in the final draft’s wording, and (b) to determine whether their comments have been appropriately considered in the preparation of the proposed final ISAP.

General overview of the responses to the final review

We received 15 comment letters; 13 from Full Member Associations, and 2 from Standard Setters other than FMAs,

Nine of the fifteen commenters were satisfied with the proposed final ISAP 1A as published for final review. The other six commenters made specific suggestions which were carefully considered by the ISAP 1A Task Force of the ASC. We have made minor edits in response. Comments received can be viewed in their entirety on the IAA website at

http://www.actuaries.org/CTTEES_ASC/ISAP1A/Final_Review_Comments/Final_Review_ISAP1A_CommentsReceived_Sept2016.pdf

Summary of Comments, and Our Responses:

Submitted By / Date Received
1. / Institute of Actuaries of Korea / 9 September 2016
Comment / We have reviewed the proposed final ISAP 1A and the only feedback we want to submit is the following: The effective date of ISAP 1A should be after the IFRS Phase 2 effective date.
Response / ISAP 1A applies to all models, not just IFRS models so we believe it is not appropriate to tie its effective date to IFRS Phase 2.
2. / Czech Society of Actuaries / 12 September 2016
Comment / On behalf of the Czech Society of Actuaries, I express satisfaction with the final version and with the way how the comments have been taken into consideration.
Response / Thank you
3. / Canadian Institute of Actuaries / 12 September 2016
Comment / Response submitted by Dave Dickson, CIA President
The CIA would like to offer the following comments on the proposed final (PF) version of ISAP 1A, dealing with the governance of models.
Overall, we are supportive of the PF ISAP 1A. In particular, we note that the changes made to section 1.2 now limit the application of ISAP 1A to “models that support an entity’s decision making”. This change alleviates a number of our concerns, in particular those we had expressed concerning sections 2.1 and 2.2 of the exposure draft (ED).
There is one suggestion that we had offered in our feedback on the ED that was not reflected in the PF version, and which we reiterate here. It concerns Section 2.4.1. We had suggested that someone who develops a new model should include, as part of the documentation, the identification of required data/inputs and key assumptions. The current text in the PF version still excludes these items; this continues to surprise us.
Response / The ISAPs are principle based, rather than prescriptive. The ASC considered this comment carefully but concluded these proposed additions were more in the province of how to model rather than governance of models, and also were more on the prescriptive side. Accordingly, we did not make this change.
4. / Deutsche Aktuarvereinigung e. V. (DAV) / 15 September 2016
Comment / We, DAV, have no further comments on ISAP 1A.
Response / Thank you
5. / Casualty Actuarial Society / 16 September 2016
Comment / On behalf of the CAS, submitted by Bob Conger, CAS
Delegate to IAA Council
Our comments on an earlier draft of ISAP 1A related primarily to reliance on others, and to the situation in which an actuary is part of a team (perhaps a junior member of a multidisciplinary team). In responding to our comments, the ISAP 1A team acknowledged the issue, but the ISAP 1A team concluded that it is an ISAP 1 issue, and that this issue should be reviewed as part of the next review of ISAP 1. The team did not make adjustments to ISAP 1A in response to these CAS comments.
The reliance on others takes on additional complexions in the governance of models because of the likelihood of relying on highly complex and opaque models developed by others (whether the others are employed in the same organization or employed in a third party vendor). Many such models are developed by multi-disciplinary teams, not limited to actuaries. And reliance on another party for a complex model is a different matter than relying on someone else to provide data and information. Complex models include a great many explicit and implicit assumptions about the behavior of the universe (the structure and parameters of catastrophe models being a clear example); and these models produce results that are affected by the assumptions, and that sometimes are relied upon to guide material decisions.
We recognize the Actuarial Standards Committee’s decision to embed common elements such as Reliance on Others within ISAP 1, and to incorporate these elements in other ISAPs by reference rather than by repetition. The need to amplify the Reliance on Others section is ISAP 1 so that it provides sufficient guidance for situations arising in the governance of models appears to us to be a significant matter, and we look forward to the review and modification of that section of ISAP 1.
Response / Thank you for these suggestions for improvement to ISAP 1. We agree that work needs to be done on ISAP 1 addressing the responsibilities of an actuary working on a multi-disciplinary team. It is on our list of potential improvements to ISAP 1. We plan to survey the Member Associations on if and when ISAP 1 should be updated.
6. / Society of Actuaries in Ireland / 16 September 2016
Comment / The Society of Actuaries in Ireland is satisfied with the proposed draft.
Response / Thank you
7. / Japanese Society of Certified Pension Actuaries / 18 September 2016
Comment / I will send the comments on the proposed final ISAP 1A by the Japanese Society of Certified Pension Actuaries (JSCPA) as an attached file.
Akihiro Hotta
Chair, Post-employment benefits Accounting Standards Committee of JSCPA
Because the way in which ISAP 2 and ISAP 3 relate to ISAP 1A is still ambiguous, we cannot agree to the proposed final draft at this time.
The proposed final draft lacks consistency in the following aspects:
(1) Because ISAP 1A states that it applies to all models, it can be read that ISAP 1A should apply to all models covering all aspects of actuarial practice, including ISAP 2 and ISAP 3.
(2) There is no statements in ISAP 2 and ISAP 3 that requires compliance with ISAP 1A. (The recently released proposed final draft of ISAP 5, however, states that compliance with ISAP 1 and ISAP 1A is a prerequisite.)
According to the SOI, ISAP 1A is to be drafted separately from ISAP 1, reflecting voices from several member associations wishing to avoid problems that would otherwise have arisen with their ongoing task of adopting ISAP 1. However, as stated above, there are inconsistencies in the overall ISAP structure.
If you weigh the fact that several member associations faced problems adopting ISAP 1 in those days against maintaining the coherence of the overall ISAP structure, the latter is far more important. Therefore, the IAA should give priority to ensuring consistency among all ISAPs. In other words, it should focus on developing ISAPs with their consistency at any time in mind by viewing them as a set of standards.
Although our initial understanding was that ISAP 1A relates only to the actuarial practices of insurance companies, the proposed draft of ISAP 1A seems to apply to models in all areas of actuarial practice. Therefore, while ensuring the consistency of all ISAPs in that light, you could, for instance, add a note stating that member associations who experience problems in adopting ISAPs would be given some latitude.
If the IAA is okay with such an approach, would it develop ISAP 1B, ISAP 1C, ISAP 2A, etc. to circumvent specific problems that may arise among member associations when adopting the standards? We do not think that would be a wiser way.
Response / The due process for ISAPs requires extensive review before change can be made. This makes it impractical to keep the ISAPs perfectly synchronized. If and when we revise ISAPs 2 or 3 we will make that change (if ISAP 1A has not been merged into ISAP 1 at that time). We also note that a standard-setter adopting ISAP 2 or ISAP 3 is free to include a cross reference to ISAP 1A in its adopted version, and we would encourage such an action.
Neither the IAA nor the ASC have stated that ISAP 1A applied only to insurance models. The growth of complex insurance models probably drove its creation, but it has always been an ISAP that applies to models in any area of practice.
We would like to clarify the role of the ASC. The ASC drafts, and the IAA adopts, model standards. We do this for two reasons:
1.  To provide a useful tool to Member Associations and other standard setters to assist them to create actuarial standards of practice that apply to its members (which an ISAP, as such, never does). Member Associations have no obligation to adopt ISAPs and are free to modify them in any way to suit their local needs (or for any other reason).
2.  To encourage all actuarial standard setters to adopt actuarial standards of practice and encourage the convergence of actuarial standards of practice worldwide.
8. / Canadian Actuarial Standards Board / 18 September 2016
Comment / Please find attached our response to your request for comment on the Final ISAP 1A on Governance of Models.
On a general note we found the changes from the exposure draft very extensive and wondered if a re exposure would be in order. However we also have very few comments and found these to be of high quality so this is more of an observation than a concern.
In the Proposed Final ISAP 1A:
• Paragraph 2.1.2 we found the suggestion that the validation should be performed by ‘a team’ as a very high and onerous standard likely achievable only by larger organizations and warranted only for very complex models. We would suggest replacing ‘a team’ with ‘individual(s)’
• We found paragraphs 2.2.1 and 2.3.1 ‘Understand the model’ very vague and could be subject to a very wide degree of interpretation. We also did not feel these added much and could be easily removed.
In the accompanying glossary:
• We find the definition of model somewhat ambiguous. It may not be sufficiently clear that a single model may have many model runs. Controls for the inputs and outputs of each run are needed, but validation is not. It is easier to clarify this when using the UK/Cdn/(and until recently US) terminology of distinguishing of model specification, model implementation and model realization/run. Then one can state clearly that a model will have one or more model runs determined by distinct sets of input. It is also not clear "that simplify a more complex system" is intended to modify "methodologies" only, and not "assumptions, data, and methodologies".
Response / We have made minor edits to the ISAP:
·  Replaced team by individual(s) in paragraph 2.1.2.
·  Deleted the phrase “that simplify a more complex system”, in the definition of model (in the Glossary) since it is unnecessary.
The other suggestions fell into the category “nice to make” rather than “clearly improve”. We used the latter test at this stage of development.
9. / Svenska Aktuarieföreningen / 18 September 2016
Comment / Svenska Aktuarieföreningen has no further comments on "Proposed Final ISAP 1A on Governance of Models"
Response / Thank you
10. / Institut des Actuaires (France) / 18 September 2016
Comment / Thank you very much for this new submission, the final draft was distributed to our working group and we did not receive any new comments.
We also reviewed how our comments were taken into account and outcome is deemed reasonable.
Given timing of our working group meetings we will normalize the closure of this review in the next days only with recognition that any additional feedback would be post deadline.
Response / Thank you
11. / Polish Society of Actuaries / 18 September 2016
Comment / The Polish Society of Actuaries welcomes the proposed ISAP and has no comments to its contents.
Response / Thank you
12. / Italian FMA Istituto degli Attuari and Ordine degli Attuari / 18 September 2016
Comment / We agree with the final draft's wording as well, in our opinion, the TF has appropriately considered the comments.
Response / Thank you
13. / Royal Dutch Actuarial Association / 20 September 2016
Comment / The Royal Dutch Actuarial Association has reviewed the final draft of ISAP 1A, especially on the follow up on our comments made. We appreciated that a Report on Treatment of Comments has been provided. This is very useful for understanding the changes which have been made, although we note that not for all our comments a response has been provided. Our overall conclusion is that the standard has improved and is suitable to use in practice. We note that we believe that some small improvements as mentioned in our former response still could be made, but as we believe these are small improvements we can support the standard in its current form.
Response / Thank you
14. / Financial Reporting Council (UK) / 22 September 2016
Comment / Thank you for the opportunity to comment on the final draft of the Exposure Draft. I am responding on behalf of the Financial Reporting Council.
We are pleased that our suggestions have been considered and amendments have been made to reflect most of them.
We welcome the restructuring of the content in 2.1 and believe it enhances the clarity of the standard. However it is not clear to usthat the change to 2.1 achieves the intended result of clarifying that it is the risks resulting from the use of the model, rather than the risks arising due to the complexity of the model, that drive the level of model governance. It still appears to us to place more emphasis on model complexity as the driver
Response / We edited the stem of 2.1. to connect it directly to an intended user drawing an incorrect conclusion
15. / Institute & Faculty of Actuaries (UK) / 27 September 2016.
Comment / There are, however, some comments that we felt it might be useful to raise, notwithstanding the IFoA’s overall position that the substance of its response has been addressed:
1.  It may be unnecessary to include the phrase “understand the model” in sections 2.2.1, 2.3.1 and 2.5.1;
2.  In section 2.1.2 there are examples included at the first bullet and this might not be appropriate for inclusion in a standard (as opposed to an IAN or other form of guidance material);
3.  In section 2.5.4 the word ‘repeated’ might be a more appropriate word than ‘redone’; and
4.  Other than in its numbering (the use of ‘1A’) ISAP 1A does not make any reference to any special relationship between it and ISAP 1. It would be useful to understand further whether there are still plans to incorporate ISAP 1A into ISAP 1 or whether it will simply remain as a new (and separate) standard.
Response / 1.  We made no change to “understand the model” as deleting it would not clearly improve the ISAP.
2.  We deleted “Examples of” which addresses this concern.
3.  Either word would fit but we prefer “redone” as it gives the validator leeway to change the tests used if appropriate. “Repeated” implies the same tests must be used.
4.  We plan to survey the Member Associations on if and when ISAP 1 should be updated. At that time, we will also ask about merging ISAP 1A into ISAP 1 (which we would like to do if it will not be too disruptive).

Page 2 of 8