ABA Model Validation Working Group

Survey of Model Validation Organization, Model

Catalogue, and Validation Policy and Practices

May 2008

MODEL VALIDATION SURVEY

May 2008

The American Bankers Association Model Validation Working Group requests your participation in completing this survey on important model validation topics. The survey resultsare intended to help participants learn what others are doing to address these areas that are significant in helping their institutions achievesuccess in risk modeling.

Organization.The topics for this survey include the model validation organization, including reporting structure, number of staff and composition (experience and education), assistance by other company staff, and use of external resources.

Model Catalogue. The institution’s definition of a “model” is important in what is included in the catalogue of models falling under the requirements of internal model validation policies and compliance with OCC Bulletin 2000-16 “Risk Modeling.”The survey asks participants to indicate those key determinants used in their definition.Questions also request the level of detail used to classify the models in the catalogue.This would include business unit, business purpose (i.e., risk measurement, loss forecasting, asset and financial instrument valuation, performance management, and business and product forecasting).The questions also ask whether a ranking and prioritization scheme is used and the metrics applied to determine the level of importance or materiality.

Validation Policy and Practices.How the institution’s Model Validation Policy and Practicesdefine “independent” governance and requirements for validation determine the level of responsibility for business units and/or the model validation unit to complete their work.Survey questions focus on those responsibilities and validation practices performed by those units.Responses will determine to what extent institutions have established consistent and standard model validation practices across the banking industry.

Responses to the individual surveys by the participating institutions will be maintained in strictest confidence by the American Bankers Association.The summary of the responses will be presented in a manner that will not disclose the institution and the respondent to the question or survey.

Please respond to the survey questions and note any comments on the clarity of the questions and how you would respond differently based on alternative interpretations.

Please submit your completed survey to me by Friday, May 23, so that we can compile results before the Working Group discussion of June 5.

Thank you very much,

American Bankers Association

ORGANIZATION

Org 1. Which division or business line does the model validation function report through?

Chief Risk Officer Chief Financial Officer Internal Audit

Other (Indicate organizational alignment and comment on the rationale for this reporting relationship.)

Org 2.Governance and oversight committee

Is a governance and oversight committee established to oversee compliance with the model validation policies?
Does this committee’s responsibilities include approving the results of reviews and recommendations to address model deficiencies and areas of improving validation practices?

Org 3.Define the Model Validation function (office or department) management and staff structure. (Check if applies)

Define the Model Validation function management and staff reporting structure.
Management position that establishes policy, organization, and operating process for Model Validation:
  • Official and functional title:

  • Job description:

Management position that conducts model validation engagements and oversees staff or external consultants to complete validation objectives, scope, and reporting:
  • Official and functional title:

  • Job description:

Org 4. Staff Composition

Staff Positions / Number ofStaff (Actual/Target) / Official and Functional Title (Each) / Highest
Degree
(Each) / Yrs Experience in Modeling (Each)
Senior staff
Junior staff
Entry Level staff
Interns
Other Staff *
Consultants

* Auditors, programmers, technical writers, data management, etc.

MODEL CATALOGUE/INVENTORY

Cat 1. / Model Definition. Define models that are to be included in model catalogue.
Cat 2. / Is the model catalogue organized according to classifications (families, tiers, decision tools, etc.)? If so, what classifications and rationale do you use?
Cat 3. / Other types of applications/spreadsheets included in catalogue. (Explain if applies.)
  • How many of these ‘other types” are in the Model Validation catalogue?

  • How are these models/spreadsheets defined and “validated” under the Model Validation Policy and Procedures? (Pleaseexplain.)

Cat 4. Number and types of model.

Number / Define “Other”
Models in catalogue (total)
  • “High impact”

  • Other

Loan Adjudication
  • Loan scorecards (includes bureau data)

  • Appraisal valuation modeling (AVM)

  • Other (Please define.)

Structuring
  • Loan securitization (I/O, residual cash flow)

  • Servicing rights

  • Other (Please define.)

Customer Behavior
  • Deposit forecasting

  • Credit loss (PD, LGD, EAD, allowance/provisions)

  • Loan prepayment

  • Product sales forecasting and analytics (price sensitivities and elasticity’s, new products and promotional programs)

  • Other (Please define.)

Portfolio Valuation and Risk Measurement
  • Asset liability management and Interest Rate Forecasting

  • Economic capital forecasting and allocation

  • Derivative/option pricing

  • Trading VaR

  • Mortgage banking (pipeline, rate locks, warehouse, closed loans, MBS, and MSRs)

  • Acquisitions (M&A)

  • Other (Please define.)

Cat 5. / Is a measure of exposure or complexity (i.e., materiality, asset size, and/or transaction size) used to assign prioritization and frequency of validation for models? (Pleasedescribe the measure and how it is applied to rate and rank your models.)
Cat 6. / How are updates to the model catalogue performed, and business unit confirmation conducted (survey and/or re-certification)? (Please explain.)

VALIDATION POLICY AND PRACTICES

Comments
P&P 1. / How long can a model be in use prior to validation?
P&P 2. / How often do you revalidate a model? Does it depend on whether the model is classified as “high impact”?
  • Target

  • Actual

P&P 3. / How is the responsibility of performing “independent” model validations assigned in your bank (segregation of duties)?
P&P 4. / Which of the following procedures does your institution use in reviewing and testing performance results and key modeling components?
  • Theory and conceptual foundation

  • Assumptions

  • Calculations and statistics (functions and models)

  • Output and results

  • Documentation and procedures review (conceptual, development, validation/testing Results, operations, change control and business recovery plan)

  • Other procedures used in your validation program (define)

P&P 5. / Which of the following practices does your institution use?
  • Stress testing is a way to produce alternative scenarios using sensitivity analysis. Stress tests make risks more transparent by estimating the potential losses on a portfolio in abnormal markets to assess the vulnerability of a portfolio to major changes in the economic environment or to exceptional but plausible events.

  • Sensitivity analysis determines the extent the model output varies with the change of (a) input parameters, (b) calculations, and (c) underlying assumptions. It is done by changing the values of parameters either systematically or randomly, then measuring differences in the outputs.

  • Backtesting means the use of statistical methods to compare estimates of model components and/or performance risk/value measures to realized outcomes.

  • Benchmarkingrefers to a comparison of internal estimates across banks and/or with external benchmarks (e.g. external ratings, vendor models or models developed by supervisory authorities).

  • “Fit for Use.” In addition to an evaluation of the ability of the model to estimate accurately (loss estimates, valuation, and forecast), validation comprises an evaluation of the model operations and processes. This involves important issues like data quality, the internal reporting, how problems are handled and how the model outputs are used by management. It also entails the training of personnel and a uniform application of the model across the organization. Although quantitative techniques are useful, especially for the assessment of data quality, validation of the model operations and processes is mainly qualitative in nature and should rely on the skills and experience of typical banking supervisors.

  • Other validation procedures. (Please explain what they are and how they are used.)

P&P 6. / Do your model validation results indicate a risk or compliance rating or score? (Please explain.)
P&P 7. / Do you use a model validation “opinion” in your validation reports to management and Committee to emphasize the condition of the validation findings or validation performed? (Please explain the opinions and how they are selected for the results of the validation.)

1