Model risk management: Defining a model, governance and analytics

Model risk management: Defining a model, governance and analytics

Jon Hill, Global Head of Model Risk Governance at Credit Suisse provides insights in to the usage of model development, processes of model validation policy, the best practices for validating quantitative models and the role of stress testing over the next 6-12 months and more. 

  1. We look forward to you leading a Masterclass at the upcoming Risk Americas 2017 Convention: Model Risk Governance and Validation Best Practices, May 25. What are the key topics and highlights which will be discussed during the Masterclass?

If anyone had asked me two or three years ago how model risk management is performed at leading financial institutions, I would have said through the creation of independent model validation groups. That is no longer a complete answer. More recently US regulators have been raising the bar and the level of model risk management to a higher level by mandating rigorous model governance as the over-arching framework for effective model risk management. The masterclass will include a lead-off presentation on model risk governance (what exactly is model governance, how is it crafted and implemented at leading institutions, etc.) followed by presentations on various aspects of model validation, including an examination of the requirements set forth by SR11-7/OCC 2011-12 guidance. I will also give presentations on best practices for dealing with model owners & stakeholders during a validation engagement and how to prepare for a regulatory bank exam.

  1. What, in your opinion, are some of the key considerations for model development and its usage?

The requirements for development of advanced quantitative models used in finance have evolved rapidly over just the last several years. It is no longer acceptable for model developers to deliver code for a model to IT and then disconnect from the process. Quantitative models must be documented to a greater degree of precision than ever before by the developers (who are usually reluctant to make the effort to write good documentation) and the documentation has to address all of the aspects of the model as described in SR11-7. Among these are input data quality and governance, testing (including stress testing to CCAR shock levels), a complete list of assumptions and weaknesses of the model and a testament of the appropriateness of the model for the intended applications (as well as annotation of applications not suitable for the model). A key requirement most often overlooked is developmental evidence. Many practitioners interpret this SR11-7 requirement to be the set of tests the developers used to confirm that the model is implemented correctly and performing as intended. But this only part of the developmental evidence requirement. Model developers must also document the thought processes that informed the final model selection and in particular should devote some part of the documentation to justify why a particular modeling approach was chosen over alternative candidates. As an aside, validators must realize that the ‘best’ model for a given application is not always the ‘right’ model to use. I will delve more deeply into this apparent contradiction between ‘best’ and ‘right’ during the masterclass.

  1. Can you define model risk governance, and what this is, in comparison to model validation?

Probably the best way to understand the distinctions between model risk governance and model validation is in terms of the standard model life cycle: identification, development & implementation, independent validation, ongoing monitoring and annual reviews, and lastly model retirement. Validation sits squarely in the middle of the model life cycle as the critical Second Line of Defense (the first LOD being development and the third LOD being internal audit). Governance, on the other hand, impacts in some way every phase of the model life cycle, including validation, by setting standards and enforcing policy. It that sense governance sits above and reigns over the entire model life cycle while validation is one step in the middle of that cycle.

  1. Can you briefly tell the audience the processes of model validation policy and the best practices for validating quantitative models?

In the current regulatory environment, model validation policies must be fully compliant with the requirements of SR11-7. While SR11-7 officially applies to US conforming bank and non-US banks doing business in the US, many European financial firms have adopted SR11-7 as their standard as well. SR11-7 describes what must be included in a rigorous quantitative model validation but it is not highly prescriptive in how the various stages of validation are to be carried out such as assessment of the conceptual soundness of the model, description of the development evidence for the model, additional independent testing performed by the 2nd LOD, and most importantly, how to pose ‘effective challenge’. These requirements and industry best practices for fulfilling them will be addressed during several sessions of the masterclass.

  1. What are the key considerations for FIs when developing effective model validation methodologies and ensuring they are adopting best practices to address the many challenges posed by the requirements for posing ‘effective challenge’ during the validation?

Probably the most straightforward way is an exercise that is being applied at many compliant banks: performing a detailed gap analysis between the 2nd LOD’s model validation template and SR11-7, mapping each requirement from the FRB’s guidance to the section of the validation template that addresses that requirement. If all of the validation requirements of SR11-7 can be mapped into a validation template and the template is fully populated during the validation, then validators will have increased confidence that they have posed effective challenge to the model.

  1. How do you see the role of the Stress Testing and Model Risk professional changing over the next 6-12 months, especially with increasing technological and political advances changing?

All US SIFIs (Significant Financial Institutions) are required to perform annual CCAR (Comprehensive Capital Analysis and Review) stress testing to ensure that they have adequate capital reserves to navigate and survive another market meltdown such as the 2008 financial crisis. Each year the bar is raised with respect to CCAR model development, validation and stress testing requirements. For example, two years ago the FRB-supplied model input shocks included the possibility of negative interest rates for the first time and this created numerous failures and challenges for models that implicitly assumed rates could only be positive. The increasing demands on modelers, validators and the models used for CCAR has created a strong demand at many firms for Model Risk professionals, especially those with previous CCAR experience. This is truly a golden age for model risk managers in terms of job opportunities and enhanced visibility to senior management. Many firms, including Credit Suisse, are raising model risk management to the same level of importance and visibility as the traditional triumvirate of market, credit and operational risk. Model Risk management is rapidly becoming the fourth leg of enterprise wide risk management at leading financial firms.