What are Machine Learning models and how do you test Machine Learning models

What are Machine Learning models and how do you test Machine Learning models

By Jacob Kosoff, Head of Model Risk Management & Validation, Regions Bank

Best Practices on Understanding and Validating Machine Learning Model – Day 1 – 1:50pm
May 14, 2019 is Day One of Risk Americas 2019. That day Jacob Kosoff, the head of Model Risk Management and Validation at Regions Bank, will teach the best practices in understanding and validating machine learning models.

This machine learning session will discuss the most common and promising uses of ML in banking as well as the heightened model risks that require additional transparency, testing and monitoring to effectively utilize ML models.

What is ML?
Machine Learning models are a subset of artificial intelligence (AI) models and are the most commonly used AI models in banking applications. Machine Learning (ML) using structured data that predicts the relationships between drivers and outcomes is the most commonly used form of ML to replace regression or rules-based systems in banking today.  ML provides systems with the ability to learn from data and build models without specifying rigid parametric forms which typically requires somewhat human intensive involvement to specify or select proper forms. There are many different approaches in ML to solve specific problems. One methodology may perform better than others depending on the nature of the data and its implicit complexity.  Building complex ML models involves selection of hyperparameters which can be both time consuming as well as computationally intensive.  With advances in computing power such as parallelism of algorithms (by breaking down computing tasks into smaller sub-tasks) as well as graphics processing unit (GPU) accelerators, the process of hyperparameter tuning has become more automated. Since the functional forms of the models are very flexible, the models can learn from the data to identify new patterns through model retraining with minimum human intervention. However, this does not eliminate the responsibility to understand the model drivers and any changes in implementation.

Testing ML models
The code for a ML model is complex and requires significant testing at implementation. ML models require frequent assessment of detailed, granular performance monitoring; regular assessment of model fit and variable relationships; and assessment of model updates to include transparency into changes in key variable relationships and performance.  Human intervention is required for all of these assessments in banking today.

ML offers a great benefit when added to a suite of model methods, being utilized for many applications where traditional methods are not able to tease out relationships in the data. Many vendors are offering ML solutions in new model versions or updates to remain competitive in their applications.

However, this does not preclude the process of selecting the best model method(s) for the information value in the available data, business requirements and performance expectations. When a ML model is selected as the best methodology, there should be heightened requirements around transparency, model development rigor, initial and ongoing testing as well as governance around model changes or updates.

The complexity and widespread development and use of ML models introduce significant additional model risk to the enterprise, and heightened attention on model risk management should be applied.

The opinions expressed in the article are statements of the author and are intended only for informational purposes, and are not opinions of any financial institution and any representation to the contrary is expressly disclaimed.