Measurement and quantification of model risk: Aggregating model risk for a holistic view

By Apurva Anand, Director – Model Risk Management, Barclays

What, for you, are the benefits of attending a conference like ‘Model Risk Management Europe” and what can attendees expect to learn from your session?

A conference like Model Risk Management Europe accords participants an opportunity to interact with and learn from peers as well as industry leaders. It also acts as a platform for attendees to discover more about topical issues and emerging trends in Model Risk.
Model risk quantification is still in its infancy and I hope my session provides participants ideas to measure and quantify model risk.

In your opinion why is it important to measure and aggregate model risk using a holistic view?

I believe quantifying the risk associated with the use of models is one of the biggest challenges facing model risk managers today. There are 2 factors to note here:

Model interactions: Model risk/uncertainty for individual models can be quantified using techniques such as sensitivity analyses. However, that provides just a partial picture of the aggregate model risk that institutions face. Models interact with each other either through direct feeder-dependent relationship (e.g. loss and pricing) or indirect relationship (e.g. scenario and limits), which enhances model risk. Measuring aggregate model risk is important to ensure that the enhanced risk from these interactions is captured adequately.
Standalone models: models that do not interact with other models still pose a risk. The problem here is to find a common metric over which to aggregate the model risk arising from various models producing diverse output.
When measuring model risk from a holistic view what are some of the key metrics that influence the model? And how do they effect measuring the risk?

Aggregating model risk across a set of models that use different data, design and produce different quantities is complicated. Changing any of these components would change the model output – therefore a range of different model outputs is possible.

The simplest way to think about impact assessment is to think of it from the perspective of aggregate measures, such as – Net Income or CET1 ratio. This provides objectivity to the exercise and metrics to aggregate over. E.g., an application scorecard decides which customer to accept and who to reject. The model risk here is that the model decides to accept a bad customer (increased loss) or reject a good customer (lost revenue); thereby affecting the Net Income of the bank.

The key to successfully embed model risk quantification, however, is technology – infrastructure and databases. Where databases are connected and models are run on a single platform, aggregating model risk becomes a much easier task.

What factors are important in the quantification of model risk?

The most important consideration is that a one-size-fits-all approach might not work across a wide variety of models. The data, design and assumptions for each model is different. In addition, the output produced by different set of models vary. Therefore, understanding and quantifying idiosyncratic factors affecting individual models is important.

The other aspect to bear in mind is that model risk needn’t/wouldn’t be a number; it will be a range of outcomes to say the model output could be x% higher or lower. This flows from the previous point – a model is a combination of data, design and assumptions. Varying these factors individually or collectively will result in a range of output that in turn informs the model risk.

What does the future hold for model risk management and what do you see as a key emerging trend?

I think there are a couple of changes that will happen in the near future:

Transition of model validator to model risk manager
Model risk management being seen as enabler for adoption and use of models rather than impediment

Both of these changes will be driven primarily due to 2 emerging trends:

The maturing of the MRM function
New challenges (AI/ML) and opportunities (automation) requiring various stakeholders to work closer together.