Advanced Model Risk Agenda

Session details 

• Reviewing regulatory expectations for model risk teams
• Expectations across different sized organizations
• Uses of automation, AI and machine learning
• Global regulatory requirements and lessons learnt across jurisdictions
• Revisions to model risk guidelines to include AI and machine learning
• Application of SR11-7 to evolving model risk programs

Session details 

• Explosion of model inventories and new methodologies
• Managing data limitations
• Best practice to manage evolution of inventory
• Methodologies and controls variations across inventory
• Inclusion of non-quantitative models
• Model components across entire risk appetite
  • Moving beyond financial models
• Updates and modifications to policies and procedures to accommodate change

Session details 

• • • Maturity of model risk function
    • Automating model risk practices
    • Automating manual tasks and processes
    • Ensuring quality control and consistency
    • Maintaining standards to do more with less
    • Revising model risk frameworks to include AI and machine learning
    • Scaling up model risk in a fast-paced deployment environment
    • Speed and agility of model risk management

Session details 

• • Cybersecurity validation requirements
    • Threat identification and mitigation
    • Categorization criteria for models
  • OCC booklet inclusion of cyber security model component
  • Expansion of areas leveraging AI and machine learning
  • Technical expertise and talent
  • Understanding nuances and requirements across domains

Session details 

• • Success stories implementing an AI or machine learning approach
  • Managing model input and interpreting output
  • Transparency of uses of AI and machine learning
  • Regulatory appetite and expectations
  • Aligning with fair lending requirements
  • Understanding model risk requirements using AI and machine learning
  • Updating on a continuous and ongoing manner
  • Enhancing data governance programs for use of AI
    • Increased volume and complexity of data
  • Risk mitigants on AI and machine learning
  • Managing black box models with no access to data

Session details 

• Vendor compliance with model risk regulations
• Governance and oversight of vendor models
  • Transparency of AI models
  • Documentation for effective validation
• Visibility and transparency challenges
• Explainability of black box models
• Determining assurance and control processes
• Identifying use cases for black box models

Session details 

• Explaining decisions from input to output
• Privacy concerns with excess data collection
• Data requirements with fragmented programs
• Understanding which inputs drive certain outputs
• Development of AI best practice document – PPI/ Policy institute??
• Ensuring compliance with consumer laws and regulations
  • Integrating compliance teams
• Customer requirements for explainable AI
• Countering misconceptions on explainability
• Defining and understanding the difference between interpretable and explainable from an AI and machine learning perspective
• Separating explainability and fairness

Session details 

• Integrating fairness into credit models
• Model development perspectives to consider fairness from outset
• Managing Headline and reputation risk
• Determining appropriate metrics
• Collecting data for fairness testing
• Managing algorithmic fairness in uncertainty
• Understanding ethics and fairness of AI results

Session details 

Session details to come

Session details 

• • • • Adversarial testing
      • Empirical examples

Session details 

• Overview of Large Language Models
• Pros and cons of Large Language Models
• Use cases for LLMs in Model Risk Management

Session details 

• Analyzing bias in data sets
• Quantifying bias
• Reducing bias in data sets
• Aligning decisioning with ethics of organization
• Qualitative approaches to analyzing data bias
• Regulatory expectations and update on bias
  • CFPB and Fed exams
• Standardization to detect bias

Session details 

• The role of emerging technologies such as low-code/no-code platforms, open-source software, and trends in the evolution of EUCs
• The impact of analytic assets, including low-code/no-code and open-source applications, on model risk management
• Approaches for integrating EUC/analytic asset control functions into existing model risk management frameworks

Session details 

• Provision techniques to estimate impairments
• Regulatory expectations of management and judgement overlays
• Adjusting models in a fast-changing economic environment
• Mitigations or validations to make business users comfortable with models
• Frequency to update models in a fast-changing environment
• Impact of pandemic data and overlays on outlook
• Recalibrating or rebuilding models post pandemic
• Reactions to significant interest rate changes
  • Challenging assumptions of a different environment
• Managing unanticipated and unprecedented outcomes
• Identifying models predicated on low inflation
  • Modifying assumptions or deploying alternatives

Session details 

• Recalibrating or refitting models
• Reduction in model development timelines in an evolving environment
• Rules or triggers requiring a model risk deep dive vs. streamlined review
• Spotting performance deterioration or degradation
  • Identifying early indicators or triggers
• Sensitivity of machine learning models to change
• Identification before model is decommissioned
• Credit modeling in a downturn
• Scenario and sensitivity analysis
  • Planning ahead and identifying models requiring action

Session details 

• Best practices to get ahead of inflation as a potential long-term risk to business forecasting, analytics and models
• In the absence of sufficient data, what risk assessments we ought to be doing?
•  What models and analytics may be more vulnerable to a lengthy high inflation period?

Session details 

• We analyze of the impact of asset price bubbles on the markets for cryptocurrencies through applying the theory of local martingales.
• We find consistently across several widely traded cryptocurrencies that in isolation there is no evidence of a  bubble, but modeled jointly with an equity market index we do detect a bubble.
• We also apply the principle of relative entropy to measure the model risk from failing to detect a bubble through ignoring the the relationship to equity prices.
• We find that in most cases the model risk “multipliers”, used for establishing a model risk reserve

Session details 

• Developing transparency across the organization
• Aligning with data, audit and compliance
• Developing a control framework for advancing technology
• Criticality of integration across teams
• Auditing AI, machine learning and automation
  • Inclusion of technology audit team

Session details 

• Ensuring robustness of models exposed to different environments
• Model monitoring practices
• Ensuring robustness and resilience during validation
• Developing a proactive oversight approach
  • Integrating at development and validation