Validation
AI & ML

AI & ML in financial services

Artificial Intelligence (AI) applications and Machine Learning (ML) powered models are increasingly used in traditional financial institutions (incumbent banks) as well as FinTech firms, where the business model often exclusively depends on AI and ML.

Examples of AI and ML applications in finance:

Deployed Models

  • Credit scoring and loan monitoring
  • Fraud and suspicious transaction monitoring, AML
  • Client recommendation systems (marketing)

Ongoing adaptation

  • Investment advisory and personalized banking services
  • Trading and order execution
  • Cybersecurity
  • HR
  • ML-powered challenger models
  • Data cleaning and reconciliation

Cutting-edge applications

  • Derivatives pricing with AI
  • Market risk management
  • Sentiment analysis of news, social media, documents, external and internal communications

Why Probability?

  • Probability & Partners has long-standing operational expertise in building and deploying ML models.
  • This also gives us a unique edge in the ability to validate such models.
  • We can help your organization with full-stack validation of AI or ML solutions irrespective of the area of application.
  • We create a tailormade validation that best suits the materiality, importance for business, and stage of deployment of your AI/ML model(s)
  • We can help you not only with analyzing algorithms for biases but also with advising on the optimal trade-off between performance and bias.
  • Probability & Partners has experience with so-called “black-box” validations performed without obtaining the complete model documentation or source code. Such a setup is ideal for IP-sensitive validation assignments.
  • Probability & Partners can advise your organization on the ethical aspects of developing and using AI and ML solutions.

Increased model risk in AI/ML solutions

AI and ML models provide cutting-edge solutions with potentially large upside. However, they also exacerbate model risk due to their increased complexity. Although standard model validation frameworks can be applied to ML solutions, their specific risks need significant adjustments to the model validation framework.

Additional high-risk elements of ML models:

  • Dependence on large quantities of reliable data;
  • Potential gender, age, and other biases leading to substantial reputational risk;
  • Challenges in implementation, e.g., large computational and database requirements that would prevent efficient deployment.

Specific validation components

Alongside typical validation, our AI/ML validation contains the following specific components:

  • Data policy: availability, quality, adequacy/representativeness,  biases;
  • Actual modeling choices (choice of the ML model and possible alternatives);
  • Outcomes’ interpretability; 
  • Potential biases towards protected attributes and their mitigation. We can help you to find the right balance between fairness and model performance;
  • Hyperparameters choice and sensitivity;
  • Benchmarking to industry-standard (classification, regression) models;
  • Model deployment and operational assessment.

Black box fintech solutions validation

  • Probability & Partners has experience in validating Fintech’s “unique selling point” AI and ML solutions.
  • For IP-sensitive validation assignments, we execute them without obtaining the full model documentation or source code.
  • In such validation assignments, the model is treated as a “black box,” and detailed diagnostics are performed by:
    • Applying it to actual or synthetic (artificial) input data
    • Building challenger models and comparing model performance
  • We also assess your data policy, deployment and operational characteristics, cybersecurity, compliance to regulations such as GDPR and AI Fairness, and other material or reputational risks.

Examples of Assignments

Innovative FinTech firm

Our team performed a validation of the labor market model to be used for credit applications. In addition to technical model validation, we have also advised the firm on the business model.

Ambitious Dutch FinTech start-up

Probability & Partners validated the ML model for credit scoring of the fintech start-up firm. We structured the validation as a “black-box” exercise to safeguard intellectual property. We also reviewed the business model given the ML model performance.

Research

Contact