Our team performed a validation of the labor market model to be used for credit applications. In addition to technical model validation, we have also advised the firm on the business model.
AI and ML models provide cutting-edge solutions with potentially large upside. However, they also exacerbate model risk due to their increased complexity. Although standard model validation frameworks can be applied to ML solutions, their specific risks need significant adjustments to the model validation framework.
Additional high-risk elements of ML models:
- Dependence on large quantities of reliable data;
- Potential gender, age, and other biases leading to substantial reputational risk;
- Challenges in implementation, e.g., large computational and database requirements that would prevent efficient deployment.
- Probability & Partners has experience in validating Fintech’s “unique selling point” AI and ML solutions.
- For IP-sensitive validation assignments, we execute them without obtaining the full model documentation or source code.
- In such validation assignments, the model is treated as a “black box,” and detailed diagnostics are performed by:
- Applying it to actual or synthetic (artificial) input data
- Building challenger models and comparing model performance
- We also assess your data policy, deployment and operational characteristics, cybersecurity, compliance to regulations such as GDPR and AI Fairness, and other material or reputational risks.
Examples of Assignments
Probability & Partners validated the ML model for credit scoring of the fintech start-up firm. We structured the validation as a “black-box” exercise to safeguard intellectual property. We also reviewed the business model given the ML model performance.
- AI Fairness in Financial Services: How to quantify and improve fairness in machine learning and AI applications?
- AI Fairness in Financial Services: Trust at the Center
- Machine Learning in Finance: Now and looking ahead
- Machine Learning for Credit Applications: A Recent EBA discussion paper
- Fairness in AI and ML: Part II