What’s Model Risk and Why Does it Matter?

With the big data revolution of recent years, predictive models are being rapidly integrated into more and more business processes. This provides a great amount of benefit, but it also exposes institutions to greater model risk and consequent exposure to operational losses. When business decisions are made based on bad models, the consequences can be severe. The stakes in managing model risk are at an all-time high, but luckily automated machine learning provides an effective way to reduce model risk.

Prior to the financial crisis of 2008, Model Risk Management within the Banking industry was driven by industry best practices rather than regulatory standards (which brings to mind the saying “a fox guarding the hen house”). However, after the financial crisis, financial regulators around the world stepped up to the challenge of reigning in model risk across the financial industry.

 

Now, regulation is being targeted towards much smaller banks in the U.S.

In 2011, the Federal Reserve Board (FRB) and the Office of Comptroller of the Currency (OCC) issued a joint regulation specifically targeting Model Risk Management (respectively, SR 11-7 and OCC Bulletin 2011-12). This regulation laid the foundation for assessing model risk for financial institutions around the world, but was initially targeted towards Systemically Important Financial Institutions (a.k.a., SIFIs), which were deemed by the government to be "too big to fail" during the Great Recession.

Now, regulation is being targeted towards much smaller banks in the U.S. The Federal Deposit Insurance Commission (FDIC) recently announced its adoption of Supervisory Guidance on Model Risk Management, previously outlined by the FRB and OCC. The FDIC’s action was announced through a Financial Institution Letter, FIL-22-2017. This is huge. The new regulation greatly reduces the minimum threshold for compliance for banks from $50 billion to $1 billion in assets. This will require large capital investments from regional and community banks to ensure alignment to regulatory expectations; something that the SIFIs have a very long head start on.

 

If there is any doubt on the classification of a process, regulators wanted to encourage banks to err on the side of “model.”

At the heart of this regulation is the notion of “model risk.” You might be thinking what is model risk, and how can it be mitigated? This is a complicated question, but before we dive in to model risk, I have another simpler question that must be answered first. What is a model? The regulators have provided a universal definition that has been adopted across the financial industry. They define a model to be “a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates.”

 

 

Figure 1: The main components of a model as defined by banking industry regulators.

 

 

Therefore, if a process includes inputs, calculations, and outputs then it falls under the regulatory classification of a model. This is a broad definition, but since the intent was to mitigate model risk, a broad definition of a model was established to maximize the impact of the regulation. If there is any doubt on the classification of a process, regulators wanted to encourage banks to err on the side of “model.”

With the definition of a model now in place, the regulation next defined model risk as “the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports.” In other words, model risk can lead to tangible losses for the bank and its shareholders. Regardless of where a bank is using a model in their enterprise, model risk primarily occurs for two reasons:

  1. A model may have been built as it was intended, but could have fundamental errors and produce inaccurate outputs when compared to its design objective and intended use; or,
  2. A model may be used incorrectly or inappropriately, or its limitations or assumptions may not be fully understood.

 

 

Regardless of the technology at the disposal of the model developers or model validators, there is no replacement for a sound model governance process.

The need for an effective Model Risk Management (MRM) framework can be demonstrated with countless case studies of recent MRM failures. For example, Long Term Capital Management was a large hedge fund led by Nobel laureates of economics and world class traders, but ultimately failed due to unmitigated model risk. In another recent example, a large global bank’s misuse of a model caused billions of dollars in trading losses. The details of these examples are often the topic of Business School case studies and debate, but there is no arguing that model risk is very real and must be managed. But, how?

The FDIC’s new regulation can be broken down into three main components used to manage model risk:

  • Model Development, Implementation, and Use - The initial responsibility to manage model risk is on those developing, implementing, and using the models. Model development relies heavily on the experience and judgment of developers, and model risk management should include disciplined model development and implementation processes that align with the model governance and control policies.
  • Model Validation - Prior to the use of a model (i.e., production deployment), it must be reviewed by an independent group - Model Validation. Model validation is the set of processes and activities intended to independently verify that models are performing as expected, in line with their design objectives and business uses. The model validation process is intended to provide an effective challenge to each models’ development, implementation, and use. The model validation process is crucial to effectively identify and manage model risk.
  • Model Governance, Policies, and Controls - Strong governance provides explicit support and structure to risk management functions through policies defining relevant risk management activities, procedures that implement those policies, allocation of resources, and mechanisms for testing that policies and procedures are being carried out as specified. This includes tracking the status of each model on an inventory across the entire enterprise.

 

Traditional model development methods are time-consuming, tedious, and subject to user error and bias.

Initial alignment to these new regulatory requirements required SIFI banks to invest millions of dollars to build new processes and teams, and now that same burden lays with community and regional banks. It is impossible to over-emphasize the need for an institution to have sufficient model governance, policies, and controls. Regardless of the technology at the disposal of the model developers or model validators, there is no replacement for a sound model governance process. But, isn’t there a more efficient way to use technology to reduce model risk, while increasing the transparency and auditability of the model development, implementation, and use process? Here at DataRobot we think that the answer to that is an unequivocal “YES!”

Traditional model development methods are time-consuming, tedious, and subject to user error and bias. Instead of manually coding steps (such as variable selection, data partitioning, model performance testing, model tuning and so on), best practices can be automated through the use of  automated machine learning. Automated machine learning allows for easy replication of the model development process, which gives model validators more time to independently assess and review the model and its potential limitations, and ultimately drives value for the validation process.

 

The new field of automated machine learning offers a much stronger framework for model development and validation than traditional manual efforts, while more closely aligning to the ever-increasing regulatory requirements and vastly reducing “model risk.”

In other words, automated machine learning produces both highly accurate and highly interpretable models. As my colleague, Colin Priest, summed it up in a recent article, with the help of automated machine learning, there is no longer a trade-off between model accuracy and interpretability. You can in fact have your cake and eat it too! The new field of automated machine learning offers a much stronger framework for model development and validation than traditional manual efforts, while more closely aligning to the ever-increasing regulatory requirements and vastly reducing “model risk.”

Automated machine learning delivers the tools to optimize and accelerate model risk management, making it easier for banks of all sizes to gain value from a robust model risk management framework.

 New Call-to-action

About the Author:

As the head of Model Risk Management at DataRobot, Seph Mard is responsible for model risk management, model validation, and model governance products, as well as services. Seph is leading the initiative to bring AI-driven solutions into the model risk management industry by leveraging DataRobot’s superior automated machine learning technology and product offering. Seph has more than 10 years of experience working across different banking and risk management teams and organizations. He started his career as a behavioral economist with a focus on modeling microeconomic choices under uncertainty and risk, then transitioned into the financial services industry. Seph is a subject matter expert in model risk management and model validation.