Model validation in AML - a step too far!

Model validation, a term very familiar to teams on the credit risk side of things at banks. Lately this model validation has started making it's way into the compliance/AML space (primarily after the 504 rule came into being in US). More and more regulators are jumping on this bandwagon and so are the banks as they need to comply. I have a view point on this topic which I will try to share in this article. I will suggest you read with an open mind and think along as I believe such exercises are unnecessarily slowing us down in fight against criminals.

What is it?

If you are new, believe me in the near future your regulator will come knocking at your door and request such a validation report. Let me start with the basics. Assuming you have a Transaction Monitoring (TM) solution in place, you are well aware this TM system consumes your back-office data, processes this data and runs algorithms to generate alerts. These alerts are further assessed by your alert handlers/investigators to identify suspicious ML/TF activities. All the processes underlined earlier have a potential to have missed certain elements, got manipulated/incorrectly coded or just wrong. Model validation helps in finding out loopholes in the technical chain of events - whether these steps have been implemented correctly and perform as intended.

What is the issue then?

"Conceptually" model validation exercise sounds like a great idea! This can ensure you have covered any surprise well before the regulators have a chance to find something odd. As mentioned earlier, I do have some reservations around this exercise which I will list down in following paragraphs.

A bit Too early - One of my biggest concerns is, are we a bit too early as an industry to spend time and resources on this heavy duty exercise. Majority of banks still have a basic TM solution in place. A significant number of banks are in the process of performing risk assessments and linking them to TM rules (again mandated by regulators) while only a handful of banks are in the advanced stages to bring in innovation. It requires a significant amount of time/resources/intelligence to manage a better performing TM solution. While doing so, a thorough checks and balances exercise is performed in roll-out of TM solution during your UAT/testing cycles. Model validation goes a bit deeper (and independently) to ensure what you are doing is indeed correct and complete. It's more or less overlapping to the exercise you may have already performed, so why the rush? Keep in mind this concept of model validation originated from US where having a rule based control around AML is much easier to validate than a risk based control (as in EU and other parts of the world).

Time investment - is it worth it? Keep in mind model validation exercise is no joke. You have to invest a lot of time and energy in digging through every rabbit hole and find out anything which might have gone wrong or is incorrect. Take for example, you need to ensure all your product/transaction makes it through to the TM solution, the values are not changed on the way, any and every transformation is recorded and the reconciliation is complete. Then inside the TM solution, you need to re-validate (as if you did not have any UAT report) all the TM rules, their working, their results, the thresholds used and more. As I mentioned earlier, you are redoing the same exercise which you might have done during the installation of the TM solution in the first place. This makes me wonder, is it worth the exercise to find out potentially one field in one table might be sending value X instead of Y? I think going by the 80-20 rule should suffice in the TM space.

Third party dependence - This being a thorough and independent exercise, more often than not, banks are relying on third parties to perform this exercise. This is a good thing and helpful too as you do need a fresh pair of eyes. The only challenge is, while this will be a thorough job you end up with a 500+ pager deliverable from this third party. For them it's a great achievement, but for you (as a bank), you will end up with a laundry list of loopholes which you have to answer, correct and fix within a timeframe. My personal experience has been that you spend lots and lots of time clarifying mis-interpretations rather than solving any real problem.

Missing the bigger picture - I think we are at a cross-roads in the industry where there are two extremes in TM space - a fancy AI/Machine Learning solution which can apparently solve all your problems OR get so much governance and paper work (such as model validation) in place that it does give confidence to the senior management but nothing tangible happens on ground. The problem with the second approach is that we end up missing the bigger picture. All paperwork/documentation/validations are fine but we still need to keep focus on the primary objective - empower your teams to perform their best in fighting criminals. I keep on hearing experienced staff is getting more scarce than ever. If they end of spending time in such kind of exercises - you know what can go wrong!

Time to introspect - I call upon the regulators to introspect on this topic too, particularly the ones who give the flexibility or risk based approach in monitoring. On one hand you want to offer flexibility (which is required) but at the same time tie the hands to prove+document each and every decision taken to allow such flexibility. You need to assess the maturity level of the banks who come under your umbrella before guiding/instructing them to perform "good-to-have" exercises such as model validation. There are already too many programs running at several banks and this addition will only make matters worse.

I can keep on going on an on but I hope I was able to make my point a bit more clearer by raising the challenges in this space. I will leave this discussion open-ended and allow you, as a bank, to think it through. There is no black/white solution to this new initiative of "model validation". However you need to be very much aware on what you are getting into and is it really necessary at the stage you are in the process of having a mature TM control.

Conclusion

I can imagine nothing will change just because I shared this piece, but having seen different phases the AML/CFT process has gone through, my experience says we as an industry need to introspect. We have to focus our energy on what is more important and critical rather than dedicating our time in solving "good-to-have" and "what-if" scenarios. Credit risk models have been there since eternity and have gone through different maturity cycles and hence having such a validation makes sense. AML on the other hand is still a new born process (relatively speaking). We need to trust our work force and channel their energies where it's most required - identifying and fighting criminals. If not, as the header image suggests, we will be dragging behind unnecessarily and the criminals will have a free run!