Research Insights

The Most Dangerous AI Models Aren’t the Wrong Ones. They’re the Overconfident Ones

Published: August 2026

A few years ago, if you had asked me what makes a machine learning model trustworthy, I probably would have pointed to accuracy.

Like many people working in AI, I spent a lot of time looking at performance metrics. Accuracy, precision, recall, ROC-AUC, RMSE. The higher the score, the better the model.

At least that was the assumption.

But over time, I started to realise that there is a more important question hiding beneath all those numbers:

What happens when a model is wrong, but completely confident that it is right?

That question ultimately led me into the world of uncertainty in machine learning.

One of the things that fascinates me about modern AI is that we have become incredibly good at building systems that make predictions. We can predict customer churn, detect fraud, estimate property values, identify medical risks, and automate decisions at a scale that would have seemed impossible a decade ago.

Yet many of these systems behave like that one person in every meeting who always has an answer, regardless of whether the answer is correct.

The confidence is there.

The humility is not.

And that can be dangerous.

Think about a doctor making a diagnosis. Most of us would rather hear:

"I think this is the correct diagnosis, but there is some uncertainty."

than:

"I am 100% certain."

The first response creates trust because it recognises limitations. The second can become risky if that certainty turns out to be misplaced.

I increasingly believe that AI systems should be held to the same standard.

What surprised me most while exploring uncertainty was how differently traditional statistics and modern machine learning approach the problem.

Traditional statistical models have always taken uncertainty seriously.

When statisticians provide an estimate, they rarely stop there. They usually provide confidence intervals, margins of error, or measures of variability. In other words, they communicate not only what they think is happening, but how sure they are about it.

Machine learning often takes a different approach.

The focus is usually on predictive performance. If a model achieves impressive accuracy, it is considered successful.

But accuracy alone does not tell us whether a particular prediction should be trusted.

Those are not the same thing.

A model might correctly classify thousands of observations while still being highly overconfident about the few occasions when it fails.

The more I worked on this topic, the more I realised that there is an important difference between:

Being right

and

Knowing how likely you are to be wrong.

That distinction matters in almost every real-world application of AI.

In healthcare, uncertainty can help clinicians determine when additional tests are needed.

In fraud detection, uncertainty can prevent legitimate customers from being incorrectly flagged.

In customer churn prediction, uncertainty can help organisations prioritise intervention more intelligently.

In autonomous systems, uncertainty can be the difference between safe behaviour and catastrophic mistakes.

The prediction itself is only part of the story.

Understanding confidence is what transforms a prediction into a trustworthy decision.

One lesson that stayed with me came from comparing different machine learning approaches.

Some models produced very precise predictions but were less reliable in terms of coverage. Others provided broader prediction ranges but captured reality more consistently.

Initially, I assumed narrower predictions would always be better.

Now I am not so sure.

Imagine two weather forecasters.

One constantly gives very precise forecasts but gets caught out by unexpected changes.

The other provides slightly wider prediction ranges but is rarely surprised.

Which forecaster would you trust more?

That is essentially the same challenge we face with uncertainty estimation in AI.

A system that appears precise is not necessarily a system that is trustworthy.

Perhaps the biggest change in my thinking is that I no longer see uncertainty as a technical feature reserved for statisticians and machine learning engineers.

I see it as a product feature.

Users deserve to know when an AI system is uncertain.

Organisations deserve to know when a prediction carries elevated risk.

Decision-makers deserve more than a single score or classification.

As AI increasingly moves into high-impact environments, uncertainty should become part of the conversation from the beginning rather than being bolted on at the end.

Today, my interest in uncertainty extends beyond machine learning performance. It connects directly to my broader research interests in trustworthy AI, algorithmic accountability, and the governance of increasingly autonomous systems.

As AI becomes more capable, its decisions will influence more aspects of everyday life. The challenge is not simply building systems that can make decisions.

The challenge is building systems that understand the limits of those decisions.

I have come to believe that trustworthy AI is not defined by perfection. No model is perfect.

Trustworthy AI is defined by honesty.

The ability to say:

"Here is my prediction. And here is how confident I am in it."

That may sound simple, but I suspect it will become one of the most important capabilities any AI system can possess.

Because if an AI system makes a prediction, shouldn't it also tell us how confident it is?

About the Author

Dr Iyanu Emmanuel Olatunbosun is an interdisciplinary researcher specialising in Responsible AI, FinTech innovation, digital financial inclusion, and sustainable digital transformation.

Related Research 

Olatunbosun, I. E., & Oparah, F. C. (2026). “Comparative Evaluation of Uncertainty Measures in Traditional Regression and Supervised Machine Learning Models”. International Journal of Hybrid Intelligent Systems. SSRN preprint. Under review. http://dx.doi.org/10.2139/ssrn.7094438

What Working in Banking Taught Me About AI, Risk, and Trust

Published: June 2026

When I first started working in the financial sector, I was surprised by how many important decisions were already being influenced by intelligent systems. At the time, I simply saw them as useful tools and technological improvements. Over time, I became increasingly curious about how artificial intelligence was quietly changing the way financial institutions identify risks, make decisions, and serve customers.

I still remember being struck by how quickly certain decisions in banking were being made. Fraud alerts appeared almost instantly, unusual transactions were flagged before customers noticed them, and risk assessments that once required significant manual effort were happening quietly in the background.

The more I explored that question, the more I realised that banking has quietly become one of the world's most active adopters of artificial intelligence. From fraud detection and credit scoring to regulatory compliance and transaction monitoring, AI is no longer an experiment in many institutions. It is becoming part of the infrastructure. Financial organisations are using tools such as machine learning models, robotic process automation, behavioural biometrics, natural language processing, and AI-enhanced risk management systems to support critical functions across their operations.

What stood out to me was not simply the scale of adoption.

It was the shift in mindset.

Traditionally, risk management was often reactive. Something happened, and the organisation investigated it. A suspicious transaction was reviewed after it occurred. A compliance issue was identified after an audit. A customer defaulted before warning signs were fully understood.

AI is changing that dynamic.

Instead of asking, "What went wrong?" organisations are increasingly asking, "What is likely to go wrong next?"

That difference is profound.

Imagine driving a car that only tells you about a hazard after you've already hit it. Now imagine a vehicle that continuously scans the road ahead and warns you before the danger appears. That is essentially the transition taking place in risk management.

One thing I kept coming back to throughout my research was the growing confidence financial professionals have in data-driven systems. Many now trust machine learning tools to process vast amounts of information more quickly and consistently than traditional manual approaches. AI systems can analyse transaction patterns, flag unusual activity, support compliance checks, and identify emerging risks in real time. What once required large teams and lengthy review processes can now happen continuously in the background.

Fraud detection is perhaps one of the most visible examples. A human analyst might struggle to review millions of transactions. An AI system can monitor those transactions simultaneously, looking for subtle behavioural patterns that may indicate something unusual. The result is not just speed. It is the ability to recognise signals that might otherwise be missed.

The same applies to regulatory compliance. Tasks such as Know Your Customer (KYC) and Anti-Money Laundering (AML) checks are increasingly being supported by AI-enabled systems that help institutions manage regulatory obligations with greater efficiency and consistency. [

But perhaps the most important lesson I learned is that technological capability is not the same thing as good judgement.

Just because an AI system can make a decision doesn't mean it should make that decision alone.

That may be where the real conversation begins.

The benefits of AI are easy to see. The challenges are often less visible.

Many AI systems learn from historical data. If historical decisions contain bias, those biases can find their way into future recommendations. A lending model trained on imperfect data may unintentionally disadvantage certain groups. A risk model may produce outcomes that appear objective while silently reflecting decisions made years earlier. Concerns around algorithmic bias, explainability, accountability, and privacy remain very real challenges for financial institutions.

The issue is not that AI makes mistakes.

Humans make mistakes too.

The issue is understanding how decisions are being made and who remains accountable when things go wrong.

I often think about this through the lens of trust.

When a customer applies for credit and receives a decision, should they be able to understand why that decision was made?

I believe they should.

When a transaction is flagged as suspicious, should there be a clear path for human review?

Again, I believe there should.

Technology should strengthen trust, not replace it.

The most effective financial institutions should not only be those that simply automate the most processes. They should be the ones that combine technological innovation with transparency, governance and human oversight. AI can help identify risks, but people must still define values, exercise judgement, and take responsibility for outcomes.

As I reflect on the future of banking, I am optimistic. AI has enormous potential to improve risk management, strengthen compliance, enhance fraud prevention, and support better decision-making. Yet the future of financial services will depend not only on how intelligent our systems become, but also on how responsibly we choose to use them.

The future of banking will not be shaped by how much AI we adopt, but by how wisely we choose to use it. And that leaves me with a question I continue to reflect on: as artificial intelligence becomes more intelligent, are we becoming more thoughtful about the responsibility that comes with it?

About the Author

Dr Iyanu Emmanuel Olatunbosun is an interdisciplinary researcher specialising in Responsible AI, FinTech innovation, digital financial inclusion, and sustainable digital transformation.

Related Research (Published 2025)

Olatunbosun, I. E., & Olatunbosun, A. R. (2025). Artificial Intelligence and Risk Management in Financial Institutions: Evidence from the UK Banking Sector. EthAIca, 4, 436. https://doi.org/10.56294/ai2025436