Responsible AI

The Most Dangerous AI Models Aren’t the Wrong Ones. They’re the Overconfident Ones

Published: August 2026

A few years ago, if you had asked me what makes a machine learning model trustworthy, I probably would have pointed to accuracy.

Like many people working in AI, I spent a lot of time looking at performance metrics. Accuracy, precision, recall, ROC-AUC, RMSE. The higher the score, the better the model.

At least that was the assumption.

But over time, I started to realise that there is a more important question hiding beneath all those numbers:

What happens when a model is wrong, but completely confident that it is right?

That question ultimately led me into the world of uncertainty in machine learning.

One of the things that fascinates me about modern AI is that we have become incredibly good at building systems that make predictions. We can predict customer churn, detect fraud, estimate property values, identify medical risks, and automate decisions at a scale that would have seemed impossible a decade ago.

Yet many of these systems behave like that one person in every meeting who always has an answer, regardless of whether the answer is correct.

The confidence is there.

The humility is not.

And that can be dangerous.

Think about a doctor making a diagnosis. Most of us would rather hear:

"I think this is the correct diagnosis, but there is some uncertainty."

than:

"I am 100% certain."

The first response creates trust because it recognises limitations. The second can become risky if that certainty turns out to be misplaced.

I increasingly believe that AI systems should be held to the same standard.

What surprised me most while exploring uncertainty was how differently traditional statistics and modern machine learning approach the problem.

Traditional statistical models have always taken uncertainty seriously.

When statisticians provide an estimate, they rarely stop there. They usually provide confidence intervals, margins of error, or measures of variability. In other words, they communicate not only what they think is happening, but how sure they are about it.

Machine learning often takes a different approach.

The focus is usually on predictive performance. If a model achieves impressive accuracy, it is considered successful.

But accuracy alone does not tell us whether a particular prediction should be trusted.

Those are not the same thing.

A model might correctly classify thousands of observations while still being highly overconfident about the few occasions when it fails.

The more I worked on this topic, the more I realised that there is an important difference between:

Being right

and

Knowing how likely you are to be wrong.

That distinction matters in almost every real-world application of AI.

In healthcare, uncertainty can help clinicians determine when additional tests are needed.

In fraud detection, uncertainty can prevent legitimate customers from being incorrectly flagged.

In customer churn prediction, uncertainty can help organisations prioritise intervention more intelligently.

In autonomous systems, uncertainty can be the difference between safe behaviour and catastrophic mistakes.

The prediction itself is only part of the story.

Understanding confidence is what transforms a prediction into a trustworthy decision.

One lesson that stayed with me came from comparing different machine learning approaches.

Some models produced very precise predictions but were less reliable in terms of coverage. Others provided broader prediction ranges but captured reality more consistently.

Initially, I assumed narrower predictions would always be better.

Now I am not so sure.

Imagine two weather forecasters.

One constantly gives very precise forecasts but gets caught out by unexpected changes.

The other provides slightly wider prediction ranges but is rarely surprised.

Which forecaster would you trust more?

That is essentially the same challenge we face with uncertainty estimation in AI.

A system that appears precise is not necessarily a system that is trustworthy.

Perhaps the biggest change in my thinking is that I no longer see uncertainty as a technical feature reserved for statisticians and machine learning engineers.

I see it as a product feature.

Users deserve to know when an AI system is uncertain.

Organisations deserve to know when a prediction carries elevated risk.

Decision-makers deserve more than a single score or classification.

As AI increasingly moves into high-impact environments, uncertainty should become part of the conversation from the beginning rather than being bolted on at the end.

Today, my interest in uncertainty extends beyond machine learning performance. It connects directly to my broader research interests in trustworthy AI, algorithmic accountability, and the governance of increasingly autonomous systems.

As AI becomes more capable, its decisions will influence more aspects of everyday life. The challenge is not simply building systems that can make decisions.

The challenge is building systems that understand the limits of those decisions.

I have come to believe that trustworthy AI is not defined by perfection. No model is perfect.

Trustworthy AI is defined by honesty.

The ability to say:

"Here is my prediction. And here is how confident I am in it."

That may sound simple, but I suspect it will become one of the most important capabilities any AI system can possess.

Because if an AI system makes a prediction, shouldn't it also tell us how confident it is?

About the Author

Dr Iyanu Emmanuel Olatunbosun is an interdisciplinary researcher specialising in Responsible AI, FinTech innovation, digital financial inclusion, and sustainable digital transformation.

Related Research 

Olatunbosun, I. E., & Oparah, F. C. (2026). “Comparative Evaluation of Uncertainty Measures in Traditional Regression and Supervised Machine Learning Models”. International Journal of Hybrid Intelligent Systems. SSRN preprint. Under review. http://dx.doi.org/10.2139/ssrn.7094438