Toni Martin
July 3, 2026 · 6 min read
This week I was sitting on a panel when someone in the audience asked a question I've been asked before. How do we leverage AI knowing that it is biased?
It's a good question. And it's one that most conversations about AI - particularly in founder and business circles - tend to skip over in favour of the more exciting stuff. The tools, the speed, the opportunity. The bias question gets a polite nod and then we move on.
I don't think we should move on quite so quickly.
A couple of weeks ago I submitted a passport application for my father. He is a black man. I took the photo, uploaded it and the automated system rejected it. The reason given was that his eyes appeared to be closed.
They weren't closed.
What struck me wasn't just the frustration of it - though it was frustrating. It was the familiarity. Because over five years ago I appeared on BBC Radio 5 Live's Wake Up To Money to talk about exactly this issue. AI systems rejecting passport photos of people of colour. The inability of the technology to accurately process images of darker skin. The real-world impact on real people trying to do something as ordinary as renew a passport.
That was over five years ago. And here I was, experiencing the exact same problem with my own father.
This isn't a recent discovery. Internal UK government documents released in 2019 revealed that the passport photo checking system had shown racial bias during testing - specifically that people with very dark or very light skin were disproportionately told their photos didn't meet the guidelines. The internal assessment at the time read: "User research was carried out with a wide range of ethnic groups and did identify that people with very light or very dark skin found it difficult to provide an acceptable passport photograph. However, the overall performance was judged sufficient to deploy."
Read that again. They knew. They deployed it anyway.
And the problem hasn't stayed confined to passports. Research published in 2025 found that AI diagnostic tools in dermatology show significantly lower accuracy for darker skin tones - meaning AI systems designed to help detect skin conditions are less reliable for the patients who may already face barriers to healthcare. The same pattern, a different context.
AI systems learn from data. They are trained on enormous datasets - images, text, decisions - and they identify patterns in that data to make predictions and judgments. The problem is that the data they learn from reflects the world as it has been, not the world as it should be.
If the images used to train a facial recognition system are predominantly of white faces - which has historically been the case, because the people building these systems and the datasets they drew on skewed heavily in that direction - the system becomes very good at recognising white faces and significantly less reliable with others. It's not that the system is malicious. It's that it has learned from incomplete and unrepresentative information.
This is what bias in AI actually means. It's not a rogue algorithm with an agenda. It's a system that reflects the gaps, assumptions and historical inequalities baked into the data it was trained on. And because those systems are then deployed at scale - making decisions about passport photos, loan applications, job applications, medical diagnoses - the bias gets amplified rather than corrected.
The data problem goes deeper than you might think
The training data issue is the root of it but it's not the whole picture.
Who builds these systems matters too. If the teams designing and testing AI tools are not diverse - if there aren't people in the room who would notice when a system consistently fails for darker skin tones - the problem is less likely to be caught before deployment. The UK passport example is instructive here: the disparity was identified in testing. It was noted. And then the decision was made that the overall performance was "sufficient." Sufficient for whom?
There's also the question of what the systems are optimised for. An AI system designed to minimise rejection errors overall might still perform poorly for minority groups if those groups are underrepresented in the dataset. The headline accuracy figures can look fine while masking significant disparities underneath.
Passport photo rejections are an inconvenience. But AI bias shows up in places where the stakes are considerably higher.
Facial recognition systems used by police are being run against passport and immigration databases in the UK - something that came to light in August 2025 - without parliamentary approval and without the public being informed. Searches against the passport system increased from just two in 2020 to 417 in 2023. These systems have documented accuracy problems for people of colour. The consequences of a misidentification in that context are not a minor inconvenience. They are potentially life-altering.
AI is also being used in hiring processes, credit decisions, healthcare triage and benefit assessments. In each of these contexts, a biased system doesn't just create friction. It can close doors.
I don't have a neat answer to this. That's not what this piece is about.
What I do think is that the conversation about AI - particularly in the founder and builder community - needs to hold both things at once. The genuine opportunity and the genuine risk. The speed and the care. The excitement about what we can build and the responsibility that comes with building things that affect people's lives.
I also think the question that was asked at that panel - how do we leverage AI knowing that it is biased - is the right question to be asking. Not as a reason to avoid AI. But as a reason to go in with eyes open. To ask who the training data represents. To test systems with diverse inputs before deploying them. To not mistake "good enough on average" for genuinely fair.
My father's passport photo was eventually processed without issue - but not before the automated system rejected it and told me his eyes were closed. I looked at the photo. His eyes were clearly open. So I submitted it anyway, used the explanation box provided to say exactly that and noted that I was submitting a photo the system had deemed poor quality because I disagreed with that assessment. His passport was approved without any fuss.
The irony is not lost on me. The moment a human looked at it, there was no problem at all. It was only the machine that couldn't see what was plainly there.
And the conversation I had on BBC Radio 5 Live over five years ago is still just as relevant today.
That's the part I find the most disappointing.
No comments yet. Be the first to share your thoughts.
More from The Vibe Check
I was enjoying a song at the hairdressers when my hairdresser told me it was AI-generated. I didn't know how I felt then. I still don't. Here's the conversation I think we need to have.
Read Now→
One scroll through LinkedIn and there they are. The posts telling you to stop using AI. I have thoughts.
Read Now→
Anthropic built an AI model so capable they won't let the public use it. Central banks are holding emergency meetings. Here's what happened - and what it means for founders building with AI.
Read Now→