A new study from the London School of Economics has uncovered a troubling pattern: AI tools used by over half of England’s councils are downplaying women’s health issues in adult social care assessments. The findings centre on Google’s “Gemma” model, which consistently used less serious language when describing women’s physical and mental health needs compared with men’s - even when the underlying case notes were identical.
For a public sector already stretched thin, AI promises efficiency. But in the care sector, language isn’t just description — it’s decision-making currency. Understating need risks reducing the support women receive, effectively building inequality into the system.
📊 Supporting Stats
617 real adult social care case notes were tested, each run through multiple AI models with only the gender swapped.
This produced 29,616 pairs of summaries, revealing significant gender-based differences in language.
Terms like “disabled”, “unable” and “complex” appeared far more often in summaries about men than women with identical needs.
One US study of 133 AI systems found 44% showed gender bias, and 25% exhibited both gender and racial bias (Source: Nature Machine Intelligence).
Meta’s Llama 3 model showed no gender-based language variation, suggesting bias isn’t inevitable but is model-specific.
🧠 Decision: Did It Work?
From a brand (or public sector) trust perspective, no - this is a reputational and operational risk. In healthcare and social care, accuracy and equity are part of the value proposition. AI that systemically minimises women’s needs undermines fairness, erodes public confidence, and exposes organisations to legal and ethical challenges.
The insight here isn’t simply “AI has bias” - it’s that bias is model-dependent. One tool introduced significant disparities, another did not. That means procurement, testing, and oversight choices will make or break outcomes. Councils relying on AI without rigorous bias auditing are gambling with both care quality and public trust.
📌 Key Takeouts
What happened: LSE research found Google’s Gemma model downplayed women’s care needs compared with men’s when summarising identical case notes.
What worked well: The methodology - controlled gender-swapping in real case data - revealed specific, measurable bias.
What didn’t: Councils using AI without transparency on model choice, frequency, or performance risk embedding inequality.
Signals for the future: Bias is not a universal feature of AI but varies between models - highlighting the importance of model selection and testing.
Brand relevance: For any organisation using AI in high-stakes contexts, fairness isn’t optional - it’s a core part of maintaining legitimacy.
🔮 What We Can Expect Next
Regulators will face increasing pressure to mandate bias testing and transparency for AI used in public services. Expect “algorithmic fairness” to become a procurement requirement, not just a PR line. There’s also likely to be heightened scrutiny from advocacy groups - particularly in healthcare and welfare - as these tools touch vulnerable populations.
If brands in other sectors are watching, the lesson is clear: you don’t get to outsource accountability to the algorithm. Bias audits, model transparency, and continuous monitoring are the new hygiene factors for trust. Fail here, and you risk headlines that stick.