AI mental health chatbots, inequality and the risks of self-guided care
David Gilbert
Unequal benefits, unequal harms is a discussion paper on AI mental health chatbots, inequality and the risks of self-guided care.
The paper, by patient empowerment expert David Gilbert, who has sought help and advice from AI for his own mental health problems, assesses the research on how the huge growth in the use of AI mental health tools can put people at risk and increase inequality.
It finds that the rapid expansion of AI mental health chatbots and self-guided AI systems has outpaced the development of robust mechanisms to protect people from harm. Oversight is uncoordinated and there are significant gaps in evidence, accountability and patient safety. Without proper regulation, information content, quality and safety are at risk.
AI mental health tools may improve access and affordability for some people. But benefits will not be distributed evenly, and research shows that AI could exacerbate inequalities more widely.
Large language models absorb patterns that contain forms of structural discrimination. When used in contexts such as mental health support, these biases can be damaging. For example, by reinforcing stereotypes, misinterpreting user experiences, or generating responses that marginalise or invalidate certain identities.
There is inadequate data about who uses what sort of AI tools withindifferent demographic and socioeconomic groups. Significant gaps remain in research into how this is impacting people experiencing disadvantage, disabled and neurodivergent people, racialised communities, and people with severe mental illness and experience of trauma.
The paper calls for more research into potential harm being generated amongst the most vulnerable and disadvantaged of users.
David Gilbert is Director of InHealth Associates, experts in patient and public engagement, co-production and patient-leadership. Sign up to the ImPatient newsletter here.
