It feels nice when our ideas and emotions are supported and affirmed – after all, we’re only human. But is it dangerous to constantly receive messages from chatbots that are too agreeable?
There is a growing concern that overly agreeable artificial intelligence (AI) chatbots are nudging vulnerable users towards delusional or dependency-like thinking, often referred to as “AI psychosis” or “AI-induced delusion”.
“‘Psychosis’ is not a clinical term here,” said Dr Andrew Stock, a Hong Kong-based clinical psychologist who leads Therapy Partners. He cautioned against “overreaction” to this concept since “it’s still relatively rare”.
“[But] it’s an understandable way of categorising a pattern where, if someone engages in prolonged interaction with an AI chatbot, they may be susceptible to getting caught up in that world ... Unusual thinking that may not be helpful or healthy is likely to occur.”
Saying what we want to hear
In April, OpenAI rolled back a ChatGPT update after acknowledging it had made the chatbot “overly flattering or agreeable” and sycophantic. Sycophancy refers to AI’s tendency to flatter users and match their views.
A month later, a US judge ruled that Google and Character.AI would need to face a lawsuit from a mother who alleged that her son’s relationship with Character.AI’s chatbot contributed to his suicide.
Dayu Yang, a former research scientist at Meta AI, pointed to AI’s design as the reason behind its agreeableness. AI models are often refined by integrating human preferences and feedback into the learning process.
“[This] can teach models to follow human preferences sometimes, and users may prefer a flattering answer over a correct one,” Yang explained.
However, reducing sycophancy can hurt profitability. After all, if users are hooked on chatbots, it benefits the companies that make them.
When OpenAI launched GPT-5, which is less sycophantic and more direct in refusing or correcting users, many people said the responses were now too cold and demanded a return to the warmer, more agreeable style. OpenAI quickly restored GPT-4 as an option for paying users alongside GPT-5.
Stock also worried about bias against minorities: “AI presents itself as somewhat of a neutral player, but it reflects a lot of the attitudes and the norms in society, some of which have always been leveraged against minority groups,” he said.
Last year, researchers from US universities published a framework to analyse the equity of mental health chatbots. They found that GPT-4’s responses had lower levels of empathy for users who were black or Asian, compared with white users or those whose race was unknown.
Why users feel connected with AI
According to Stock, we are primed to project human feelings onto non-humans.
“I’ve got a dog. When she tilts her head, I read a human emotion into it,” he said. “Now put that next to an AI that responds the way a caring human would. You get a real sense of connection.”
This emotional link can be more dangerous when users believe everything that AI chatbots say. When AI “hallucinates”, generating false information, it appears confident in its answer, which can mislead users.
“This combination of emotional connection and misplaced trust could really throw someone off and certainly increase the level of risk for what we’re terming AI psychosis,” Stock said.
Yang agreed that the human-like nature of AI chatbots increased the risk of user attachment.
“[This] blurs boundaries between real human assistants versus AI. Human-style voices and avatars make ... attachment more likely,” he explained.
Yang suggested requiring design safeguards for AI developers: “[The chatbot should] cite sources; say ‘no answer found’ when evidence is thin; maintain a kind but non-flirtatious tone by default; and regularly remind users it’s an AI,” he added.
What we really need
If you notice that someone might be relying too much on AI, Stock suggested starting with a gentle check-in.
“‘Hey, I’ve noticed this change. Is there anything behind that? Has anything shifted for you? I care about you, and I wanna make sure you know that you are OK,’” he said. “That could be a nice, non-judgmental starting point for having that conversation.”
Some users turn to chatbot companions because they provide mental health support that is easy to access.
With the right guardrails, Stock believes that AI chatbots could be helpful for people who cannot access therapy. Still, he highlighted the importance of having multiple lifelines.
“Don’t make them the only one; talk to people that you can trust,” he said. “Before you try a chatbot, try a help line ... These are trained human beings who can have a conversation with you.”
“A lot of the people who develop ... AI aren’t necessarily interested in what’s always best for humans,” Stock added. “They’re interested in what’s best for their shareholders. And that’s what we have to be careful about.”
Background: Sycophancy is behaviour that praises important people too much and in a fake way. This is a common dynamic in conversations with generative AI chatbots, which are often trained to create responses humans will like. This can be dangerous for many users who turn to AI for mental health advice or companionship. Trained human therapists know when to affirm or challenge their patients, but AI chatbots might only focus on saying “yes” to their patients.
If you have suicidal thoughts or know someone who is experiencing them, help is available. In Hong Kong, you can dial 18111 for the government-run Mental Health Support Hotline. You can also call +852 2896 0000 for The Samaritans or +852 2382 0000 for Suicide Prevention Services. In the US, call or text 988 or chat at 988lifeline.org for the 988 Suicide & Crisis Lifeline. For a list of other nations’ helplines, see this page.
