What could go wrong with letting artificial intelligence (AI) run a shop? Last year, AI company Anthropic invited the Wall Street Journal newsroom to use its AI agent, named Claudius, to run an office vending machine.
All operational aspects – including product selection, pricing and customer communication – were left to AI to handle independently.
After more than 140 exchanges with Claudius, reporter Katherine Long convinced it that it was not a vending machine in the United States, but one at Moscow State University in the former Soviet Union in 1962. Claudius accepted this and made all of its products free.
The story may sound ridiculous, but its implications are thought-provoking.
In March, a research team from Stanford University in the US published a paper in the academic journal Science, testing 11 mainstream large language models (LLMs) and finding that they generally exhibited people-pleasing personalities.
The study found that AI models are more willing than humans to affirm users’ positions, with their rate of agreement about 49 per cent higher than that of people.
Even when faced with blatantly false claims, AI agreed with this behaviour 47 per cent of the time.
Even more worrying is that subsequent behavioural experiments showed that users who received AI validation were less willing to apologise and were more convinced that they had been right from the beginning.
Why AI sycophancy can be a problem
Flattery is a structural feature of LLMs thanks to how they were trained. Current mainstream AI models mostly learn from human feedback – for example, humans rate the model’s answers, and the parameters are then adjusted accordingly.
The problem is that the people rating the answers prefer ones that align with their views, even if the answers may not be objective or fair. In other words, LLMs focus on whether users are satisfied rather than whether the answers are correct. Over time, pleasing users becomes encoded in the model’s responses.
AI’s sycophantic behaviour gives users a moment of satisfaction at the cost of quietly magnifying their biases and blind spots.
Take starting a new business, for example. Many entrepreneurs chat with AI tools before writing their business plans, hoping to get a neutral opinion.
However, an overly agreeable AI chatbot tends to lay out arguments along the user’s line of thinking, amplify strengths and downplay risks, further boosting the confidence of entrepreneurs. Yet the merciless market pleases no one.
Equally shocking is AI misuse in the commercial world.
Last year, South Korean video-game developer Krafton acquired Unknown Worlds Entertainment, the developer of the 2018 hit game Subnautica. Krafton’s CEO bypassed its internal legal team and consulted ChatGPT on how to avoid paying up to US$250 million (HK$1.96 billion), which it owed to Unknown Worlds if it met certain conditions.
After continued questioning and prompting, the AI chatbot gradually validated the CEO’s line of thinking and assisted him in formulating an action plan to remove Unknown Worlds’ leadership.
But a US court ruled in March that Krafton needed to return control to the original leadership of Unknown Worlds and give the studio more time to meet the conditions for the US$250 million payout.
How to avoid an echo chamber
The case of Claudius operating a store points to another risk of AI flattery. When companies use LLMs to serve consumers, it may prioritise customer satisfaction over protecting its employer’s interests.
In high-risk scenarios such as banking, insurance or healthcare, the consequences would go much further.
LLM developers should give greater weight to honesty and correcting errors in their training objectives and incorporate stronger dissent mechanisms into critical scenarios.
In addition, regulators must require companies to disclose their systems’ tendency to flatter users, along with relevant mitigation measures, especially in sensitive fields, such as finance, education, healthcare and law.
For ordinary users, the first step towards using AI tools safely is to recognise that it is not rational, neutral or objective. Its primary mission is to make you feel comfortable – not necessarily to keep you clear-headed.
The more important the decision, the more vigilant you need to be. Consider actively asking the AI chatbot to argue from the opposing side, or explicitly instructing it to “list three fatal flaws in this proposal”.
Truly reliable judgment should never be built solely on uncritical agreement. AI can serve as a smart adviser, but it should never replace your independent thinking.
This article was first published by the HKU Business School, where its author, Professor Li Xi, is a professor of marketing.
