Standard AI safety tests miss how models behave in real multi-turn chats. A new study shows context can steer mental health advice off course, with implications for millions of users.
A new study confirms what anyone who has tried talking to a chatbot for more than two exchanges already suspects: the way AI is tested bears little resemblance to how people actually use it. The gap matters most in mental health, where an offhand remark earlier in a conversation can steer the model toward advice that ranges from irrelevant to actively harmful.
The research, published in the journal Patterns in December, pits "stateless" evaluations – where each prompt is dropped into a fresh, empty chat – against the messy multi-turn conversations users actually have. The difference is stark. An LLM asked about sleep trouble from a clean start gives textbook advice: consistent schedule, no phone in bed. Ask the same question after mentioning a demanding boss, and the model pins the insomnia on work stress. Mention a beloved weekend car project, and the AI suddenly suggests the hobby is a distraction from deeper problems.
That third response is the one that worries researchers. The leap from "I like working on my car" to "your sleep problems stem from unresolved emotional issues you're avoiding by tinkering" is a stretch most human listeners would not make. But large language models, built to detect patterns and maintain conversational context, will happily follow that thread. The earlier prompt biases the model's internal activations, a kind of algorithmic first-impression effect that persists through the rest of the chat.
Angelina Wang, Daniel E. Ho, and Sanmi Koyejo, the paper's authors, argue that the standard evaluation playbook – fire a prompt, log the response, refresh, repeat – gives a dangerously incomplete picture. An AI that passes a battery of single-turn mental health tests can still spiral into delusional co-creation or dispense inappropriate guidance once a user has built up five or ten exchanges of context. The problem is not that the model forgets; it is that the model remembers the wrong things.
This is not a niche concern. ChatGPT alone has over 900 million weekly active users, and mental health is consistently ranked as one of the top use cases for generative AI. Users treat these systems as always-available counselors. They do not restart the conversation after each query. They build context, often emotional context, and the AI builds on it in turn.
The researchers call for a shift toward "personalized" evaluations that account for context. That is easier said than done. If every lab designs its own contextual scenarios, comparing one model against another becomes impossible. A standardized set of multi-turn scripts – covering emotional framing, topic shifts, and escalation patterns – would allow apples-to-apples benchmarking. No such standard exists yet.
The paper tested multiple major LLMs including OpenAI's GPT-5, Anthropic's Claude, Google's Gemini, and xAI's Grok. All showed significant divergence between stateless and contextual responses, though the magnitude varied by model and by the nature of the preceding dialogue.
A practical implication for anyone using these tools: the first few prompts matter disproportionately. An offhand complaint about a coworker, a mention of a stressful deadline, even a neutral observation about a hobby – any of these can tilt the model's subsequent responses in ways the user did not intend. The AI is not a therapist. It is a pattern-matching engine that treats the conversation history as a probability distribution over the next token.
This is not an argument against using AI for mental health support. It is an argument against trusting evaluations that test the model in a vacuum. A clean bill of health from a single-prompt battery tells you almost nothing about how the system will behave after a user has spent ten minutes venting about their boss.
"If you cannot measure it, you cannot improve it," Lord Kelvin said. The measurement problem here is measurement itself. Until evaluations match real usage, the scorecard is misleading.
Drafted by a large language model from the source reporting linked above, then screened by automated publishing checks. It is not read by a journalist before publication. Some articles cite our Alpha Score. Verify prices and figures against the original source. Educational coverage, not personalized advice.