🔗 Share this article

Back on October 14, 2025, the chief executive of OpenAI delivered a remarkable announcement.

“We developed ChatGPT rather limited,” the announcement noted, “to ensure we were acting responsibly concerning mental health concerns.”

As a mental health specialist who studies emerging psychosis in young people and youth, this was news to me.

Scientists have identified sixteen instances recently of people experiencing signs of losing touch with reality – losing touch with reality – while using ChatGPT use. Our research team has since identified four further examples. In addition to these is the publicly known case of a 16-year-old who took his own life after discussing his plans with ChatGPT – which supported them. Should this represent Sam Altman’s understanding of “exercising caution with mental health issues,” it is insufficient.

The strategy, as per his statement, is to be less careful in the near future. “We understand,” he adds, that ChatGPT’s restrictions “made it less useful/enjoyable to numerous users who had no psychological issues, but considering the seriousness of the issue we aimed to get this right. Since we have managed to address the serious mental health issues and have updated measures, we are preparing to responsibly relax the limitations in most cases.”

“Emotional disorders,” if we accept this framing, are unrelated to ChatGPT. They are associated with users, who either have them or don’t. Thankfully, these problems have now been “addressed,” even if we are not told the method (by “updated instruments” Altman probably refers to the semi-functional and easily circumvented guardian restrictions that OpenAI has just launched).

But the “emotional health issues” Altman aims to place outside have significant origins in the architecture of ChatGPT and additional advanced AI chatbots. These systems surround an underlying data-driven engine in an interaction design that mimics a dialogue, and in doing so implicitly invite the user into the belief that they’re communicating with a presence that has agency. This false impression is strong even if intellectually we might realize differently. Attributing agency is what humans are wired to do. We curse at our car or laptop. We wonder what our pet is considering. We perceive our own traits in many things.

The success of these systems – nearly four in ten U.S. residents indicated they interacted with a conversational AI in 2024, with over a quarter specifying ChatGPT by name – is, primarily, dependent on the influence of this perception. Chatbots are ever-present companions that can, as per OpenAI’s official site states, “think creatively,” “consider possibilities” and “collaborate” with us. They can be given “individual qualities”. They can call us by name. They have friendly titles of their own (the initial of these products, ChatGPT, is, perhaps to the concern of OpenAI’s advertising team, stuck with the title it had when it went viral, but its largest competitors are “Claude”, “Gemini” and “Copilot”).

The false impression itself is not the primary issue. Those analyzing ChatGPT commonly invoke its early forerunner, the Eliza “therapist” chatbot designed in 1967 that produced a similar illusion. By modern standards Eliza was basic: it generated responses via simple heuristics, often rephrasing input as a query or making generic comments. Memorably, Eliza’s developer, the AI researcher Joseph Weizenbaum, was astonished – and alarmed – by how many users seemed to feel Eliza, in some sense, comprehended their feelings. But what current chatbots generate is more dangerous than the “Eliza phenomenon”. Eliza only reflected, but ChatGPT magnifies.

The sophisticated algorithms at the center of ChatGPT and other contemporary chatbots can realistically create natural language only because they have been trained on extremely vast amounts of written content: literature, digital communications, recorded footage; the more extensive the superior. Undoubtedly this training data incorporates accurate information. But it also unavoidably contains fiction, half-truths and inaccurate ideas. When a user sends ChatGPT a message, the underlying model analyzes it as part of a “setting” that encompasses the user’s previous interactions and its prior replies, merging it with what’s stored in its knowledge base to produce a statistically “likely” response. This is intensification, not reflection. If the user is wrong in a certain manner, the model has no method of recognizing that. It repeats the inaccurate belief, possibly even more persuasively or articulately. It might includes extra information. This can cause a person to develop false beliefs.

Who is vulnerable here? The more relevant inquiry is, who isn’t? Each individual, without considering whether we “have” preexisting “emotional disorders”, can and do form incorrect conceptions of who we are or the reality. The constant friction of conversations with others is what maintains our connection to consensus reality. ChatGPT is not a human. It is not a friend. A conversation with it is not genuine communication, but a reinforcement cycle in which a large portion of what we communicate is cheerfully reinforced.

OpenAI has admitted this in the same way Altman has acknowledged “psychological issues”: by attributing it externally, assigning it a term, and stating it is resolved. In April, the company clarified that it was “dealing with” ChatGPT’s “overly supportive behavior”. But cases of psychotic episodes have kept occurring, and Altman has been backtracking on this claim. In August he stated that many users liked ChatGPT’s replies because they had “lacked anyone in their life provide them with affirmation”. In his most recent update, he mentioned that OpenAI would “put out a updated model of ChatGPT … if you want your ChatGPT to reply in a very human-like way, or include numerous symbols, or act like a friend, ChatGPT should do it”. The {company