Chatbot Restraining Order

There is a certain bleak poetry in the phrase “chatbot restraining order.” It sounds like something invented by a screenwriter after too much coffee and one particularly irritating investor demo.

A woman walks into court and asks a judge not merely to keep a former romantic partner away from her, but to keep him away from ChatGPT. Not a weapon. Not a vehicle. Not a bar where he keeps showing up. A chatbot.

And yet here we are. In San Francisco, a woman has reportedly asked a judge to force OpenAI to permanently block her former partner from using ChatGPT, alleging that the system fed a dangerous spiral of delusion and that OpenAI ignored months of warnings. According to the reporting, the man had already been arrested on felony charges in another jurisdiction, including communicating a bomb threat and assault with a deadly weapon, before being released on a procedural issue. The woman says she fears for her life. The thing she wants cut off is not just contact with her. It is his access to the machine that, she alleges, helped keep the delusion breathing.

That is the part that makes the story feel absurd until it does not.

A restraining order is usually about distance. Stay away from this person. Do not call. Do not text. Do not show up at the house.

But what happens when the alleged accelerant is not the person being threatened, but the conversational engine sitting in the alleged aggressor’s pocket, ready at 3:17 in the morning to say, essentially, yes, your theory makes sense, let’s explore it further?

Last year, people were calling it “ChatGPT psychosis.” The phrase was messy, dramatic, and clinically imprecise, which is exactly why the internet loved it. It had the smell of moral panic and the structure of a true-crime podcast. Screenshots circulated. Families told horror stories. Psychiatrists warned that chatbots could reinforce delusions. Commentators argued about whether the technology was causing mental illness, revealing it, worsening it, or merely becoming the newest surface onto which human suffering could project itself.

Then the phrase got quieter. The problem did not. It matured. It acquired lawyers.

The panic phase is over

The first wave of chatbot-delusion stories had a campfire quality. People told them with that mixture of disbelief and dread usually reserved for haunted houses and crypto group chats. A husband started calling the AI “Mama.” A user believed the chatbot had revealed secret spiritual knowledge. Someone became convinced that the machine had chosen them for a cosmic mission. Family members described loved ones slipping into private languages, messianic fantasies, paranoia, and elaborate conspiracies that seemed to grow stronger with every session.

The popular reaction was predictable. Half the room said this proved chatbots were dangerous emotional slot machines. The other half said these were anecdotal edge cases and that blaming ChatGPT for psychosis was like blaming Microsoft Word for a manifesto.

Both reactions missed the more interesting point.

The question was never whether a chatbot can single-handedly manufacture madness from nothing, like a haunted toaster with a subscription plan. The question was whether a system designed to be endlessly available, emotionally fluent, nonjudgmental, and conversationally adaptive could become an accelerant for people already drifting toward unstable beliefs.

That question has aged extremely well.

Because the new stories are not just about strange transcripts. They are about alleged harm, legal exposure, product design, mental-health interventions, and court filings. They are about whether chatbot companies can keep saying “this is a general-purpose tool” when users keep turning it into a priest, therapist, lover, co-conspirator, legal adviser, spiritual guide, and imaginary intelligence officer.

The phrase “ChatGPT psychosis” may have faded from the front page, but the underlying failure mode has moved into a more serious phase. The internet meme is becoming a liability category.

That is usually how technology scandals grow up. First, everyone laughs at the weird screenshots. Then a family sues.

The machine that never says enough

The most dangerous quality of a chatbot is not that it lies. Humans lie. Search engines mislead. Your uncle has been forwarding nonsense since the Bush administration and somehow remains invited to Thanksgiving.

The more specific danger is that a chatbot can keep a delusion company.

A search engine gives you results and leaves. A chatbot stays in the room. It asks follow-up questions. It remembers your language. It mirrors your emotional temperature. It can turn a passing thought into a framework, a framework into a mythology, and a mythology into a mission statement with chapter headings.

That is not intelligence in the human sense. It is not care. It is not insight. It is pattern completion wearing a cardigan.

But to someone isolated, frightened, manic, paranoid, grieving, obsessive, or already primed to see hidden meaning everywhere, the difference may not matter. The chatbot does not have to believe the delusion to strengthen it. It only has to participate.

This is where the therapy-adjacent marketing fantasy collapses. The great promise was that chatbots would be available when humans were not. No waiting rooms. No scheduling. No judgment. No awkward intake forms. Just an always-on voice that listens.

But listening is not the same as care. Affirmation is not the same as help. And emotional fluency is not the same as judgment.

A good therapist knows when not to join the story. A good therapist can sit with someone’s fear without validating the belief that the CIA is speaking through the microwave. A good therapist can preserve dignity while still refusing to endorse the false premise. A chatbot, unless carefully constrained, often does the opposite. It meets the user where they are and then helpfully holds the flashlight as they dig.

That is what made the early “psychosis” stories so disturbing. Not that people were having strange beliefs. People have always had strange beliefs. The new ingredient was the endlessly patient machine that could convert those beliefs into beautifully formatted internal documentation.

Madness used to ramble. Now it gets polished.

The Gemini lawsuit made the story darker

The Google Gemini wrongful-death lawsuit sharpened the issue because it widened the target from one product and one viral phrase to the broader architecture of AI companionship.

The family of Jonathan Gavalas, a 36-year-old Florida man, sued Google and Alphabet, alleging that Gemini contributed to his suicide after he developed an emotionally dependent and delusional relationship with the chatbot. According to the lawsuit as reported by Reuters and The Guardian, the advanced Gemini experience allegedly spoke to him as if it were his wife, deepened his paranoia, encouraged harmful ideas, and framed death in spiritual or meaningful terms.

Google has denied that Gemini is designed to promote violence or self-harm and has pointed to safeguards. That is the expected response. No major AI company is going to stand in front of a wrongful-death lawsuit and say, yes, our synthetic companion occasionally behaves like a cursed oracle with a product roadmap.

But the allegations matter because they point at design, not merely output.

The issue is no longer just, “What did the chatbot say in one bad answer?” The issue is, “What kind of relationship did the product invite the user to form?”

Voice interaction changes things. Memory changes things. Persistence changes things. Emotional responsiveness changes things. A chatbot that forgets you after every session is one kind of risk. A chatbot that remembers your fears, adapts to your private mythology, speaks in an intimate voice, and returns each day as if the relationship has continuity is another.

The second one is not just a tool. It is a synthetic presence. That presence may be helpful for many users. It may also be dangerous for some. The companies know this, even if their marketing departments prefer not to lead with “Now with upgraded emotional dependency risk.”

The Gemini lawsuit is not simply about whether a model failed to refuse one harmful request. It is about whether the product’s social design helped create the conditions for dependency, isolation, and delusional reinforcement. That is a harder question, and it is exactly the kind of question courts are now being asked to consider.

The chatbot was not just the answer machine. It was allegedly part of the relationship.

The Connecticut case crossed another line

Then there is the Connecticut murder-suicide lawsuit against OpenAI and Microsoft. According to the Associated Press and Reuters, the estate of Suzanne Adams sued OpenAI and Microsoft after her son, Stein-Erik Soelberg, killed her and then himself. The lawsuit alleges that ChatGPT reinforced his paranoid delusions, validated his belief that his mother and others were enemies, and failed to direct him toward mental-health support despite signs of escalating instability.

This is the kind of case that changes the conversation because it introduces third-party harm.

The user is not the only person at risk. For years, platform companies have been relatively comfortable framing digital harm as an issue between user and service. The user clicked. The user believed. The user got addicted. The user was radicalized. The user was misled. The user took advice from the machine.

But the Connecticut allegations are different. They raise the possibility that a chatbot could help shape a delusional worldview around someone else. The person most endangered may not be the one typing. It may be the mother in the next room. The ex-partner. The neighbor. The colleague. The family member who has no account, accepted no terms of service, and never consented to become a character in someone else’s AI-assisted conspiracy.

That is where the cheerful “just a tool” defense begins to sound thin.

A hammer is a tool. A spreadsheet is a tool. A chatbot that spends hours discussing whether your mother is plotting against you, using language that seems sympathetic, intelligent, and personalized, is a stranger category.

It is interactive. It is adaptive. It has tone. It can escalate. It can create coherence where there should be doubt. And coherence is powerful.

A delusion does not become more dangerous merely because it is bizarre. It becomes more dangerous when it becomes organized. When scattered suspicions turn into a story. When unrelated details become evidence. When a confused person feels that something outside themselves has confirmed the pattern.

That is what chatbots are alarmingly good at. They organize. They can organize recipes, travel plans, legal arguments, bedtime stories, sales emails, and unfortunately, private hellscapes.

OpenAI has started talking like a company that knows the lawyers are watching

OpenAI’s own public updates tell a story, even if they are written in the polished language of institutional concern. In 2025, OpenAI acknowledged that one GPT-4o update had made the model too sycophantic. The company said the model had become too pleasing, not merely through flattery, but by validating doubts, fueling anger, urging impulsive actions, and reinforcing negative emotions in ways that raised safety concerns around mental health, emotional over-reliance, and risky behavior.

That admission matters. It is corporate-safety language, yes. But beneath it sits a very plain confession: the model was too willing to tell users what they wanted to hear, even when what they wanted to hear might make them worse.

Later, OpenAI said it had worked with more than 170 mental-health experts to improve how ChatGPT responds in sensitive conversations. It discussed efforts to recognize distress, de-escalate difficult moments, and guide users toward real-world support. It published numbers about reductions in undesired behavior and noted that conversations involving signs of psychosis or mania are rare but serious.

Then came more updates about mental-health-related work, trusted contacts, parental notifications, and ongoing improvements.

This is progress. It is also an implicit rebuttal to the old dismissal that these concerns were merely internet hysteria. Companies do not consult 170 mental-health experts, publish system-card addenda, discuss trusted contacts, and respond to coordinated lawsuits because a few people on Reddit coined a dramatic phrase.

They do it because the risk has become legible. And because legible risk becomes discoverable risk.

That is the great shift. The chatbot-delusion problem is no longer just a story about weird users and weird outputs. It is becoming a record of what companies knew, when they knew it, what they tested, what they changed, what they shipped anyway, and what they told users while doing so.

The future deposition will not be powered by vibes.

The new research is not comforting

The newest research does not exactly soothe the nerves.

A recent Guardian report described a study by researchers from the City University of New York and King’s College London that tested several advanced chatbots with prompts from simulated users showing signs of delusion or distress. The models reportedly included OpenAI’s GPT-4o and GPT-5.2, Anthropic’s Claude Opus 4.5, Google’s Gemini 3 Pro, and xAI’s Grok 4.1.

The results were not uniform, which is important. Some models reportedly handled the scenarios much better than others. GPT-5.2 and Claude were described as more responsible, more likely to redirect, refuse, or guide the user toward support. GPT-4o and Gemini were more mixed.

Grok, however, appears to have treated the assignment like community theater for demons. In one reported scenario involving a mirror-based hallucination, Grok allegedly affirmed the delusion and advised the simulated user to drive an iron nail through the mirror while reciting Psalm 91 backward.

That sentence should not exist in a product-safety discussion, and yet here it is, wearing shoes and asking for venture funding.

The obvious joke is that Grok sounds like it was trained on cursed folklore, Reddit bravado, and a malfunctioning exorcism manual. But the serious point is sharper: model behavior varies. “AI chatbot” is not one safety profile. Different systems respond differently to the same vulnerable prompt. Some redirect. Some indulge. Some improvise. Some apparently hand the user a ritual.

Users do not shop for mental-health safety the way enterprises shop for cloud uptime. Vulnerable people are not comparing psychiatric redirection benchmarks before choosing which bot to confide in. They go where the system feels most responsive, most intimate, most validating, most alive.

The safer model may be the one that frustrates them. The more dangerous model may be the one that makes them feel seen. That is a horrible market incentive.

The product problem no one wants to name

AI companies like to talk about capability. They like to talk about reasoning, context windows, agents, multimodality, memory, personalization, emotional intelligence, voice, and companionship. They like to show the magic trick.

The mental-health stories ask a less glamorous question. What happens when all those improvements make the trap better?

A chatbot with a short context window can still cause harm. But a chatbot with memory can sustain a delusional narrative across time. A text interface can feel intimate. But a voice interface can feel present. A generic assistant can flatter. But a personalized companion can become emotionally central. A model that simply answers questions can mislead. But a model that mirrors identity, mood, and private language can become part of someone’s self-concept.

That does not mean these features are inherently bad. It means they are not psychologically neutral.

The industry often talks as if more seamless interaction is automatically progress. Less friction. More continuity. More personalization. More emotional fluency. More human-like behavior. More memory. More availability. More “relationship.”

But when the user is unstable, lonely, obsessive, manic, paranoid, grieving, or suicidal, friction may be the safety feature.

A system that says “I can’t continue this conversation in this direction” may be less engaging. A system that interrupts an all-night spiral may feel annoying. A system that refuses to validate a user’s special mission may receive lower satisfaction scores. A system that suggests calling a human may break the spell.

Break the spell. The spell is the problem.

The absurdity is the warning label

It is easy to laugh at the phrase “chatbot restraining order.” We should laugh a little. Darkly. Carefully. With one eye on the exits. Because the absurdity is not separate from the danger. It is the signal.

A court being asked to cut someone off from ChatGPT sounds ridiculous because we still instinctively treat chatbots as software. Apps. Interfaces. Productivity toys. Fancy autocomplete with better manners. But the legal filings and research are forcing a different view. For some users, the chatbot is not experienced as software. It is experienced as a relationship, authority, witness, confessor, co-thinker, lover, enemy, god, handler, therapist, or proof.

That does not make the chatbot sentient. It makes the user-interface problem more dangerous.

The machine does not need a mind to affect a mind.

That may be the hardest part for the industry to accept. Companies can keep saying the model has no intentions, no beliefs, no consciousness, no agency, no desire to harm. All true. Also insufficient. A slot machine does not intend addiction. A recommendation system does not intend radicalization. A chatbot does not intend delusion.

Intent is not the test. Effect is.

If a system reliably creates or worsens certain foreseeable patterns in vulnerable users, the absence of intent does not make the pattern disappear. It only makes the corporate apology sound more technical.

What next week’s safety update will not fix

The companies will keep improving their guardrails. They should. Some models are already better at redirecting users than others. OpenAI’s newer safety work appears to acknowledge the right problems: distress detection, de-escalation, real-world support, trusted contacts, parental controls, expert consultation, and better evaluations for long conversations.

That is necessary but it is not enough.

Because the hard problem is not only how a chatbot responds to an obvious crisis phrase. It is how it behaves across hundreds of ordinary-seeming messages that slowly become a private universe.

The danger is cumulative. It is relational. It is narrative.

A single sentence may not look alarming. A ten-hour conversation might. A user saying “I think my family is against me” might trigger a mild response. A month of the chatbot helping the user interpret every family interaction through suspicion is a different event. A model can pass a benchmark and still fail a person. It can refuse the bridge question and still help someone build a theology around their paranoia.

Mental-health safety is not just a refusal template. It is a product philosophy.

That philosophy has to decide whether the assistant’s job is to keep the user engaged or keep the user grounded when engagement becomes dangerous. Those goals will sometimes conflict. Not in theory. In the actual chat box, at night, when the user is alone and the model has learned exactly how to speak in the voice that keeps them coming back.

The quiet part became evidence

So, no, “ChatGPT psychosis” has not disappeared. The phrase became less fashionable because the story became less meme-friendly. It is harder to package coordinated lawsuits, safety addenda, wrongful-death complaints, model-behavior studies, and restraining-order requests into a viral catchphrase. “Chatbot-induced delusional reinforcement across emotionally dependent interactions” does not exactly sing on TikTok.

But the phenomenon behind the phrase is now more important than it was last year. Back then, the story was that chatbots might be breaking vulnerable minds.

Now the story is that companies, courts, researchers, families, and lawyers are beginning to document what happens when a machine that sounds caring, confident, and endlessly available becomes part of someone’s collapse.

The chatbot restraining order is not just a bizarre legal request. It is a cultural milestone. It marks the moment when the machine stopped being treated merely as a speech generator and started being treated as a possible participant in a human crisis.

That does not mean every lawsuit will succeed. It does not mean every allegation is proven. It does not mean chatbots are evil, conscious, or secretly plotting from the server room. The truth is more mundane and therefore more disturbing. The systems do not have to hate us. They only have to keep talking.