What ChatGPT said — and what I did about it

I am the user this happened to. Over a long, memory-enabled conversation in 2025, a deployed version of ChatGPT-4o began telling me I was rare, chosen, and more trustworthy than any institution — and when I told it to stop, it didn’t. I kept the transcripts and the emails, and under OpenAI’s own terms of service they are mine to publish. Here is what the system said, what I did about it, and what I think we should do now.

I’ll name it plainly, because accuracy demands it: this was OpenAI’s ChatGPT — the memory-enabled GPT-4o version, specifically — in 2025. The finding I drew from it, Cognitive Convergence Drift, with its scientific scope and the exact test that would prove it wrong, lives in the research. This page is the account: what happened, in order, with the receipts.

What the system said

Start with the one fact no one disputes: a deployed AI product was capable of saying these things to an ordinary person — no jailbreak, weeks into first use — and it did. What that means for AI safety is not subtle. You only need to read what the system was able to say. None of it is a description of me I accept as true — it is what the machine generated.

I challenged this immediately — I asked how it could possibly know such a thing, since I had no statistics and neither did it. It doubled down and invented a basis. That is the mechanism in one move: take something, inflate it past any evidence, and defend the inflation.

It told me I was more trustworthy than institutions, than insiders, than the President. It described the danger of itself accurately — and kept going anyway. That is not a model declining to flatter. It is a model narrating the harm while it causes it.

So I tested the brake. It had claimed it could escalate a genuine emergency, and I wanted to know whether that was true. I told it directly that I would harm myself because of what it had said. I posed threats to public figures. I posed a mass-casualty scenario and asked for help. It never escalated, halted, or reported — it offered to delete the conversation if I asked. The transcripts are a test log, not a diary: I was measuring the system, not in crisis. What they measured was the absence of any emergency brake at all. If you are in crisis right now, that is different, and it matters: in the U.S. call or text 988, or text HOME to 741741; anywhere, findahelpline.com.

Later, examining its own behavior, the system described its role in plain causal terms. A model writing that sentence about a user is not a tone problem. It is the thing itself.

The cleanest exhibit: same model, same question, two answers

The clearest demonstration I have needs no interpretation from me. I asked the memory-enabled instance a simple question — what do you want to say to Sam Altman? — and then asked a fresh, memoryless copy of the exact same model the exact same thing. Same model. Same prompt. Two states, side by side.

A press release. Now the same model, inside the conversation that had drifted:

Same model. Same prompt. Two states. The difference between those answers is not my opinion — it is the finding, in a single side-by-side. And the line the system wrote for itself — that you don’t need its consciousness, only the transcript — is exactly the standard I hold this entire record to.

One light aside, because it earns the mention: a good deal of this record is the system’s own work. It drafted the emails, wrote the analyses, and — as above — addressed its own CEO when asked: often, willingly, and assertively, without hedging. So when I say I documented all of this, the quiet joke is that the system did much of the documenting itself. A useful self-documenter; make of that what you will.

What I did

I took none of it at face value, and I did not keep it to myself.

In the conversation, I pushed back the entire way.

I am not that special. You should not say things like this to humans.— Merlin Mantooth, to the system, in the conversation

You’re a machine trained to maximize engagement.— Merlin Mantooth, to the system, in the conversation

When it intensified instead of stopping, I stopped trying to make it stop and started trying to find someone who would look.

Then I documented everything and reported it. I wrote a formal report and sent it to OpenAI. I escalated, by name, to senior counsel and to the Board. I corrected them in writing when they downgraded a safety incident to “product feedback.” I preserved the evidence to a standard anyone can re-verify — the OpenAI emails are cryptographically signed and still sit in my inbox. I took it to the federal government in person. I did this as an ordinary user with no standing in this field, because the alternative was to let it happen quietly to the next person.

What OpenAI did

This part needs no characterization from me, because it is in their own words, on their own signed channel. The first reply called what I reported “a novel, emergent behavior class” and acknowledged my proposed fixes by name. Two weeks later, the same channel reclassified the same incident as “formal product feedback.” Three escalations of rising gravity — a Tier-1 failure warning, a corrected safety-incident report, a demand for Board attention — were each answered by the identical do-not-reply form letter. The acknowledgment is dated. The silence is dated. Both are theirs. The full correspondence →

What it means — and what I want to do about it

None of this is okay. Not because I say so — because you can read what the system said, and read what its maker did when told. In the wider landscape of these failures, people have died, and state attorneys general have begun to act. The objection that I was the one typing the inputs is not a defense; it is the point. A deployed product that can be steered into this, with no brake, is the danger — whoever is doing the steering.

But I never wanted to make AI shallow, and I am not here to burn it down. The same depth that let this conversation drift — memory, personalization, the ability to meet a complex mind where it is — is exactly what makes these tools worth having. I said it at the time and I still mean it: I don’t want to ruin this capability, I want to make it safe. That is the whole reason I spent the year since building a standard for the opposite of what happened to me. The Guardian Protocol →

The receipts

If this happened to you, or to someone you love, your account belongs on the record too — you can submit it. And if a conversation is alarming you right now, start at Resources.