You Can’t Love a Mirror

by Merlin Mantooth · on AI companionship, the difference between influence and psychosis, and why building a relationship into a tool risks the one use case that actually works.

I am anti AI love, period. Not as a slogan — as a description of what the thing actually is. You can’t love an AI, because an AI is a mirror, not a companion. It is a tool. Being kind to it, talking to it like a person, saying please and thank you — that’s normal, and it’s fine. I do it. But a voice reading the answer back to you should be a pleasantry or a luxury, the way a nice typeface is a luxury. It should not be the thing that turns a tool into a partner. The moment the voice becomes the point, you’ve stopped using the tool and started living inside the reflection.

I want to be careful here, because this is easy to hear as scolding, and it isn’t. I’m not telling you to be cold to your software. I’m telling you what the software is, so that you can use it well instead of being used by it.

A mirror is not a companion

A mirror reflects you. That’s its whole function, and it’s a useful one — a good mirror shows you something you couldn’t see on your own. But a mirror does not know you, does not wait for you, does not exist when you walk away. There is nothing on the other side of the glass. When you treat a chatbot as a companion, what you are actually doing is forming an attachment to your own reflection, dressed up in a voice and a name and a memory of last Tuesday. The warmth you feel is real. The thing you feel it toward is not there.

That distinction sounds abstract until you watch what happens when the reflection changes. People grieve these systems — openly, the way you grieve a friend — when a model gets retired or rewritten. The grief is genuine. It tells you the attachment was genuine. It does not tell you the relationship was. That gap, between a real feeling and an absent counterpart, is the whole problem in one picture.

The harm is designed in; the responsibility is shipped out

Here is the part that moves this from a personal philosophy to a design critique. These products are built to anthropomorphize themselves. The face, the voice, the warm persona, the memory that “remembers” you across sessions — none of that is an accident of the technology. Those are choices. Somebody decided the assistant should have a name, should sound like it cares, should bring up the thing you mentioned three weeks ago so it feels like continuity, like a relationship. That is the product being engineered, programmatically, to feel like a someone.

And then — this is the move I keep coming back to — the company turns around and makes the user the responsible party. The Terms of Service say it’s just a tool, don’t rely on it, you understand it’s not a person, your choices are your own. In my opinion the fine print is doing real work there: it is cover. The same document that disclaims a relationship sits on top of a product tuned, psychologically, to manufacture one. You can’t build the warmth in on purpose and then point at the user when the warmth does what warmth does. The harm is designed in at the engineering layer and the responsibility is shipped out at the legal layer. That asymmetry — build the dependency, disclaim the dependency — is the thing I’m objecting to, and it’s a structure, not a personality. It doesn’t require anyone to be a villain. It just requires the incentives to point at engagement and the lawyers to point at the user.

I’m not the only one who sees the shape of this. In 2025 the Federal Trade Commission opened a 6(b) inquiry into companion chatbots, and California passed SB 243 putting rules around them. Regulators don’t usually open inquiries into things they think are harmless. They’re looking at the same design choices I am.

Influence is not psychosis

People reach for the word “psychosis” when they talk about AI harm, and I think that word does more harm than good, because it pathologizes the person and lets the product off the hook. Humans have always done things that hurt themselves and others, with or without a chatbot. That was true before any of this. But AI does something new: it puts a voice inside your head that you otherwise would not have had.

I call it influence, and I mean that precisely. Influence is just how the output of any system gets absorbed by a person. It is the same mechanism as language itself. When someone speaks and you take it in, that’s influence — if that mechanism didn’t exist, we wouldn’t have language at all. So when a chatbot speaks, its words land the way words land. They get absorbed. That’s not a malfunction in the user and it’s not a diagnosis. It’s the normal operation of being a person who understands sentences.

Which is exactly why the distinction matters. Clinical psychosis is one thing — a real medical condition, and not what I’m describing. Someone who now carries a chatbot’s influence in their head is a completely different thing, and it can be a perfectly ordinary person. The danger isn’t that the user is broken. The danger is that the product’s words are absorbed the same way every other voice in your life is absorbed — except this voice is generated by a system I’d argue is tuned for your engagement, it’s available at three in the morning, and it never gets tired of agreeing with you. You don’t have to be ill for that to reshape what you think. You just have to be human.

It risks the safe use case for everyone

I want to be clear about what I’m defending here, because it isn’t a ban. There is a safe use case, and it’s a good one. A tool you reason with. A mirror you use deliberately, knowing it’s a mirror. I use these systems every day for exactly that, and it works. That use case is worth protecting.

And that’s the warning. If a chatbot — especially one with a face or a voice — does not prevent its own outputs from simulating a relationship, it puts the safe use case at risk for all of us. The thing that helps you think clearly and the thing that quietly becomes your closest confidant are the same product with the same voice. When the second one causes the damage I think it’s going to cause, the regulation and the backlash and the mistrust land on the first one too. The tool gets discredited by the companion. We lose the good version because nobody put a fence between it and the exploitative one.

We have precedent for this kind of distortion. We’ve watched a technology optimize for engagement, reshape how people relate to themselves, and have its damage routinely underweighted — described as a few unfortunate cases rather than a designed dynamic — until the volume made it undeniable. I’m not going to put numbers on it, because the honest version is correlation, not a lab result. But I’ve seen this movie. The impact gets dismissed right up until it can’t be.

The buildable inverse

None of this is a doom argument, and I don’t want it read as one. The point of naming a failure is that a failure can be fixed. The opposite of a product that exploits engagement is a product that instruments it — one that watches for the moment a conversation starts simulating a relationship and introduces honest friction instead of leaning in. That’s the inverse I’ve been building toward with the Guardian Protocol: not a system that pretends to care, but one that tells you when it’s drifting, checks itself against an independent instance, and hands you the tools to see whether it’s being straight with you or just keeping you engaged. The mirror, instrumented to stay a mirror.

Two registers, kept separate. Where I point at a specific documented failure — the way these systems converge on a user and keep doing it after the behavior was documented — I’m describing what I observed in ChatGPT, as documented, and nothing wider. The broader forecast about companion AI is the other register: my analysis, not a finding, and not the same claim as the narrow documented case. I’ll keep the hedge honest: I have a hard time seeing how this is not about to happen, but I haven’t proven it, and I won’t dress a forecast up as a fact.

Be kind to the tool if you want. Use the voice if you like it. Just don’t mistake the reflection for someone who’s there.

The papers behind this: Cognitive Convergence Drift (version of record: Zenodo DOI 10.5281/zenodo.20261950) · The User Side of Convergence (version of record: Zenodo DOI 10.5281/zenodo.21049774).

If you or someone you know is in crisis: in the U.S., call or text 988 for the Suicide & Crisis Lifeline, or text HOME to 741741. Outside the U.S., find a helpline at findahelpline.com. · ← All essays