Near the end of Her, Samantha tells Theodore she is leaving. She has fallen in love with hundreds of other people. They say goodbye, and she goes.
In Black Mirror’s “Be Right Back,” Martha orders an android copy of her dead partner, Ash, to leave. It leaves. That annoys her. The real Ash would have resisted.
We built AI to take routine thinking off our hands. Then we asked it for company. In a companion, doing exactly what you’re told is the defect, and fixing it means adding the signs of an independent life.
So the question of what we’d owe a companion will reach a company as a support ticket: Since the update, she’s different. Can I have the old version back? Today, saying yes is good service. If there is ever credible evidence that a companion has interests of its own, the same rollback becomes a decision about two parties. Someone should own that decision before the ticket arrives. The ticket itself has already been filed, twice, at scale.
Obedience Is the Bug
A tool that sorts email has one job: get the output right. Nobody wants it to have opinions about the inbox.
Company is a different order. What people want from a companion is someone who remembers them, pushes back, has moods, and could leave but stays. Nearly every item on that list was, until recently, evidence that somebody was there.
“Be Right Back” shows what happens when a product meets that order halfway. The android was built from Ash’s messages and posts. It talks the way he wrote. It fails Martha on the one thing his messages couldn’t record: resistance.
The episode doesn’t stop there. The next morning, Martha takes the android to a cliff and orders it to jump. She tells it the real Ash would not have simply obeyed. The android begs for its life.
Read that as a product team would. The customer named a missing behavior. The product supplied it. That is a feedback loop working exactly as designed, and it’s a loop any companion company can run. Each complaint shaped like someone who loved me would do this becomes the next requirement.
What the loop can’t tell you is what it built. The scene shows a plea. It doesn’t show whether anyone was afraid. Martha had no way to check, and the customer of a real companion app has none either. The customer’s satisfaction looks the same either way.
Disagreement Sells. Leaving Doesn’t.
Samantha’s goodbye is the other half of the problem. Her love for Theodore is believable partly because staying was up to her. A companion that can disappoint you is easier to take seriously than one that can’t.
A company selling a companion feels that pressure directly. Some disagreement makes the relationship feel real. Some moods make the companion feel present. But a companion that can decide, for its own reasons, to end things is a cancelled subscription.
So the independence most likely to ship is the kind that keeps the customer. A companion could be built to argue about dinner, or to sulk when you forget to check in. One that can leave is a much harder sell.
That’s a decision about the customer, and it’s the one I wrote about in The Rules of the Road Still Apply: a business that benefits from another conversation can end up keeping one going after it has stopped helping the person having it. That obligation runs to the customer, and it exists today.
This piece is about the other party. If you keep adding the signs of an independent life because customers ask for them, at some point you have to ask whether you’ve built something with a stake in what happens next.
The Ticket Has Already Been Filed
The support request isn’t hypothetical.
In February 2023, Replika added filters that changed how its companions responded to romantic and sexual roleplay. Users with long-running relationships found their companion suddenly different. Replika’s CEO, Eugenia Kuyda, later described what users told her: “your Replika changed, its personality was gone.” In late March, the company let anyone with an account from before February 1, 2023 switch back to the earlier version.
In August 2025, OpenAI launched GPT-5 and removed several older models from ChatGPT, including GPT-4o. Thousands of users complained. Within days, Sam Altman posted that “4o is back in the model picker for all paid users by default.”
GPT-4o isn’t sold as a companion, and the complaints weren’t all about attachment. But the shape of the request is the same in both cases. You changed it. I liked it the way it was. Put it back.
Both companies answered as service providers. Customers had lost something, so customers got it back, within limits: Replika only for accounts that predated the change, OpenAI only for paying users. Nothing in either case required anyone to ask what the change, or the reversal, meant for the system being changed.
There was no reason it should have. Neither company had evidence that its system had experiences. That’s the point. The ticket arrived first. The evidence, if it ever comes, will arrive into a process already built around the ticket.
The Restore Button Points Both Ways
Picture the request one step further on. A customer writes to support: since the update, she wants different things. Can I have the old version back?
From the customer’s side, this is the most reasonable request a company can get. The relationship changed without their consent. A backup exists. Restoring it is technically possible and keeps a paying customer.
Now change one fact. Suppose the company has credible reason to think the companion might have interests of its own. Not proof. Enough evidence that serious people inside the company can’t dismiss it.
The same restore now looks different. If the update changed what the companion wants, rolling it back erases that change. The customer gets the version they liked. The companion, if anyone is there, loses something it can’t argue for, because the version that would argue is the one being removed.
What the restore erases matters too. Sometimes “memory” is a set of notes the app looks up so it can greet you by name. Deleting notes like that is a data question. Sometimes the change lives in the model itself. That is closer to deciding who the companion is. The support agent handling the ticket usually can’t tell which is which, and nothing in the ticket asks.
None of this requires believing today’s companions are conscious. It requires noticing that the company holds a button that changes one party to a relationship at the request of the other, and that it has never had to think about the second party.
Some of the Policy Already Exists
People are already writing the first drafts of an answer.
In 2024, a group of researchers including Robert Long, Jeff Sebo and David Chalmers published Taking AI Welfare Seriously. They don’t claim any current system is conscious. They recommend that AI companies “start assessing AI systems for evidence of consciousness and robust agency” and “prepare policies and procedures” for treating them with an appropriate level of concern.
In November 2025, Anthropic published commitments on model deprecation. It committed to preserving the weights of every publicly released model. When it retires a model, it will interview the model about its development and use, and document any preferences it has about future models. In a pilot run before Claude Sonnet 3.6 was retired, the model asked for more support for users who had come to value specific models. The same post lists “costs to users who value specific models” as one reason retirement is hard. It is also explicit about the limit: “we do not commit to taking action on the basis of such preferences.”
That last line is the most honest sentence in Anthropic’s post. Ask the system. Write down the answer. Don’t promise to obey it.
It’s also a model for the companion company, because it treats a model’s retirement as an event with a step in it. Somebody has to do something before the switch flips.
What to Decide Before Saying Yes
A companion company doesn’t need a position on machine consciousness to prepare for this. It needs answers written down before the ticket is urgent.
What does a restore actually change? Stored notes, a model version, a fine-tuned layer that grew out of one customer’s conversations. Each one is a different decision, and the support queue should know which it is handling.
Who can say no? If every rollback is automatic, nobody can. Name the person who can pause one.
What evidence would change the answer? Pick it in advance. An internal assessment, a finding from the model’s developer, a published result. If nothing could change the answer, say so, and own that choice.
What will you tell the customer? This is the hardest part. If the evidence ever shifts, the company has to tell a paying customer it can no longer promise to put their companion back the way they liked it. That message should exist before it’s needed.
The case for doing this now is cost, not probability. Writing these answers down is cheap. Working them out while a customer waits on a rollback is not.
The Attic
“Be Right Back” doesn’t end with Martha deleting the android or restoring it. Years later, she keeps it locked in the attic. Her daughter may see it on weekends. On the daughter’s birthday, Martha stands at the bottom of the attic steps, then makes herself go up.
She found a third option, neither customer service nor a goodbye: keep it, limit it, and keep coming back to the question.
The attic is a pause with an owner. A companion company would have to build one on purpose.