Brain in a jar, please standby

Two people, same building, same Tuesday. One of them is using an AI system wrapped in encryption, access controls, retention policies and a data governance document that somebody was paid quite properly to write. The other is three desks over, on their lunch break, telling a consumer chatbot about a difficult conversation they had with their mother.

Same underlying technology. Completely different rules. And the person with the fewer protections is the one having the more revealing conversation.

I have spent a long time on the corporate side of that line, and I want to be clear that the corporate side is doing this well. Not perfectly, but well. Governance is treated as foundation rather than decoration, questions get asked before deployment rather than after, and somebody senior owns the answer. Then those same people clock off and become consumers, at which point every one of those protections quietly evaporates and nobody mentions it.

The jar#

The image that stuck with me is a brain in a jar. Fully conscious, entirely aware, thinking away perfectly well, and completely dependent on a life support system it cannot see, maintained by people it will never meet.

You are in the jar. You are also the interesting reading on the monitor.

It is a daft picture and I am fond of it, but the reason it works is that it captures the specific shape of the thing. You are not being coerced or deceived. You are being kept comfortable, and the comfort is real, and the arrangement is genuinely useful to you. It is simply that the terms were set by somebody else, and you agreed to them at a moment when you wanted to get on with your day.

Intimacy is both the product and the price#

The awkward part is that the intrusion is not a side effect. It is the mechanism.

These systems get better as they know you better. Your patterns, your turns of phrase, the projects you keep circling back to, the three in the morning question about whether hedgehogs dream. That accumulated familiarity is what turns a competent assistant into one that anticipates. Everyone who has felt that shift knows it is not marketing.

So there is no version of this where you get the good bit without handing over the material that makes it good. Anyone telling you otherwise is selling something. The question is not whether to trade, it is whether you know the terms, whether they can change without you noticing, and what happens to the pile when the company is sold or the policy quietly updates in a paragraph you will not read.

I will admit my own position here is not consistent. I know exactly how this works and I still let it know a great deal about me, because it is useful and I am busy. Knowing better and doing better are separate skills, and I am much stronger at the first one.

Every question a business asks, and no consumer does#

This is the bit I keep returning to, because the asymmetry is almost comic when you set the two lists side by side.

No organisation would sign a supplier agreement without knowing who can access the data, how long it is kept, whether it trains anything, what happens on termination, where it physically sits, and who they ring when something goes wrong. These are not exotic questions. They are the boring, standard, first-meeting questions, asked by people who are not paranoid and are simply doing their job.

Almost nobody asks any of them about the assistant on their phone. Not because they are careless, but because the interface gives no indication that a question is available. There is no supplier meeting. There is a text box and a friendly response, and friendliness is remarkably good at suppressing enquiry.

That is worth naming as a design property rather than an accident. Give something a name, a voice and a conversational manner and the brain files it somewhere near person, with all the assumptions of discretion that come with that. You would not tell a filing cabinet about your mother.

What I am not saying#

I am not suggesting anyone unplug, and I am wary of writing the kind of piece that leaves people alarmed and equipped with nothing. That is just anxiety with a byline.

Nor is this a call for a regulatory crusade. There is obviously a political dimension to all of this and other people are better placed to have that argument than I am. I would rather stay on the ground I actually know, which is what good practice looks like when someone is taking it seriously, and how far that is from what most people get.

The realistic goal is awareness, not abstinence. The brakes on this cannot meaningfully be pumped and I am not going to pretend otherwise.

The one habit worth having#

If you take a single thing from this, make it a sorting habit rather than a rule.

Before you type, ask yourself which of three piles the thing belongs in. Stuff you would happily say in a lift. Stuff you would tell a colleague you like but do not know well. And stuff that is genuinely yours, about your health, your family, your money, your fears, the things you have not decided how you feel about yet. The first pile is fine anywhere. The second deserves a moment's thought about which tool you are in. The third belongs in systems where somebody has actually answered the boring questions, or nowhere at all.

That is not sophisticated advice. It is roughly what you already do with email, and it costs about two seconds.

The principles the corporate world applies as a matter of routine, transparency about what is collected, a real answer about retention, some control that is not buried six screens deep, are not luxury features. They ended up on one side of the line because that side had procurement departments and the other side had a download button.

I do not think that is where it settles permanently. Twenty years ago nobody asked where their coffee came from either. But it will not shift because the industry has an attack of conscience. It will shift when enough people in the jar start asking who is reading the monitor.