The users assumed the platform was handling it. The platform assumed the users were.

Nobody was hacked. But, there was still an issue that must be discussed, and awareness needs to be raised.

Over the weekend, private conversations with a widely used AI assistant, Claude.ai, turned up in public (Google) search results. There was a technical shortfall involved, and it is being addressed. That isn't the interesting part.

The interesting part is how ordinary the failure was. Someone generated a share link to send to a colleague. The link was public — that is what the feature does. It ended up somewhere a crawler could reach it. Search engines did what search engines do, the indexed the link and the contents.

Two reasonable assumptions collided.

The people using it assumed that if the platform offered the button, the platform had thought through what came after pressing it. The platform treated the button as the user's decision — you chose to make this public, so it is public.

Neither assumption is unreasonable. Both were held sincerely. And in the gap between them sat, by the reporting, medical records, confidential company documents, and work nobody intended to publish.

It's worth being precise about where this stands. Search results were cleaned up quickly, but deindexing is not deletion. A link that no longer appears in search still resolves for anyone who saved it, and third parties had already archived thousands of messages. The exposure is quieter. It isn't undone.

Here's the part that should bother anyone writing an AI policy.

A year ago, a different major assistant had the same category of failure. In that case, making a conversation searchable required the user to deliberately tick a box. The vendor removed the feature anyway — because, in their words, it created too many chances for people to share things they didn't intend to.

So the experiment has already been run. Explicit, opt-in, user-controlled consent was tested by someone, and the company that tested it concluded that people don't reliably understand what they're consenting to.

A year later, here we are again.

Earlier this year I wrote about where a framework for using AI responsibly actually lives. Partly in the systems the vendor builds. Partly in the judgment of the person at the keyboard. And partly in the organizational layer in between — the one most companies skip, because they assume the top and the bottom have it covered.

This is what skipping it looks like.

Which makes the useful question not whether a particular vendor is safe. It's this: who at your company decides which AI output may be made public, and where is that written down?

For most organizations the honest answer is that nobody decides, and it isn't written anywhere. So it gets decided by whoever is on a deadline, in the moment, alone.

That isn't a security failure. It's a governance gap. No vendor patch closes it.