A contractor reading your most private ChatGPT conversations might know roughly where you live and what you’ve asked the chatbot before. That’s not a hypothetical. According to The Next Web, it’s been happening under an internal program OpenAI calls Project Lily, and the company didn’t volunteer a single word about it to users until a reporter from 404 Media started asking questions.
The setup works like this: OpenAI pays hundreds of contractors to read real ChatGPT conversations and score the quality of the chatbot’s replies. The reviewers don’t see account usernames. But they do see a summary of a user’s memories, which can include details about past conversations, habits, and approximate location. OpenAI says a Privacy Filter model scrubs personal information before a human ever sees anything. That sounds reassuring. But OpenAI’s own documentation for that model admits it makes mistakes, misses uncommon identifiers, and tends to under-redact when it lacks enough context to make a judgment call.
So the privacy layer is imperfect by the company’s own admission. And still, users weren’t told humans might be reading their chats at all. When 404 Media asked OpenAI to point to where it discloses this, the company didn’t respond. After the story published, it pointed to a help page. That’s not disclosure. That’s damage control.
The legal dimension here is significant. Last September, the Court of Justice of the European Union ruled in EDPS v SRB that a data controller’s obligation to inform users applies at the moment of collection, not at some later point, and is judged from the controller’s perspective, not the recipient’s. That means the question isn’t whether a contractor in North America could figure out who wrote a specific prompt. The question is whether OpenAI told users what it was doing with their data before it collected it. The answer, based on what’s been reported, is no.
Italy’s data protection authority already fined OpenAI 15 million euros earlier this year, with 9 million of that tied specifically to processing data without a sufficient legal basis. The regulator also ordered the company to run a public awareness campaign on Italian television and radio for six months. That’s a significant penalty, and it suggests European regulators are not treating this as a minor technical compliance issue.
The default settings also deserve scrutiny. The “improve the model for everyone” option is switched on automatically for free, Plus, and Pro accounts. Enterprise, Business, and Edu accounts have it off by default. And if a regular user does turn it off, that only applies to new conversations, not anything already collected. So the data is already gone.
The contrast with OpenAI’s enterprise pitch is hard to ignore. The company spent much of this year marketing zero data retention as a premium feature for corporate clients. Consumers, meanwhile, get a default setting that feeds their conversations into the training pipeline, monitored by human contractors they were never told about.
One reviewer quoted in the original reporting put it plainly: “I don’t think they would imagine some contractor somewhere is analyzing the conversations.” That same company has also been encouraging those users to connect their bank accounts. The trust gap between what OpenAI asks of users and what it tells them keeps getting wider.