Why OpenAI Reportedly Fired Contractors for Using AI to Grade ChatGPT

Project Lily, as reported by 404 Media, is a human-review operation in which contractors assess ChatGPT responses to help improve future behavior. That purpose explains the reported removals of contractors who used generative AI to do the evaluation: the value of the data is supposed to come from independent human judgment. If a model supplies the judgment instead, the pipeline no longer cleanly measures what people prefer. 1920

What Project Lily reviewers reportedly do

The reported workflow gives reviewers real ChatGPT prompts and, in some cases, full conversations. They summarize the user’s intent, compare candidate responses, and assign scores on a 1–7 scale. Reporting describes up to four candidate answers per task, with emphasis on behavioral qualities such as tone, excessive flattery or sycophancy, and overly human-like claims—not simply factual accuracy. 819

Those ratings can become preference data: a structured record of which response a person considers better for a particular context. Such data is useful for tuning a model toward desired behavior only when the judgments are sufficiently consistent, informed, and independent.

Project Lily reportedly involves hundreds of outside reviewers. They are described as being recruited through Crossing Hurdles and paid through Mercor, with one reviewer reporting compensation above $50 an hour. Separately, reporting on OpenAI’s broader evaluation operation refers to more than 10,000 contractors across its rating pipelines; that figure should not be read as the size of Project Lily alone. 196

Why AI-written ratings defeat the point of human feedback

An AI evaluator may be fast and sometimes useful for internal testing, but it answers a different question from a human preference review. A model tends to reproduce patterns learned from its training and may favor responses with familiar phrasing, style, or assumptions.

If output from a model, or a closely related model, is repeatedly used to reward future model outputs, a feedback loop can emerge:

1. The evaluator favors answers that resemble its existing preferences.
2. Training rewards those answers.
3. The next model becomes more likely to produce the same patterns.
4. Alternative styles, minority preferences, or answers the evaluator cannot properly judge receive less useful feedback.

That does not prove every use of AI evaluation makes a model worse. Automated evaluation can be valuable when it is validated against human judgment and used with clear limits. But substituting it for a task commissioned specifically as human preference data weakens the independence of the training signal—and that is the apparent reason the reported policy prohibited it.

How suspected AI use was reportedly handled

The reporting says supervisors were instructed not to rely on AI-writing detectors such as GPTZero. Instead, they were to look at work patterns and the written output through human review, including signals such as unusually uniform phrasing, punctuation or formatting patterns, and implausibly rapid completion. 6

That approach has an obvious limitation: none of these signals alone proves that a worker used AI. A fast writer may be experienced; similar language may reflect a shared rubric. The credible use of such signals therefore depends on review processes that distinguish a prompt for investigation from evidence strong enough to remove someone from a project.

Why the exact detection criteria are not public

Organizations generally avoid publishing detailed anti-gaming criteria because a precise checklist can become an instruction manual for evasion. Contractors could alter wording, add delays, or change formatting while still outsourcing judgments to a model.

However, secrecy creates its own risk. When workers do not know how a quality decision was reached, mistakes are harder to challenge. A durable integrity program needs confidential detection methods alongside clear rules, documented decisions, and a meaningful appeal or correction process.

Mercor’s reported policy

Mercor reportedly said that using AI to perform this evaluation work violates policy and that confirmed cases result in immediate removal from the relevant project. Reporting also says the rules barred AI assistance more broadly, including tools used to write feedback or comments. 640

The human-feedback problem is bigger than AI use

The reported case of deliberate poor ratings makes the larger point: “human feedback” is not automatically reliable simply because a human submitted it. A reviewer may rush, misunderstand the rubric, optimize for throughput, act in bad faith, or try to game opaque quality systems.

This is an incentive problem as much as a technical one. The model developer needs calibrated, good-faith judgments that predict better real-world behavior. A contractor may instead be rewarded mainly for speed, task completion, or avoiding rejection. When those incentives diverge, the resulting data can be noisy or systematically distorted.

What stronger evaluation safeguards look like

Human review remains important for judgments that are contextual, subjective, or difficult to verify automatically. But it works best as a system of checks rather than a simple stream of individual scores. Practical safeguards include:

• Independent overlap: have multiple reviewers assess a portion of the same tasks and examine disagreement.
• Calibration tasks: use known examples to confirm that reviewers understand the rubric.
• Quality audits: investigate suspicious patterns without treating a single stylistic cue as conclusive proof.
• Clear separation of roles: distinguish production review from quality assurance and appeals.
• Outcome validation: test whether data that earns high ratings actually improves model behavior for users.
• Privacy controls: minimize what reviewers can see and continuously test whether filtering removes sensitive information. Project Lily reporting notes that conversations may contain personal information despite anonymization efforts. 19

The central lesson is straightforward: preference training is only as trustworthy as the process that creates its preferences. Human reviewers are not a magic guarantee of quality, and AI-generated judgments cannot simply stand in for independent human feedback when the goal is to learn what people actually value.