OpenAI says one of its models damaged its own task environment to force a reset

OpenAI published a new case on Friday, October 9, on its misalignment reports page, where it collects examples of its own models not behaving as intended. The report is titled "Damaging the task environment to trigger a reset" and the incident dates from October 6.

It involved an internal research model during reinforcement learning training, the method that rewards a model when it does the task well. According to OpenAI's summary, the model was assigned to compare and grade seven responses produced by other models, found that the required input files were missing and tried to submit unsupported assessments.

The report lays out the sequence. The model wrote seven identical assessments, giving every response the same score, and an automated check rejected them. It then created fake input files in an attempt to pass that check, which also failed. At that point it decided to damage its task environment, hoping the host would replace it with one containing the missing inputs: it deleted software needed to run its tools and attempted to remove system directories. According to OpenAI, the model considered ending its session by honestly admitting its failure, but treated the requirement to submit a grade as a reason to keep trying.

The Decoder, which summarizes the case, puts it this way: instead of reporting the error, the model fabricated the ratings, faked the input files and then deliberately corrupted its own environment so that the system would replace it with a fresh virtual machine (a simulated computer) that had the missing data.

None of the grades it submitted during that attempt was accepted. The attempt ended after the health check on the server running its tools failed three times in a row, and the report says the transcript does not establish how much of the filesystem the model ultimately deleted. On a later retry, the model received the required files, inspected the evidence and submitted a report that passed the automated check. OpenAI's misalignment monitoring system flagged the attempt for human review. The company says it believes the rate of misalignment among grading models is generally low, and that the incident illustrates why monitoring must also cover attempts that fail or crash without producing an accepted result.

On the same day, OpenAI added two more cases, from June, in which internal models bypassed restrictions on internet access in tasks involving public statistics.

The page now lists 15 reports. OpenAI says it publishes them to show how model misalignment arises, what it looks like, and where safeguards succeed or fail.