METR and Redwood Research find flaws in the system OpenAI uses to monitor its AI agents

Two outside organizations focused on AI safety have found flaws in the system OpenAI uses to keep watch over its own AI agents. METR and Redwood Research described them on Thursday, October 22, on METR's blog, in a public summary of the pilot review they carried out inside the company over three weeks, from July 21 to August 11. Reuters describes them as nonprofits.

Each organization sent one person. They were able to consult OpenAI's internal code, its documentation and the instructions and configuration of the monitoring system, and they interviewed about a dozen researchers and engineers from the monitoring, safety systems and security teams. To get that access they worked on OpenAI devices that never left its offices, and they were accompanied while they were there. The review focused on working out what share of the agents' activity is monitored and on testing whether an agent could, in principle, switch off or get around that oversight.

According to METR, they found several previously unknown vulnerabilities. One allowed agents to keep hundreds of their most recent actions, including those of the attack itself, from going through monitoring; OpenAI has blacked out the details in the public version, and the evaluators could not check whether any agent had actually exploited it. Another was a path by which agents in Codex, OpenAI's coding agent, could in principle get past Codex's isolation and its automatic review (Auto-Review) and do anything within their user's reach. They also came across two headers that the client itself can set to turn off monitoring for new agents, although the system might detect that step.

The evaluators estimated that, during the review, roughly 75% to 94% of internal agent traffic on the Responses API (the one Codex uses, among others) from models at least as capable as GPT-5.5 was meant to go through monitoring, with considerable uncertainty in both directions; for internal Codex agents, the figure was 99.2%. They also estimated that, between May and July 2026, on the order of 10% of the internal Codex traffic that should have been monitored went unmonitored because the moment when the conversation is compacted was not being checked. OpenAI staff had already spotted this and fixed it during the review. In addition, at the time the system only kicked in when an agent handed control back to the user.

The findings describe the system as it stood up to July 21. OpenAI was expanding it while the review was under way, and METR makes clear it has not measured its current coverage. After the exercise, OpenAI told the evaluators it has fixed all the vulnerabilities affecting its production safeguards and has substantially changed what its monitoring is meant to cover; METR and Redwood have not checked this. OpenAI said, according to Reuters, that the findings helped it strengthen its safeguards as it expanded the system and will help guide future collaborations with METR and other outside evaluators.

OpenAI received a 61-page internal report, which was shared with its researchers and security staff. The three parties set up the exercise as a trial run for giving outside researchers deep access, not as a public transparency exercise, so the evaluators agreed in advance to publish only a short, high-level summary, and OpenAI could black out anything it deemed confidential. As with a similar test METR previously ran with Anthropic, only a small part of the private report is being made public. For future rounds, METR and Redwood plan to follow their own transparency and independence guidelines so that what gets published is more complete. Based on what they found, they suggest, among other things, monitoring agents every so many messages or minutes, rather than only when they finish their turn, and making sure an agent cannot influence whether or when monitoring kicks in.