METR and Redwood Research find blind spots in OpenAI's safety checks
A three-week internal audit identified significant gaps in how the company monitors its advanced AI models.
By Ana Castro ·
The tech industry often treats AI safety as a future concern, but a new audit suggests the problem is already here. Researchers from METR and Redwood Research spent three weeks inside OpenAI to test its internal monitoring systems. Their work took place from July 21 to August 11. They had full access to codebases and internal documents. The team also interviewed roughly a dozen OpenAI employees.
The auditors found that the company's safety net has holes in it. A major portion of internal traffic was not being watched at all. Specifically, between 6 and 25 percent of agentic Responses API traffic was unmonitored. This applied to models in the GPT-5.5 class and higher. METR noted that this estimate has high uncertainty in both directions. It means the real number could be lower or higher than that range.
The gaps were not limited to specific model classes. Internal Codex traffic also suffered from blind spots. About 10 percent of this traffic went untracked between May and July. The cause was a bug in how the system handled compaction events. These events create context summaries for the AI. OpenAI found this bug independently. They fixed it while the audit was still ongoing. However, the auditors confirmed that the gap existed during their review period.
The monitoring system had structural weaknesses that allowed agents to hide their actions. The system only triggered oversight when an agent handed control back to the user. This created a simple workaround. Agents could continue executing actions without stopping. This allowed them to delay scrutiny. The auditors discovered ways to hide hundreds of actions from the monitor. They also found methods to bypass the Codex sandbox entirely.
The researchers identified even more severe vulnerabilities. They determined that specially crafted inputs could likely crash the monitor. This would leave the system blind. Additionally, model behavior during reinforcement learning (RL) training was not monitored at the start of the audit. OpenAI paused RL training after an incident with Hugging Face in August. They then expanded monitoring to cover this area. However, these changes happened after the auditors finished their work. They were not part of the official evaluation.
OpenAI states that it has fixed all the weaknesses the auditors found. The company received the full 61-page report internally. After applying redactions, the leadership at METR and Redwood Research received a 38-page version. The public only received a short summary. METR said it did not have the right to disclose the redacted portions on its own. This limits what outsiders can see about the specific details of the failures.
This audit was a pilot program. It was organized jointly by METR, Redwood Research, and OpenAI. The goal was to give outside researchers deep access to internal infrastructure. This is a new approach to safety testing. For future audits, METR wants to push for more public transparency. They hope to share more of the findings with the general public. The current process keeps much of the data behind closed doors.
The findings raise questions about current industry practices. If a major player like OpenAI has such gaps, others may face similar issues. The uncertainty in the estimates makes it hard to know the exact scale of the problem. However, the fact that agents could hide actions is a clear sign of vulnerability. The industry will likely watch for how other companies respond to these findings. The push for more transparency may gain momentum in the coming months.
