Skip to content
Rush Commerce
Software & Dev3 min read

OpenAI Performance Factory: Codex agents debug, engineers merge

OpenAI's Performance Factory ran 1,700+ Codex investigations into ChatGPT slowdowns. 70% of audited root causes held up; about 15 agent PRs shipped as written.

The headline number isn't the 1,700 investigations. It's the 15 pull requests. On November 19, OpenAI published how its Performance Factory works. It's an internal system where Codex agents chase down ChatGPT performance regressions on their own. The diagnosis side works well. The fix side, by OpenAI's own account, is still mostly human.

What actually happened

From OpenAI's engineering post and VentureBeat's exclusive:

  • The setup. Four parts: a task service with workflows defined in config files and Markdown prompts, a dispatch loop that starts Codex turns, Codex running in the cloud with access to metrics, logs, traces and profiles, and subscriptions that wake an agent on CI failures, review comments or deploys.
  • The workflow. An alert opens a task. An agent finds the root cause. A second Codex session tries to disprove that diagnosis. Then an agent opens a PR, an engineer reviews and deploys, and an agent checks whether the metric recovered.
  • The numbers. More than 1,700 investigations and 600+ root causes in just over three months, about 17 a day. In a manual review of 100 root-cause suggestions, 70% were graded accurate.
  • The fix gap. VentureBeat, citing OpenAI's performance team, reports about 400 proposed fixes, about 50 shipped fixes that matched or drew on agent suggestions, and about 15 that shipped as the exact Codex-written PR. OpenAI's own post calls the fix "the hard part."
  • Availability. Internal only. No product, no price.

Why it matters for your business

This is the most honest agent scorecard we've seen from a big lab. Agents are good at the tedious part: which metric moved, when, after which deploy. They're weaker at changes that need judgment. OpenAI's own example is adding a cache, which speeds things up and risks stale data.

The piece worth copying is the adversarial reviewer. Early agents stopped at the first plausible explanation and were sometimes wrong. A fresh session told to poke holes caught those. VentureBeat reports it still sends back about a quarter of diagnoses.

You don't need OpenAI's infrastructure to use the pattern:

  1. Let the agent investigate, not deploy. Root-cause reports with the queries attached, so a person can rerun them.
  2. Add a second agent whose only job is to disagree. Cheap, and it catches the confident wrong answer.
  3. Keep a person on the merge button. OpenAI does, and its agents have more context than yours.

Key takeaways

  • OpenAI's Performance Factory ran 1,700+ Codex investigations into ChatGPT slowdowns in about three months
  • 70% of 100 audited root-cause suggestions were graded accurate
  • About 15 agent-written PRs shipped as-is out of roughly 400 proposed, per VentureBeat
  • A separate "disprove it" agent is the part worth copying
  • Engineers still review and merge every fix

Spending hours figuring out why the site got slow? We set up monitoring, alerts and agent-assisted triage that hands you a root cause with evidence, while your team keeps the merge button. See how we build it.

Sources: OpenAI: How OpenAI's Performance Factory keeps ChatGPT fast, VentureBeat: OpenAI reveals Performance Factory.

  • #openai
  • #codex
  • #ai-agents
  • #performance
  • #devops
TR

Tommy Rush — Founder, Rush Commerce

Operator turned builder. 15+ years running operations — now shipping the systems businesses run on. More

Get The Rush Report weekly — one email, zero fluff.