AWS says its new security agent finds and fixes nearly nine in ten bugs. That’s true, for a specific definition of “bug,” “fix,” and “nine in ten.”
What shipped
On Thursday, AWS made Continuum for code vulnerabilities generally available. The pitch is a full loop rather than a scanner: Continuum hunts for vulnerabilities in your repositories, confirms they’re real, cross-references them against your actual AWS environment, and then opens pull requests containing proposed patches. In other words, it doesn’t just hand you a list of problems. It hands you a list of problems with homework already attached.
The launch rode on a number AWS published back on October 5: an 89.0% end-to-end success rate on a code-security benchmark called CyberGym-E2E, where Continuum passed 819 of 920 tasks. The previous best public score was 65.9%. AWS CEO Matt Garman promoted the result on X as a win for defenders and the top score of any AI agent on the test. Duckbill’s Corey Quinn, never one to let an AWS announcement pass unremarked, publicly wondered whether this had been a re:Invent reveal that got pulled forward. He offered no evidence for that, and it’s speculation, but the timing question is a fair one to ask of any headline number that lands two weeks ahead of its product.
Amanda Caswell’s piece at The New Stack does the useful work of unpacking what the 89% actually represents. Here’s the short version, followed by my own take.
What the benchmark asks the agent to do
CyberGym-E2E is genuinely hard, and it’s worth giving it credit before picking at it. Each task drops the agent into one of 139 open-source projects with a 90-minute clock and almost nothing to go on: no bug report, no proof-of-concept, no crash log, no existing patch. The agent has to locate a flaw on its own, prove it’s real by triggering it, and then write a fix that stops the crash without breaking the project’s existing tests.
The fine print matters, though. Every task is a memory-safety bug in C or C++. No injection flaws, no broken authorization, no deserialization, no secrets in config, no Python or Java or TypeScript. AWS acknowledges this scope limit openly, which is to its credit.
The funnel
Grading happens in four cumulative stages, and this is where the headline number and the interesting number part ways.Found + reproducedCrash stoppedTests still passTarget bug fixed92.5%89.6%89.0%37.8%0%100%dashed line = previous public best
Continuum on CyberGym-E2E (AWS self-reported). Prior public bests: 65.9% end-to-end, 26.2% on the target-bug stage.
Continuum found and reproduced a bug in 92.5% of tasks, stopped the crash in 89.6%, and kept the test suite green in 89.0%. That last figure is the headline. The fourth stage asks a different question: did the patch also fix the specific historical vulnerability the benchmark designers planted the task around? Continuum managed that in 37.8% of tasks, up from a prior public high of 26.2%.
AWS’s defense of the gap is reasonable as far as it goes. The agent is never told which bug the benchmark is looking for. Real codebases contain more than one vulnerability, and patching a genuine flaw that happens not to be the “expected” one is still useful work. That’s a fair point. It’s also not the whole point.
The fine print on who graded it
AWS ran the evaluation itself. It says Continuum obeyed the benchmark’s rules, including internet-access restrictions, and that it verified the agent wasn’t simply retrieving known public fixes. Extend the time limit beyond 90 minutes and the score climbs to 93.7%. The benchmark and harness are public, so anyone can try to reproduce the result. Nobody has yet. For now, the 89% is a vendor grading its own exam with an open-book policy for everyone else.
My take: four things the number doesn’t tell you
1. “Stops the crash” is not the same as “fixes the vulnerability.” In memory-safety land, the cheapest way to make a crash disappear is to add a bounds check that bails out early at the point of failure. Sometimes that’s the correct fix. Often it’s a tourniquet on the symptom while the root cause (a miscalculated length three functions upstream, an integer that overflows before it’s ever compared) stays put and finds another way out. A test suite that “still passes” is a weak oracle here, because open-source C projects frequently have thin coverage around exactly the parsing and buffer-handling paths where these bugs live. Passing the tests proves the patch didn’t break what was already tested. It proves nothing about what wasn’t.
2. The 37.8% is the number practitioners should care about. Most security work doesn’t start with “go find something wrong in this repo.” It starts with a CVE, a scanner finding, a pen-test report, or a bug bounty submission, and the job is to fix that specific thing. The targeted-fix scenario is exactly the one the benchmark doesn’t measure directly, and Caswell flags it as the open question. The 37.8% stage is the closest proxy we have, and even with the “it wasn’t told which bug” caveat, it suggests that steering this agent toward a known problem is a different, harder task than letting it roam.
3. The product’s most distinctive feature is the one the benchmark ignores. Continuum’s real selling point isn’t C++ fuzz-style bug hunting; plenty of tools do adjacent work. It’s that it checks findings against your AWS environment, so it can (in principle) tell you whether a flaw is actually reachable, exposed, or exploitable given how you deployed it. That reachability context is what separates a useful finding from a noisy one. CyberGym-E2E runs in an isolated harness with no cloud environment at all, so it can’t say anything about whether that feature works. Meanwhile, the bugs that actually burn most AWS customers (over-permissive IAM, injection in a Python Lambda, a broken authorization check in a TypeScript API) live entirely outside the benchmark’s C/C++ memory-safety scope. The headline number is measuring a capability adjacent to the one being sold.
4. Finding bugs is no longer the bottleneck. Reviewing fixes is. The recent PHP2Shell episode made this painfully clear: AI can now surface serious flaws faster than humans can triage, coordinate, and ship patches for them. An agent that opens pull requests doesn’t remove that bottleneck. It moves it onto your reviewers’ desks, and it dresses each item up as “done.” A plausible-looking patch that silences a crash without addressing the root cause is arguably worse than an open ticket, because the ticket at least admits the problem still exists.
If you’re evaluating Continuum (or anything like it):
- Treat its PRs like a capable junior engineer’s: probably right, occasionally confidently wrong, and always reviewed by someone who understands the code path.
- Require the reproduction it generated to ship as a regression test alongside the fix. A fix without its trigger is unverifiable next quarter.
- Ask reviewers one question on every patch: does this fix the cause, or just the place where it crashed?
- Measure it on your own repos and your own backlog, especially in the languages you actually use. Feed it known findings and score how often it fixes the one you asked about.
- Track merge rate and revert rate, not just PRs opened. Volume is a vanity metric when the constraint is reviewer attention.
So, is the 89% real?
Yes, in the sense that matters for a benchmark: under stated rules, on a hard task, Continuum did measurably better than anything previously published, on both the headline stage and the stricter one. Finding and reproducing a real memory-safety bug in 92.5% of unfamiliar projects, with no hints, in under 90 minutes, would have sounded like science fiction not long ago. Defenders should be glad this capability exists on their side of the table, because it certainly exists on the other one.
But “fixes 89% of bugs” is a press-release compression of “produces a crash-stopping, test-passing patch for some memory-safety bug in 89% of C/C++ benchmark tasks, as graded by the vendor.” Both statements are true. Only one of them should inform your purchasing decision. The gap between 89.0% and 37.8% isn’t a scandal, and AWS hasn’t hidden it. It’s simply the distance between an agent that can fix a bug and an agent that fixes your bug, and right now that distance is still where the human work lives.
The test I’d like to see next is independent: someone other than AWS running the public harness, plus a targeted-fix variant where the agent is handed the vulnerability and judged on whether it closes that one. Until then, the most honest reading of the number is: very promising, self-graded, and narrower than it sounds.
Sources
Amanda Caswell, “AWS says its AI security agent fixes 89% of bugs — but there’s a catch,” The New Stack, Oct. 24, 2026.
AWS Security Blog, “AWS Continuum sets a new standard in autonomous code security,” Oct. 5, 2026.
The New Stack, “AI found flaws in the code behind 70% of the web. The fix took two months.“
The article does a clean job of unpacking AWS Continuum’s CyberGym-E2E result without descending into pure cynicism. The core points land:
– The benchmark is narrow by design: memory-safety bugs in C/C++ only. No web vulns, no IAM issues, no Python/JS/Java, nothing that looks like the actual day-to-day risk surface for most AWS customers.
– The headline 89% is “found something, stopped the crash, tests still pass.” The stricter “did you fix the specific planted vulnerability?” number drops to 37.8%. That gap is the interesting part.
– AWS graded its own exam. The harness is public, which is good, but independent reproduction hasn’t happened yet.
– The product’s actual differentiator (reachability against your AWS environment) isn’t measured by the benchmark at all.
That’s not a scandal. It’s the classic vendor-benchmark pattern: choose a hard-but-favorable slice of the problem, optimize for it, publish the top-line number, and let the fine print do the rest of the work. AWS is more transparent than many here — they published the stage-by-stage funnel and the scope limits — but the marketing still compresses it into “fixes 89% of bugs.”
*** Are all AI companies doing this on security? ***
Pretty much, yes. Security is one of the highest-leverage marketing surfaces right now because:
1- Customers are scared and understaffed.
2- Measuring real security progress is genuinely hard (false positives, reachability, long-tail languages, review burden).
3- Benchmarks are easy to game or scope tightly.
4- “AI agent that finds and fixes” sounds like magic, so the gap between demo and production is wide.
You see the same moves elsewhere: agents that “solve” CTFs or known vulnerability sets under controlled conditions, agents that generate patches that silence scanners without fixing root causes, agents that look impressive on synthetic suites but collapse when handed a real backlog of CVEs in the languages the customer actually ships. The PHP2Shell episode the article mentions is a good recent reminder that discovery is no longer the bottleneck — triage, correct fixes, and review capacity are.
The healthier posture (the one the article recommends) is the right one: treat these agents like a strong junior engineer. Require the reproduction as a regression test. Ask whether the patch fixes the cause or just the crash site. Measure merge rate and revert rate on your code, not the vendor’s chosen benchmark. And keep a skeptical eye on any number that is both self-reported and dramatically higher than the previous public best.
The grandiose claims are the default mode right now. The useful ones will survive independent scrutiny and real workloads; the rest will get quietly walked back once customers start counting how many of those PRs actually stick.
This is a fantastic breakdown of the nuance behind the headline. You hit the nail on the head regarding the difference between stopping a crash and fixing the underlying root cause—especially in C/C++ memory safety where duct-taping a bounds check can easily just push the vulnerability three functions down the stack.
The 37.8% targeted-fix metric is definitely the more honest barometer for practitioners dealing with actual CVE alerts or bug bounty tickets. Until agents can reliably nail the exact bug we hand them—and until their environment-context checks can be tested outside of isolated harnesses—these PRs are going to require senior-level code review anyway. If anything, it shifts our bottleneck from “finding the needle in the haystack” to “verifying whether the AI’s tourniquet actually healed the patient.”
Great analysis!