1Password's AI patching benchmark is misleading
Trail of Bits published a post arguing that 1Password's report, published August 6, 2026, gives defenders a misleading picture of AI patching. The headline figure states models produced clean fixes only 26% of the time. Trail of Bits says that figure includes experiments that deliberately instructed agents to apply the wrong fix and experiments in which agents could not compile or test their patches. It identifies four choices behind the headline: the sample selected six vulnerabilities because their fixes were complex, with clean-fix rates ranging from 3% to 60% across those bugs; two prompts telling agents to apply the wrong fix accounted for 22% of the data; one evaluation mode preventing agents from building or running code accounted for 36% of the data; and models ran at different reasoning settings, with GPT-5.5 at medium effort and Opus 4.8 at high, the tools' defaults, and neither tested at its highest available setting. Trail of Bits says it reanalyzed the patches and test results published with the study, keeping trials where agents could run code and were not instructed to apply the wrong fix, and excluding runs the study classified as having consulted the upstream fix. In those trials, it reports 2,634 of 3,067 patches generated by 1Password's models (86%) blocked the supplied exploit. Trail of Bits states that blocking that exploit does not establish a complete repair, but says the results show useful patching capability under reasonable working conditions that the headline fails to convey. It also says the instructions and grading introduce further problems, and is releasing two agent skills: post-patch-validation to help agents test fixes, and review-walkthrough to help engineers review them.
Trail of Bits says teams taking the 26% headline at face value may leave repairable vulnerabilities unaddressed, based on its reanalysis showing 86% exploit blocking in testable trials.