1Password's AI patching benchmark is misleading
Trail of Bits published a critique of 1Password's August 6, 2026 report on AI patching, calling its headline clean-fix rate of 26% misleading. Trail of Bits says the sample used six complex vulnerabilities, that two prompts instructing agents to apply the wrong fix accounted for 22% of the data, and that one evaluation mode prohibiting building or running code accounted for 36%. Trail of Bits reanalyzed published patches and test results, finding 2,634 of 3,067 patches (86%) blocked the supplied exploit in trials where agents could run code and were not told to apply the wrong fix.
Trail of Bits says teams taking the 26% headline at face value may leave repairable vulnerabilities unaddressed, based on its reanalysis showing 86% exploit blocking in testable trials.


