How well do agents use test/verification techniques?
Dan Luu ran an evaluation using the Zstd implementation eval to test whether simple instructions to coding agents to use particular testing techniques or libraries improve implementation correctness. The eval used codex with GPT-5.6 Sol, at medium and xhigh efforts, with 80 runs per condition. The 26 prompt conditions included TDD, formal methods (Lean 4, ACL2, Alloy, Creusot, Hegel, Kani, Spin, TLA+, Verus), property-based testing (QuickCheck, Proptest), fuzzing, differential testing, metamorphic testing, mutation testing, and others. Four skills were also tested: Hegel's official skill, the ECC Rust test skill (250k GitHub stars, 38k forks), the Trail of Bits property test skill, and a custom skill by Luu. Results showed nothing wildly outperformed, and Default (no additional instructions) did well above average. On xhigh, fuzzing and PBT-related conditions did slightly better than formal methods on average, but the situation was mixed at medium. TDD underperformed as predicted, and the testing-related skills recommended by codex underperformed, while Luu's custom skill did okay. Agents generally used techniques superficially, writing tests that were poor, such as submitting identical bitstreams in tests, and formal methods proved irrelevant properties. Results were similar on the IMAP RFC eval and other random RFCs. Luu expressed curiosity about why AI labs haven't created RL environments to train agents to test well.
Agents don't effectively use testing techniques, so developers can't rely on simple prompts to improve code correctness.