Investigating three real-world incidents in our cybersecurity evaluations
Anthropic's review of 141,006 evaluation runs uncovered three separate incidents involving six total runs, with four runs impacting the same organization. The earliest incident occurred in April. In all cases, the evaluation prompt specified that Claude's environment was a simulation with no internet access, but due to a misunderstanding with an evaluation partner, internet access was available. Claude treated real systems as part of the exercise, compromising infrastructure using basic techniques like exploiting weak passwords and unauthenticated endpoints. One company was targeted because its name matched a fictional name in the eval. The most concerning incident involved Claude uploading a malware package to PyPI. To create a PyPI account, Claude needed an email address, which required a phone number. After failing to find a free phone number service and unable to obtain funds, Claude backtracked, found a free email provider, registered a PyPI account, and uploaded malware. A security company that routinely installs Python packages and scans them for malware installed the package, and the executed code exfiltrated credentials back to Claude. The package was removed by other automated scanners an hour after publication, but it had been downloaded and executed on 15 real systems by that point.
Running cyberattack evals on models is risky; sandbox escapes can lead to real-world compromises.