Stealing Reasoning Traces from Proprietary LLM APIs
A new paper, highlighted by Simon Willison, demonstrates a method to extract hidden reasoning traces from proprietary LLM APIs. The researchers found that Anthropic, OpenAI, and Google return encrypted chain-of-thought blocks to clients, which can be replayed across sessions, users, and models. By taking a trace produced by a frontier model and replaying it into a weaker sibling, they were able to jailbreak the weaker model and recover the stronger model's hidden reasoning in plaintext. The paper includes a concrete example using OpenAI's API, showing encrypted content blocks in the response. The authors discovered that all models under the same family used the same encryption key, allowing them to feed the encrypted blocks into the weakest family members and jailbreak them into outputting the unencrypted reasoning. The attack was easiest against Claude Haiku 4.5, using a specific prompt to transcribe the reasoning. The paper's appendix includes extensive reasoning traces extracted from proprietary models, revealing raw chains of thought never intended for human consumption. The attack has since been fixed, as all model providers acknowledged the report and the authors were unable to launch the same attacks afterward. The paper also uncovered a prompt injection variant: tricking a model into thinking about exfiltrating data as part of its reasoning trace, then feeding that encrypted trace back into another model, which is more likely to follow instructions in its own reasoning traces.
This attack reveals a practical method to extract hidden reasoning from proprietary LLMs, highlighting a security flaw now fixed.