LobstersSunday · August 23, 2026FREE

Robot comment classifier

The author, working on a codebase, encountered comments that were generated by AI robots. These comments contained plausible-sounding but hallucinated details, such as claims about "the usual case" or the importance of an optimization. To aid in identifying such comments, the author wrote a classifier for source code comments. The classifier correctly identifies the opening example as robot-generated with near certainty, and the second example with high confidence. However, the author cautions that accuracy is only around 80%, with misclassifications happening often enough to appear in a short table of spot-checked examples. The author attributes the accuracy ceiling to the short length of code comments (often under 20 words) and inaccurately labeled training data. The classifier's most interesting aspect is what it looks at: character frequency analysis reveals that robots use more em dashes, semicolons, unicode arrows, ellipses, full stops, commas, parentheses, line breaks, and straight ASCII quotation marks than humans. Robots also use a wider variety of symbols and higher symbol density. The author notes that the training data likely comes predominantly from Anthropic models, so the features distinguish Anthropic LLMs from humans rather than robots in general. The author concludes the model is good enough to be an additional piece of evidence but not trustworthy alone.