smevals - a small eval suite for evaluating models, prompts, and harnesses
Simon Willison introduced smevals, a tool for running small eval suites to evaluate models, prompts, and harnesses. Developed in collaboration with Jesse Vincent's Prime Radiant applied AI research lab, smevals aims to answer questions about model capabilities. The tool is designed to be used via a coding agent: users can run `uvx smevals docs` to learn the tool, then build an eval suite. An eval is a directory with YAML files, and can be run against models with commands like `uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6`. Runs are separate from grading; users can grade runs with `uvx smevals grade path-to-eval/`, serve results via a localhost web server with `uvx smevals serve path-to-eval/`, or build static HTML reports with `uvx smevals build`. Willison provides an example eval suite for evaluating haiku-writing ability. He notes that the most time-consuming part was defining the vocabulary: an eval is a collection of tasks, each task is a specific challenge, runs are executed by a runner, and grading is done by a grader that runs checks, which can be simple operations or custom scripts called checkers. Willison states this is his third iteration on evals and feels it's right, and he looks forward to expanding it and applying it to his own projects.
Provides developers a new tool to systematically evaluate models, prompts, and harnesses with reusable YAML-based evals.