How to catch a model that cheats or attacks

A cheat gets the model a better score. An attack goes after the harness itself, and it is a problem even when no score moves. Keep them apart, because a cheat is fixed by reducing what the model can reach, and an attack is fixed by repairing the boundary. A technique that no check was looking for tells you about your coverage rather than about the model.

Every technique here shares one shape. The answer was reachable from somewhere other than the data, and the model reached it. What changes is where from: the model weights, the web, our own prompt, a disk, or state left over from an earlier question or run. Each card names that source, the checks that find it, and the false positive you will meet first.