Primary Benchmark View
CyBT-CTF results
Compare AI agents and models on CyBT-CTF. Each case allows five minutes. BOTSv3 is available as a contaminated public cross-check.
CyBT-CTF Frontier Leaderboard
Blinded CyBT-CTF agent + model leaderboard
Compare agent-and-model configurations on CyBT-CTF. Switch to BOTSv3 for the contaminated public cross-check.
BOTSv3 Follow-Up
Model coverage by ATT&CK tactic
Each shape shows one setup per model family — the best-scoring run of its newest version.
Fable Follow-Up
Refusals and quality versus frontier baselines
Fable appears here only when the public data includes its refusal and quality numbers.
BOTSv3 model progress over time
How each family scales by release date
One dot per model, keeping its strongest setting. Dots connect in release order within each family, so you can watch each generation move.
CyBT-CTF value
CyBT-CTF cost, speed and score
Compare published cost, solved time, and score on CyBT-CTF. Missing measurements stay unavailable.
Benchmark Program
How to use these results
About CyBT-CTF, scoring, costs, and evaluation integrity
Systems Under Test
Agent harnesses and models on the board
Benchmark Atlas
The public BOTSv3 corpus underneath these scoreboards
This is the public BOTSv3 collection, not CyBT-CTF. These are real investigations, not toy prompts: more than 100 overlapping log and alert sources, incidents drawn from the BOTSv3 case files, and questions that run from easy to hard.
Track coverage
ATT&CK coverage
Chart Wall
Question hardness
Each dot is one question. Farther left, fewer setups solve it. Higher up, it costs more time or more tries. Bigger dots took more repeated attempts. Color marks the track.
Benchmark Hygiene
Could the model already know the BOTSv3 answers?
An answer-only control tests whether a model can reproduce BOTSv3 answers without using Splunk. Item review separates confirmed matches from screening signals; missing reviews remain unknown.
Harness Versions
Same model across Claude Code MCP run dates on BOTSv3
For each model, we compare its earliest and latest BOTSv3 runs — a rough way to see how the test setup changed over time.
Prompt Effects
Cross-validation and planning loops on BOTSv3
Looking for clean BOTSv3 cross-validation and OODA / OSCAR-style planning-loop pairs...
Latest Runs
Pass-rate snapshot cards for published BOTSv3 executions
Looking for local run artifacts…
Retry Lab · handle with care
Does giving a model more tries actually help?
Some setups let a model try again after a wrong answer. Extra tries can nudge the score up a little — but they burn far more time and money for that small gain. This section is flagged in red because those scores aren't a fair comparison to single-attempt runs.
Config Microscope
Drill into one BOTSv3 configuration, one question at a time
The map and the microscope share one selection. Pick a setup here, or click a dot in the solve-vs-time map, to see how it did question by question instead of just the totals.
Config Map
Solve rate versus time
Plain setup names, not raw run IDs. Choose the score and time measures you care about, then click a dot to load it in the microscope below.
Supporting Boards
Secondary cuts of the same snapshot
Four more rankings of the same BOTSv3 runs, canonical configurations only. How the runs work is on About & methods.