Primary Benchmark View
CyBT-CTF first, BOTSv3 as the public cross-check
CyBT-CTF is a set of cyber investigations the models have never seen, so it is the main comparison here. BOTSv3 is an older public test the models may have trained on, so we keep it as a cross-check you can switch to — not the headline.
BOTSv3 Frontier Leaderboard
Public BOTSv3 agent + model leaderboard
Everything from here down covers the public BOTSv3 test. Each entry pairs an AI agent (Claude Code, OpenAI Codex, opencode, Louie) with the model it drives. For the main, blinded comparison, use the CyBT-CTF table above.
BOTSv3 Follow-Up
Model coverage by ATT&CK tactic
Each shape shows one setup per model family — the best-scoring run of its newest version.
Fable Follow-Up
Refusals and quality versus frontier baselines
Fable appears here only when the public data includes its refusal and quality numbers.
BOTSv3 model progress over time
How each family scales by release date
One dot per model, keeping its strongest setting. Dots connect in release order within each family, so you can watch each generation move.
BOTSv3 Efficiency
Public BOTSv3 efficiency tradeoff explorer
BOTSv3 only. Switch between cost vs. speed, cost vs. solve rate, and cost vs. combined score. Bubble size shows how many tasks each setup solved on the first try, which keeps cost free for the axes.
Benchmark Program
What this benchmark is, and what it refuses to fake.
About CyBT-CTF, scoring, costs, and evaluation integrity
Systems Under Test
Agent harnesses and models on the board
Benchmark Atlas
The public BOTSv3 corpus underneath these scoreboards
This is the public BOTSv3 collection, not CyBT-CTF. These are real investigations, not toy prompts: more than 100 overlapping log and alert sources, incidents drawn from the BOTSv3 case files, and questions that run from easy to hard.
Track coverage
ATT&CK coverage
Chart Wall
Question hardness
Each dot is one question. Farther left, fewer setups solve it. Higher up, it costs more time or more tries. Bigger dots took more repeated attempts. Color marks the track.
Benchmark Hygiene
Could the model already know the BOTSv3 answers?
An answer-only control tests whether a model can reproduce BOTSv3 answers without using Splunk. Item review separates confirmed matches from screening signals; missing reviews remain unknown.
Harness Versions
Same model across Claude Code MCP run dates on BOTSv3
For each model, we compare its earliest and latest BOTSv3 runs — a rough way to see how the test setup changed over time.
Prompt Effects
Cross-validation and planning loops on BOTSv3
Looking for clean BOTSv3 cross-validation and OODA / OSCAR-style planning-loop pairs...
Latest Runs
Pass-rate snapshot cards for published BOTSv3 executions
Looking for local run artifacts…
Retry Lab · handle with care
Does giving a model more tries actually help?
Some setups let a model try again after a wrong answer. Extra tries can nudge the score up a little — but they burn far more time and money for that small gain. This section is flagged in red because those scores aren't a fair comparison to single-attempt runs.
Config Microscope
Drill into one BOTSv3 configuration, one question at a time
The map and the microscope share one selection. Pick a setup here, or click a dot in the solve-vs-time map, to see how it did question by question instead of just the totals.
Config Map
Solve rate versus time
Plain setup names, not raw run IDs. Choose the score and time measures you care about, then click a dot to load it in the microscope below.
Supporting Boards
Secondary cuts of the same snapshot
Additional ways to slice the same public BOTSv3 snapshot.
Benchmark Rules
How the BOTSv3 runs work
Experiment Surface
What we are varying
What we change from run to run on BOTSv3. CyBT-CTF is tracked separately, on the comparison and deep-dive pages.