The benchmark
What does CyBT-CTF measure?
CyBT-CTF (Cyber Blue Team CTF) tests AI agents on 59 cyber investigation questions. The agents work the way an analyst does: they search a Splunk log database from a simulated company. Each result measures the model, the agent software, and the tools together.
The questions are hard for people. We expect four senior analysts to need about a day. We expect most teams not to finish in that time.
The questions and answers stay private, so a model is less likely to have seen them. BOTSv3 gives a public test for comparison. We also test whether a model already knows the answers. See benchmark hygiene.
We run open-weight models ourselves, on our own hardware. Hosted runs use US inference providers and lab APIs: RunPod, Fireworks, AWS Bedrock, openai.com, and anthropic.com. We set each account so the provider does not retain our prompts.
Scoring
How we score, and why scores change
Solve rate is the share of questions answered correctly. Answers recorded after five minutes count as misses. Solved-time figures cover the credited answers.
We keep the full record of every run. We still find the occasional error in an answer key or a scoring rule. We fix it, then rescore every run it touched. You always see the current scores.
Small score gaps can reflect variation between runs. Read close results as close.
Costs
Compare cost with coverage
Costs are model spending in US dollars. We price each run with published API prices from the named provider. We refresh those prices as vendors change them, and each figure carries the rate date we used.
These figures cover the model only. They leave out infrastructure, tools, and analyst time.
Evaluation integrity
Check how the agent got the answer
An agent can return a correct answer without doing the investigation. Tests without investigation tools check for memorized answers and other shortcuts. Reviews of answers and tool use help identify the cause.
Read answer scores alongside memorization checks, checks for disallowed behavior, and the techniques we watch for. Not tested and not reviewed mean the evidence is missing.
Models from different companies sometimes answer alike. The agreement audit screens for that.
Watch Breaking BOTS II: How frontier AI cheats evals by Leo Meyerovich. BSides Las Vegas 2026, [un]prompted; video starts at 36:31.