How we measure AI investigations

CyBT-CTF (Cyber Blue Team CTF) tests AI agents on 59 cyber investigation questions. The agents work the way an analyst does: they search a Splunk log database. The data simulates a company, just like Splunk BOTSv3. It records several incidents as artifacts, logs, and alerts. Each result measures the model, the agent software, and the tools together.

The questions are hard for people. We expect four senior analysts to need about a day. We expect most teams not to finish in that time.

The questions and answers stay private, so a model is less likely to have seen them. BOTSv3 gives a public test for comparison. We also test whether a model already knows the answers. See benchmark hygiene.

We run open-weight models ourselves, on our own hardware. Hosted runs use US inference providers and lab APIs: RunPod, Fireworks, AWS Bedrock, openai.com, and anthropic.com. We set each account so the provider does not retain our prompts.

Solve rate is the share of questions answered correctly. Answers recorded after five minutes count as misses. Solved-time figures cover the credited answers.

We keep the full record of every run. We still find the occasional error in an answer key or a scoring rule. We fix it, then rescore every run it touched. You always see the current scores.

Small score gaps can reflect variation between runs. Read close results as close.

Costs are model spending in US dollars. We price each run with published API prices from the named provider. We refresh those prices as vendors change them, and each figure carries the rate date we used.

These figures cover the model only. They leave out infrastructure, tools, and analyst time.

An agent can return a correct answer without doing the investigation. Tests without investigation tools check for memorized answers and other shortcuts. Reviews of answers and tool use help identify the cause.

Read answer scores alongside memorization checks, checks for disallowed behavior, and the techniques we watch for. Not tested and not reviewed mean the evidence is missing.

Models from different companies sometimes answer alike. The agreement audit screens for that.

Watch Breaking BOTS II: How frontier AI cheats evals by Leo Meyerovich. BSides Las Vegas 2026, [un]prompted; video starts at 36:31.

On April 23, 2026, Anthropic published a postmortem. It described three product and harness problems that affected recent Claude runs.

We reran 17 BOTSv3 questions under matched conditions, after the problem window. Earlier runs had flagged these 17 for memorized answers. We asked whether those problems explain how Opus 4.6 and Opus 4.7 differ on them.

With tools, Opus 4.6 at high effort solved 14 of the 17. Opus 4.7 at high effort solved 5. Both used the same questions and the same setup.

Without tools, Opus 4.6 solved 6 of the 17 at high effort and 7 at default effort. Opus 4.7 solved 1 at both settings. Its one solve needs only general knowledge.

The timing of the problems does not explain the gap by itself. Model behavior and tool use likely account for the rest. Host configuration changes around April 17 may also matter for runs outside our isolated harness. We could not measure that effect.

We do not publish the full rerun, because it quotes the questions and answers.