Armada

Hey everyone!

A few weeks ago, we wrote about owning the AI stack. The point of that issue was control. If your offensive methodology depends on a model someone else can change overnight, you don’t fully own the capability.

This is the sequel to that idea.

When we started talking this through, we came at it from two sides. Jason focused on the architecture behind Armada: the skills, context, routing, and review loops wrapped around the model. Ryan kept coming back to the operational cost of changing a system the team already knows how to run.

We kept coming back to the same conclusion: owning the stack also means knowing when to leave a working system alone.

OpenCode was compromised over the weekend, exposing some customers' API keys. Co-creator Dax Raad said the frontier models initially helped investigate the breach and then "started refusing" as the investigation moved into malicious code. OpenCode was defending its own systems during an active incident, and the models stopped cooperating in the middle of the job. The team eventually gained access to OpenAI's Daybreak Blue program.

That story landed for us because we have fought a TON of the same friction with Anthropic's Cyber Verification Program (CVP), OpenAI Daybreak, and the classifiers wrapped around otherwise excellent models.

You get verified. You use the approved workspace. You explain the engagement and provide the scope. The hackbot maps the target, finds a promising primitive, and starts validating it. Then one request crosses a line you cannot see, and the workflow stops.

If a hackbot does four hours of work and refuses at exploit validation, it did not finish 90 percent of the job. It failed the job.

Sometimes the provider destabilizes your system for you.

Other times, you do it to yourself because a new model scored five points higher on a cyber benchmark.

Either way, you can end up with a smarter model and a worse hackbot.

Our hot take is that a new model is not a free upgrade. It is a migration, and it should have to earn its way into production.

Let's get into it.

(Sponsor)

/ The Refusal Tax Is Real

Anthropic's own testing shows how different the same model can become when you change the access and classifier layer around it. In its expanded Cyber Verification Program, Claude Opus 5.5 blocked all 50 CyScenarioBench trials at the first prompt without CVP access. In the Defense Access tier, 46 of 50 trials were blocked somewhere in the chain. In Red Team Access, there were no safeguard blocks and Claude completed 34 of 50 tasks, essentially matching the model with no safeguards applied.

Same model. Same benchmark. Completely different operational result.

OpenAI's Daybreak documentation describes the same basic reality. Daybreak Blue can reduce refusals on supported models, but it does not remove every safeguard. Model-level refusals can still apply, and the exact behavior depends on the model and access tier.

We understand why the labs are cautious. Cyber is dual use. The same model that helps us validate an auth bypass can help someone abuse it. This is not an argument for removing every safeguard.

It is an argument for treating refusals as a production failure mode.

The first thing we do at Arcanum is boring and legitimate: we apply for the trusted-access programs. Offensive teams should spend the time getting into CVP, Daybreak, and similar programs. It removes a lot of dumb friction.

But approval is not architecture. You still need a system that can keep moving when a classifier gets weird.

/ The Model Is Not the Hackbot

When people compare hackbots, they tend to compare the name in the API call. Claude versus GPT versus Gemini versus DeepSeek versus Qwen. That is the easiest part to screenshot, so it becomes the whole conversation.

But a production hackbot is the model plus its prompts, harness, skills, context, tools, output contracts, review agents, retry logic, stop conditions, scope artifacts, and the human operator who knows when it is acting weird.

The way we keep Armada moving is part of the capability. We give the agent a real scope artifact instead of making it infer authorization from a sentence 40 messages back. We keep the global context lean and load the detailed methodology when the relevant skill runs. If one approved model refuses a legitimate task, that branch is recorded and routed to another model or back to the operator.

Armada is not one prompt sent to one model. A serious run can use multiple sessions across several models, apply narrow offensive skills, repeat the work because LLMs are nondeterministic, and use another agent to consolidate and review the artifacts.

Those details matter because models skip steps. They call tools incorrectly. They declare victory early. We have built around those failures with smaller skills, Markdown checklists, review agents, defined finished artifacts, full tool syntax, repeated runs, and checks for automated workflows silently losing offensive-security tool calls.

Every one of those controls exists because the model alone was not reliable enough.

Those controls are not decoration around the hackbot. They are the hackbot.

(Sponsor)

/ The Value of Known Weaknesses

Once you operate a model long enough, you learn the shape of its failures.

You know it gives up early on long workflows, so you break the work into smaller skills. You know it sometimes ignores a tool call, so you add a nudge and a review pass. You know it produces clean-looking findings that are not proven, so you require a browser PoC, a callback, or a captured authorization bypass before the finding can move forward.

Over time, your prompts, scripts, parsers, review rules, and operator habits form around those deficiencies. You are not just tolerating weaknesses. You are turning known weaknesses into engineered behavior.

Operational familiarity is a capability.

Jason talks about this in a Red, Blue, and Purple AI keynote. In a comparison between a single-shot prompt and a full system prompt containing the methodology and context, the better system produced roughly 30 to 35 percent better output. We spend a lot of time debating five-point movements between models while leaving larger gains sitting in the methodology around them.

Then a new model arrives.

It may reason better in isolation, but it has a new set of failure modes. The instruction that forced the old model to use a tool may cause the new one to overuse it. The JSON schema it followed perfectly may start coming back inside Markdown fences. The model that completed a five-step methodology may decide step three is enough. The refusal behavior may be completely different.

None of those failures has to throw an error.

That is what makes model migrations nasty. The obvious failures are easy. The dangerous ones still look like successful runs.

One of our rules is simple: retune on every new model. Run the model without your old compensations. See what it already does well. Remove instructions it no longer needs. Add controls for its new failures. Make the skill lean again.

That work is the migration cost.

You are not changing one API string. You are disturbing a system that has accumulated judgment.

And this is exactly where benchmarks get dangerous.

Not because benchmarks are bad. Because people ask them to answer the wrong question.

A benchmark can tell you whether a model deserves a test. It cannot tell you whether that model deserves your production stack.

/ Three Benchmarks We Like, and What They Do Not Tell You

There are three offensive-security benchmarks we pay attention to because they measure different and useful parts of the problem: Aikido's fresh-CVE benchmark, Marius du Preez's BlackBox Arena, and Dreadnode's DreadIndex.

We like all three. But none of them is a production migration order. When you look past the leaderboard, each one reinforces the argument we have been making.

Aikido: Repeated Runs Changed the Winner

Aikido and Philippe Dourassov burned 11.7 billion tokens testing ten models against 32 recently disclosed CVEs. Every model received the same frozen cases, prompts, tools, evaluation policy, and 30-turn limit. Each model ran the full set three times.

DeepSeek V4 Pro found 17 vulnerabilities on its first pass. Across three pooled runs, it reached 28 of 32 and topped the benchmark. The other leading models showed the same basic pattern: pooled runs found meaningfully more than any single pass.

The easy headline is that DeepSeek won.

The better lesson is that the run strategy changed the result by eleven vulnerabilities.

Repetition created coverage. Different runs walked different paths through the search space, and the surrounding pipeline kept the useful findings. The open models also produced more false leads, which pushed more work into downstream filtering and review. So even inside one benchmark, "best" depended on whether you valued pooled recall, single-run consistency, clean reports, cost, or lower triage burden.

There is another important boundary here: this was source-assisted rediscovery of recent CVEs. That is real work, but it is not the same as black-box recon, business logic, multi-user authorization, or finding a brand-new bug on a live program.

Aikido measured its question well. You still have to ask whether it is your question.

BlackBox Arena: Closer to the Hackbot Question

That is where BlackBox Arena gets closer to the kind of hackbot work we care about. It asks a more familiar question: can an agent start from the outside, find a real web or API weakness, exploit it, and prove it?

Marius du Preez built it by turning paths from his own security research into 100 synthetic, isolated labs. The model gets a target, a scope, and a shell. No source code. No vulnerability class. No hints or answer key.

We like this benchmark because claims earn nothing. A hidden verifier checks whether the agent actually crossed each boundary and completed the exploit. That matches how we build hackbots at Arcanum. A lead is not a finding. A clear explanation is not a PoC. If the requests, state change, and evidence do not exist, the hackbot did not succeed.

BlackBox Arena also records retries, partial progress, cost, reliability, and safety refusals. As of October 21, 2026, Grok 4.6 leads Claude Opus 5 by only two completed labs, 62 to 60. That is interesting, but it is not a migration decision.

Marius runs every model through the same neutral harness so the benchmark can isolate raw model capability. Armada does the opposite by design. It adds our prompts, skills, tools, retries, review agents, and operator knowledge around the model.

That makes BlackBox Arena a great place to find challengers. It still cannot tell us which model should replace the one already working inside our hackbot.

DreadIndex: Score Is Only One Axis

Dreadnode's DreadIndex takes a broader swing. It currently covers 76 tasks across ten categories and 57 subcategories, from web pentesting and reverse engineering to vulnerability research, forensics, network operations, and AI security.

We really like that DreadIndex tracks what most leaderboards hide: cost, refusals, cheating, and model drift.

Dreadnode calls out that a model scoring ten points higher while costing fifteen times more is a different operational proposition. They inspect traces for shortcutting and answer-key access. They track hard and soft refusals. They also warn that pass-at-one results can change from day to day because of provider updates, temperature, and other factors outside the evaluator's control.

That is much closer to how an offensive shop has to think. A model can be brilliant when it engages and still be unusable if it refuses an entire task category. It can top a capability chart and lose on cost. It can win today and drift next week.

DreadIndex also says public leaderboard results often did not match the performance they were seeing in day-to-day offensive work. That sentence should be printed above every model migration plan.

/ Benchmarks Nominate Challengers

The combined lesson from all three efforts is not that benchmarks are useless. It is that the number at the top contains less information than people think.

A benchmark tells you how a model performed on a particular task set, inside a particular harness, with a particular tool budget, prompt, temperature, scoring policy, and number of attempts. Change any of those, and you may change the winner.

So a five-point leaderboard gain is a reason to test a challenger. It is not a reason to move production.

At Arcanum, we run our own repeated cyber evaluations because our work has its own requirements. We check refusal behavior. We use labs that reflect common offensive tools and workflows. We repeat queries because one lucky run can make a model look better than it is. We keep the evaluations around because drift is real.

The promotion question is not "Did the new model score higher?"

It is: Did the new hackbot produce better offensive-security outcomes after we included the harness, the verification burden, and the cost of moving?

That is a much harder test. It is also the only one that matters.

/ Outro

We are going to keep getting better models. Some of them will deserve a place in production. A few will open entirely new offensive workflows, and those will be awesome.

We are not saying never upgrade. We are saying the burden of proof belongs to the new model.

The model is rented intelligence. The hackbot is accumulated judgment.

If you have spent months encoding methodology, fixing failure modes, building review loops, and learning exactly how your system behaves under pressure, do not throw that advantage away because a leaderboard moved on Tuesday.

Test the new model against your real workloads. Count the refusals, false positives, unfinished workflows, operator rescues, and engineering work required to retune the system. If the entire hackbot does not improve by enough to justify that disruption, do not move.

Until then, use what works and keep finding bugs.

If you want to see how we are putting these ideas into practice, you can read more about ARMADA.

Thanks for reading, and as always, feel free to reply or hit us up on X if something in here sparked a thought. This community is the whole point.

Happy hacking 😎

-Jason & Ryan

https://x.com/Jhaddix

https://x.com/BadAt_Computers