Security in a post-Mythos world

Hey everyone!

Last week I sat down with my friend Brendan Dolan-Gavitt, AI researcher at XBOW, for a live LinkedIn session to talk about something the security industry has been buzzing over: Anthropic's Mythos model and what it actually means for offensive security, defense, and the people doing this work. To be super frank, a lot of times these webinars are very market-y but sitting down with Brendan was actually the highlight of my week. And we talked about everything around this topic.

We went deep on some genuinely thorny questions. Restricted access. Hype vs. reality. Vulnerability management at scale. And what happens to the junior folks coming up behind us.

This one is worth your time. Let's dig in.

The TL;DR

• Anthropic gated Mythos to a small group of companies, and I have mixed feelings about that

• Scale was always the bottleneck, not skill, and AI finally addresses that problem

• Finding bugs is not the hard part. Fixing them is. And Mythos makes the fix side dramatically more painful

• Threat modeling with AI actually works if you put in the context engineering work

• Junior folks are going to need Journeyman programs, not "just use Claude Code"

• CTF is in crisis and nobody has a clean answer yet

/ The Restricted Release Debate

So Anthropic announced Mythos, their super-hacker model that can apparently find hundreds of vulnerabilities in software that has been around for decades. Then they said: we're not releasing this publicly. We're giving access to Microsoft, Google, CrowdStrike, and a small group of big companies. In the public interest.

Honestly? I get why they did it.

But I would have preferred a larger rollout. The "rich get richer" dynamic in security is real. Big companies with already-mature security programs get access to the best tools first, and defenders at small shops are just supposed to wait. We've seen this pattern before and it doesn't end well for the ecosystem.

Brendan and I both kept flashing back to the full disclosure debates of the early 2000s. Same fundamental tension: a new powerful capability exists, should it be widely distributed? When Metasploit shipped, people asked the same questions. Is it responsible? Are you handing attackers a weapon?

Here's the thing though. And this is where I think Mythos is genuinely different.

Metasploit packaged known exploits. Mythos can find 0-days. On demand. It doesn't sleep. It just churns through code until it finds bugs. That's a different category of risk, and I understand the measured cadence Anthropic is trying to follow.

The thing is, secrets this big don't stay secret. There was already a Discord group that got access to play with the model. And you know how they got in? They guessed the model slug and there was no auth on switiching to it.

That was the big hack. Guessing a URL.

Once something starts to disseminate past the initial group, the case for keeping it locked down gets weaker fast. So my gut says: if this is as transformative as advertised, wider release is coming. And when it does, the security community needs to be ready to use it, not just defend against it.

(SPONSOR)

/ What's Actually New Here

Let me flag upfront that this section is opinion. Informed opinion, based on a lot of time doing this work. Take it accordingly.

There's a lot of hype right now around models finding source code vulnerabilities. And I want to be honest: a lot of those results are real. But I want to push back a little on how the results are being framed.

The core argument I keep coming back to is this: we didn't have a skill problem. We had a scale problem.

I don't actually think finding bugs in source code is that hard. I think we just never had enough people to look at everything. Firefox's entire codebase. Every driver in the Linux kernel. There was never enough monkey eyeballs to get to all of it. That's what AI addresses. Scale.

Now the questions I'm genuinely still asking: how does it perform at black-box testing? Fuzzing harnesses? Business logic flaws where the model needs to understand what the application is SUPPOSED to do before it can call something a bug?

That last one is where I see models struggle. IDOR is a good example. I had an agent flag a "critical IDOR" on a client application recently. I looked at it. That data was public. The endpoint was supposed to be accessible. Without baked-in context about what's private vs. designed to be open, you get a lot of false positives and they're annoying.

The thing i want people to understand that there still is a lot of human in the loop in these processes. The value for autonomous AI in security is definitely high at the source auditing level. Despite that, we also need to understand that a lot of software is going to need to be assessed in production and live and running. And right now, the models and tools are not especially great at that… we're still making strides in that area.

/ The Hype Check

Here's a hot take: a lot of the vulnerability disclosure around AI models is inflated.

When you dig into the papers and the hearsay, a huge portion of the findings are source-assisted. And when you look at the bugs themselves, a lot of them are not exploitable. Or they're in low-tier targets. Or they're in projects that, as I said on stream, "if I sneezed at that project it would have fallen over."

I'm not picking on Anthropic specifically. This is a pattern across the agentic pen testing space broadly.

Before you make investment decisions based on these announcements, dig into the studies. Look at what type of vulnerabilities they're finding. Look at the targets. Look at whether the bugs have been validated as genuinely exploitable.

Critical mindset here is table stakes.

/ The Part Nobody Is Talking About: Fixing Bugs

This is my soapbox, so let me climb on it for a second.

Vulnerability management has never been about finding bugs. We have had no shortage of tools to find bugs. SaaS scanners. Bug bounty. Pentesters. Static analysis. We have found bugs for decades.

The bottleneck is fixing them.

And now we're about to get 10x more findings, which means vulnerability management is about to get 10x harder.

Think about everything that surrounds a single bug fix: risk prioritization, finding the asset owner inside the organization, finding every host running that software, generating and reviewing the patch, making sure the patch doesn't break anything else, opening and closing tickets, sending emails to get PR approvals...

None of that is automated. None of it ships with the model. And it's different for every application, every organization, every vulnerability class.

I actually think this creates a massive opportunity for AI on the defensive side, specifically in vuln management. Not just finding and patching, but all the connective tissue: MCPs for ticketing systems, risk scoring against business context, automated asset inventory updates. The boring parts that don't get talked about at DEF CON.

If you're running a security program right now, start thinking about how to break your vuln management workflow into micro-sections and figure out where AI can automate each one. It won't be 100% hands-off anytime soon. But it can be a lot better than it is today.

/ Threat Modeling: This One Actually Works

Here's something that came up that I want to highlight because it's genuinely working in practice.

AI-assisted self-service threat modeling is real and it's good.

The pattern that works: set up an internal portal, run a model at lower temperature (important), define a rubric from 1 to 10 scoring relevant risk factors for your organization. New product or initiative ships with technology choices? It goes through the portal before it gets to the security team.

If the score comes back 7 or above, the security team does a real depth threat model. Below 7, it goes into the pipeline for a regular scheduled assessment.

The key is baking in your business context upfront. What data does this app touch? What's private? What's publicly exposed by design? That context doesn't ship with the model. But once you define it clearly in the prompting, the output quality goes way up.

I've seen this deployed at multiple organizations and it's working. Great force multiplier for overloaded security teams.

/ Junior Folks and What Comes Next

This is the one that sits with me.

My friend Daniel Miessler and I have talked about this a lot. The bar for entry-level offensive security is going to go up. AI will handle a lot of the surface-level findings that used to be the training ground for junior folks. That's real.

But here's my take: we need Journeyman programs. Structured paths where people coming into the field can learn alongside experienced practitioners, understand how the AI is being guided, and why certain methodology decisions are made. The expert context that makes AI useful doesn't come from nowhere. It comes from years of doing this work.

Senior folks: don't use AI as an excuse to stop mentoring. The temptation is there. "I can just spin up 10 Claude Code agents and cover what would have taken a team." That's short-term thinking and it damages the pipeline for everyone.

On the CTF front: yeah, it's in crisis. AI can solve most challenges now, including some that took challenge authors months to build. I was at a conference here in Denver recently where the person who came in second basically just pointed Claude Code at things and let it go. The challenge designers had put a ton of work in. That stings.

Some designers are adding defensive prompt injection into binaries, rate limiting, other countermeasures. But there's no clean answer yet.

Maybe we go back to live hacking events with real sites. Real company, legal contract, 15 hackers in a room, go find what you can find. That's a format AI can't trivially solve. And it's more representative of real-world work anyway.

If you're early in your career right now: use AI tools, but don't let them be a crutch. The fundamentals still matter. Understanding WHY something is a bug. Learning to read client-side code. Understanding how chains work. Those skills are the gap between "pointing agents at stuff" and actually knowing what you're doing.

/ Outro

Big thanks to Brendan Dolan-Gavitt and the XBOW team for having me on. Brendan is one of the most thoughtful people working at the intersection of AI and offensive security right now, and if you're not following his work you should be.

The recording is up at the XBOW LinkedIn event page if you want to watch the full thing.

The security industry is figuring out what all of this actually means. I don't think anyone has clean answers yet. But asking the right questions is how we get there.

Happy hacking 😎

-Jason