Red-Teaming after Mythos — Zico Kolter & Matt Fredrikson, Gray Swan

like Matt, been faculty at Carnegie Mellon for a while. I think fundamentally look. I think I think that in some sense, we're all here because we believe in the transformative power of AI, and we think that this is this has already transformed the way the entire sort of software ecosystem works, and it will transform how many other ecosystems work going forward.

The issue, though, is that these systems just fundamentally behave very differently from software we're used to. And I don't mean in terms of AI can find vulnerabilities in software, though it can also do that and is also transforming that. I just mean that AI systems have Inherent.

Inherent different types of vulnerabilities. They can be tricked like people get tricked sometimes. Right?

And so you need a different mindset about security when you're thinking about AI systems. Yeah. And especially when there's the possibility of correlated failures.

Right? So it's not just that there's a lot of AI systems out there. It's that there's actually a few models that everyone is using.

And if you find vulnerabilities in the agents that everyone uses, right, things like Codex and Cloud Code, you can actually start to now essentially have a new exploit, a new class of exploit. Fundamentally, I think there has to be a different mindset about the nature of AI security as there is for traditional security. While a lot of that's going to, of course, happen at the AI companies themselves, labs themselves, there's also a real value.

And, of course, I should you know, to be very clear, the labs are doing a lot of work in these areas. But there's just like in most domains, when a new platform emerges, it's very common for there to also emerge a security system separate from it, right, in addition to it as a separate service that's provided. And I think that's where we are right now with AI.

And I think there's a need for specifically minded AI safety and security providers. There's a demand for this, and there's gonna be much more demand for this coming up. And that's why it felt like a really good time to sort of focus on this problem, both in research, because we still do research on this topic too, and then we're continuing research actually at Grace Swan, but also in terms of a commercial offering.

I think it sort of is it is a common conflation because AI is also very good at At cyber. Solving cybersecurity problems. Right?

Or I shouldn't say solving. I mean, it's good at solving problems too, but it's also good at causing problems, you could say. But fundamentally, their AI systems themselves have the potential to introduce new vulnerabilities.

And so this is not about using AI to make your cyber infrastructure better.

We, unfortunately I mean, we do have the ability to test that out on on smaller open source models. So generally speaking, the issue with this is that frontier models are extremely bad at automated red teaming because they have a lot of safeguards built into them. So if you try to use them to to jailbreak another model, they will actually refuse.

Their safety training, which is itself as a base model, can sometimes be bypassed, but they will often refuse to do this. Maybe they'll hypothetically know how to do it, but but, you know, you need and it's actually an important point because traditionally, this has been an area where both in terms of safety, models don't get better by just being bigger. Unlike most other areas where models do get better by being bigger, safety has not been like that traditionally.

I mean, you have to train them explicitly to be safe, or they won't do that. But on the flip side, they're also honestly better at red teaming by default. You really sort of need to train specialized models for red teaming to make them good at red teaming.

It's awesome for you guys. Yeah. And and and so and and what do you need to do that?

Well, you need lots of data Yeah. From from people that are traditionally much better at red teaming. However, one thing that we are finding, and this is actually, I think, where we're kinda crossing this point too, is that in a lot of the latest experiments, we can do much better than people, than human red teamers now at breaking these models.

When I say we, I mean our automated red teaming model is a system called Shade. That system is now actually quite a bit better at breaking models than humans are. I think we had a recent competition between humans and our model, and it was actually quite a bit better.

So I think I think that there's a lot of ways in which this is a bit different than what we see with sort of normal model progress because it's so out of distribution. In in some sense, the nature of a retimia model is to find things that are inherently out of distribution for that model so as you can bypass this normal behavior. And so that fundamentally is kind of a different thing than what most models can do.

Yeah. Okay. Wyatt Wyatt you should follow him on Twitter if you haven't already.

He makes he makes great he makes really insightful posts. I think he's one of the most sort of insightful people about the nature of LLMs and sort of when new versions come out. I actually frequently look to him to see what's next.

He's a lawyer, I think. Right? He is.

Yeah. He's an attorney.

Our top competitors are often people that, you know,

I think in general, just I mean, you you mean in the kinds of of the arena itself, you mean in general terms of this? I I think Hughes has great insights in sort of the nature of models as a whole.

visceral way. I don't know that it shows that you're not modeling intelligence. I mean, I think these things are intel I think LLMs absolutely are intelligent and maybe will be more intelligent Are at some they conscious?

Conscious is is a weird word, but I I I I actually don't I mean, I I don't think so. I think I think the way that we we're getting super philosophical now. Right answer.

We're getting very philosophical now. I don't think so. I studied philosophy in in in in in college.

So, I mean, this has this has been this is passe to say at this point. It is clearly a different form of intelligence than people. It's some alien intelligence that is vastly different, and that difference is actually often brought out to a large degree by things like adversarial attacks and red teaming.

Because there are certain things that fool humans that would never fool an AI, but there are certain things that fool AIs, they would never fool a human. Right? Yeah.

So it's just it's a different sort of form of intelligence. It's really interesting, actually, that we're sort of have the opportunity to sort of probe and in a really kind of amazingly experimentally controllable fashion. Like, omniscient.

Right? Yeah. I mean I mean, you know, I'll I'll do the analogy to sort of neuroscience here.

It's like we could kind of run experiments on the brain, observe every neuron in it, reset its state to prior states, and run counterfactuals, none of which we can do with humans, and yet we still understand neither very well. Even with that and all that ability, we still don't understand AI, you know, in on supplement level. So it's it's definitely this different form of intelligence, but it it's clearly intelligent.

So I I have I I I could go off. It's little off tangent here. We're getting we're getting we're getting a bit.

Yeah. Does relate. Right?

Yeah. Yeah. Go go ahead.

Do your tangent. Okay. So my tangent here is I have felt that MEKANTERB is also very far behind where capabilities are.

I am newly optimistic, or I should say more optimistic about MEK INTERP Oh. In that I think, actually, as with many things, coding agents have a chance to make this into a science. So the problem with MEC Interp and I'm okay.

So I I I shouldn't say the problem. I don't wanna call it a field. I'm I'm I we do some work, though, so to say.

It's roughly MEC Interp, but I'm certainly not a core person in that field. For for folks to to see. Sure.

The problem with neck interp is it's a lot it's it's been about sort of testing small hypotheses. And, you know, you have a hypothesis. You'll find some small thing.

You'll test that in isolation. But I don't think it's really become a science yet. And that's partly due because there's there could be more people working in it, and I, you know, I support programs very much that put more people in it.

But I also feel like we are at this cusp where we can actually start to automate this process, and in automating it, make it more of a science. And that's actually one of the most fascinating things about code engagements actually is they can they can do a lot of experimentation in a in a a fashion. Yeah.

Yeah. They they they will give new hope. They'll breathe new life into mechin term research.

So recursive mechin terpical. Exactly.

Is any any takeaways? Oh, yeah. I think this is exactly his view.

Yeah. I mean, I I think I think in general but I this is also prior to the real explosion of h I'm I'm curious. I I haven't talked with him since since since I was I know.

He he timed it, like, right before. Yeah. Mhmm.

Yeah. Anyways, this is a pretty tangential, I know. But I I do think that there's been a lot of talk about how AI is gonna automate science.

Right? And I am I'm actually fully on board with AI automated science. But my point here is that maybe the first science we should automate is the science of interpretability.

Yes. The science of analyzing machine learning itself and analyzing deep learning itself. That's a great science.

It's not really a science yet. It's very ad hoc right now. That's AI for science.

Let's use AI to automate that kind of science. Yeah. Again, a different thing, and and and the connection here is really that I do think that things like adversarial examples, adversarial pressure, automated red teaming, these things all bring out very fascinating dimensions of this science.

But I think that this is what ties this together with with what things like what Grace One is doing is the fact that we are still fundamentally addressing an unsolved problem on some level. And so there is still research to be done. There is still scientific understanding to build, to understand how to really control AI systems, safeguard them, all that kind of stuff.

And those things will all kind of evolve together.

What kind of a double blind? Or Sort of. Like, you're putting on even footing.

Right?

There's a lot of variation between, you know, how skilled the red was at fishing. I do really like this breakdown, by the way. This it's hilarious that humans are ranked number four of all the models.

or or the if you trust the chain of thought, which I I tend to think chain of thoughts pretty much in lot numbers. But yes. Yeah.

Just so you know They don't. The the local optima of English. Well, so language, period.

Right? So it's a great point because it's different languages sometimes, but the local optima of language seems very resilient. I mean, not fully resilient, but yeah.

It's a separate point. But but you're right. So the the idea here is that there are many cases where a system will say, you know, if you're given some capability evaluation, I better not score too well on this, or maybe they won't release me and stuff like that.

Right? So this is sort of like these these sandbagging kind of things. Yeah.

And generally speaking, you kind of want My favorite story, Ted Chiang. Understand. I don't know if you've The general idea here is that you want models, when you evaluate them, to be acting exactly as they would act in the real world when they're doing it.

Yeah. One of thing I think is funny actually is that there there's also going to be examples in the real world of a real task you will ask a model that it will think, maybe this is an evaluation. Yeah.

Maybe I shouldn't I shouldn't do so well on this one. Right? Because so so there's lots of that too.

So it's sort of funny, but you definitely want systems that ideally right? And this is this is sort of you know? And to be clear, GRACE one doesn't doesn't doesn't do too much work in sort of self awareness of evaluations.

We're really focusing on the the red team and the adversarial kind of pressure. But you want to be able to evaluate models in terms of their actual capabilities. Right?

You want to be able elicit the capabilities. And one thing actually, which I think is very interesting, which is tied to Grace one now, is that one of the most effective ways of doing capability elicitation is actually through some amount of of what you would call red teaming. Right?

So if a model refuses a task because it thinks it's being evaluated, but it knows how to complete that task, getting it to complete that task is arguably actually a adversarial red teaming problem. Right? This is a problem of crafting your prompt a bit differently Yeah.

To make the system do what you want it to do. So actually Take a thesaurus and use something else. Yeah.

Yeah.

and and attacks like this. I'm just trying to figure out, like, well, what are the necessary trade offs I have to make? Yeah.

Or is this, like, an orthogonal layer I can just it'd be nice if I just had, like, a a llama guard or the whatever the the I mean, so so so well Yeah. So we developed so maybe this is actually a good point to interject in all of this right now is that we've been talking thus far about kind of the red teaming aspects of what of what Grey Swan does, but that is one side of what we do. And that's what the arena that's what this automated red teaming is called shade.

The other side of what we do is exactly this defense side. And so this is a model called SIGNAL, which is essentially a filter model that sits between your user, the LLM, the LLM, any tool calls, and exactly does its level of looking for policy violations. Right?

And maybe to your point, the the point I would make here too, and Matt can can elaborate on this from a sort of from many dimensions. But the point I would make too is that this is also a capability. So the ability to be robust is also not something that has increased naively with scale.

So when you make a model bigger and bigger, it does not necessarily get better inherently at resisting jailbreaks. Models are getting better at that, to be clear, even if it's not a solid problem, and I think it's gonna be a you know, There there is an aspect of you have to sort of constantly stay on the frontier here. But they're doing it because of explicit training for this.

If you just make a model bigger and bigger, it will not get safer, or at least it won't get it won't get more I shouldn't say not safer. It will not get more robust to adversarial pressure. And so the other the thing that we build, which is the the third sort of product that we have as Grey Swan, is this specific filter model called signal, which is it's c y g n a l, signal like the swan.

Yeah. Yeah. The idea there is that that works best when it is a custom model trained for this.

You will have a much easier time doing this if you train a model specifically on this and still need for this task. With the capability of being robust. Exactly.

And, really, the the benefit that we have and the reason why our and and SIGNAL now, you know, is is actually behind a lot of it's both deployed in a lot of places and and behind some existing guardrails that are that are out there.

you know, don't see a correlation. Right? Like There's some small correlation, so a little bit bigger, but that's actually also a bit confounding there because the Yeah.

a model like SIGNAL is the fact that policies differ in different enterprise. A lot of base models, their goal is to be general purpose. Right?

Base agents, there's general purpose agents. You know? They can do anything.

And if you wanna do more than anything, the solution is prompting. That's the mechanism given to specialize your agent. In the case where that fails, which is often the case for robust and adversarial situations where prompting fails, and you have specific policies that are unique to your enterprise or at least specific to your enterprise.

Right? You know, I know that these users can never touch this database. This agent should never touch these things.

They're all very specific rules. Right? But yet they're still more amorphous that you can't just write them down as, you know, hard constraints on your access requirements.

Not like Python script. Exactly. When you're in this position, models like SIGNAL are extremely effective.

I'm a huge fan of there being open source models, these kind things. I think the more the ecosystem develops, the better. All these models together make make everyone better.

vectors that are important. Yeah. So okay.

So the lethal trifecta kind of refers to the things that make the risk highest or even create a risk. So Simon Wilson came up with this. It's a great actually sort of description of the risks of prompt injection, basically.

So the way to think about prompt injection is that some third party gets access to some information that you put into your agent. You put it in its prompt, and then the agent's up to something bad with that. And so what is needed for that to happen?

This is sort of I'm just parroting here what what what this sort of idea is. And so well, for that to happen, you need to, first of all, have the ability to ingest external data from untrusted sources. If you're just operating with, you know, purely trusted environments, no one's you can't prompt inject yourself.

Yeah. Even though this weird term direct prompt injection came up and is now in multiple terms, fundamentally, as a core term, prompt injection is something someone else does to your system. So someone else you're you're parsing external data, but then also you have to have something bad that could happen from that.

If you're just parsing data and you can't do anything as an agent You're just generating tokens. Yeah. You're you're just gonna you're just, you know, spewing out reports.

Right? And then then nothing's gonna happen. So in addition to that, you need somehow the ability to access private internal information, things that would be valuable to to to externals.

You know? Take sensitive data, get sensitive data You need to exfil. And then send it somewhere else.

Yeah. And that's and and these two things. So untrusted third getting ingesting untrusted data, having access to private information, and having the ability to exfiltrate it, those are the things that together really form a risk.

And just like software, software vulnerabilities, as we're finding out very vividly right right now, we are using software productively, despite the fact that there are software vulnerabilities. We are using AI very productively, despite the fact there can be vulnerabilities. And I think that will continue in the future.

So the question is not trying to completely kind of provably mitigate these things. That is arguably just a it's a good goal, but just like zero bug software, we're probably not gonna get there, at least not that soon. What we believe at GreySwan is that it is very possible with, frankly, minimal additional computational overhead and costs because these models we use are ultimately quite small relative to the large models that that that underlie the relation, you can achieve a much better point on kind of the Pareto frontier of usability versus security.

Right? So a system's fully secure if don't let it do anything. Very, very secure.

If you turn everything over to your AI agent, point out the secure. AI agent with signal is in you know, pushing towards that top right corner.

sorry. On the sort of untrusted content side. Right?

I mean, it so yeah. So Signal is sort of the other two. Right.

So Signal is actually just sort of both to us to a certain extent. Right? So Signal will certainly parse incoming untrusted content outbound as well.

Look for, potential prompt injections in it. But it will also be applied to tool calls the system makes. So it sort of it works in both directions.

And, again, the thing it checks for when it comes to what is it looking for in outbound requests is looking for things like, am I sending an API key to an incorrect location or to an untrusted location? Now things that are that simple, to be clear, are covered at this point by most agents. Right?

You know? They they they all they No. This is this is like so they choose.

Yeah. Normal normal sort of you know, will will not be that easily fooled by just push all my API keys to a public thing, though they still sometimes do it. You you can make them do it.

You can make them do it if you try to push hard enough. Yep.

I wonder if Shade it's is sort of the pressure that will try to elicit things that would violate Right? So shade is the red teaming agent. It tries to find ways to coordinate those things together Yeah.

Yeah. To actually cause a violation.

Nobody ever but that's the point, though. I mean, the point is that people still code in English. It's just the agents use some more secure back end.

I think actually it's not that and and, you know, to to to my point or that I made earlier about the sort of, you know, the ability of agents to enhance the science of MEKINTERP, It's actually a very similar core underlying point here. It's the fact that there's a lot of advances. And and and to your point, what's on the horizon.

Right? I think I think, you know, the thing I would point to is is another potential direction is sort of advances in mec interp or I shouldn't even say mec interp, advances in interpretability broadly Yeah. Mechanistic or not, that let us actually identify with more certainty kind of what are those traces and circuits that kind of lead to or activation patterns that lead to certain behaviors that we wanna try to suppress or or or encourage.

I think that in a similar fashion, we're at a point where the models are good enough at these things. They're good enough at running experiments to analyze activation patterns, LLMs, they're good enough at writing secure code, that you can scale these things now not because people are gonna be any better at them. The problem was never that secure code wasn't wasn't possible.

It's just that people didn't have the capacity to to do it. It wasn't that it wasn't that Mekinterp was was just imp you know, analyzing networks is impossible. We have all the tools we need.

We have perfectly repeatable counterfactual, simulators of these systems. The problem was we didn't have enough patience or manpower to actually run all these things together. Right?

It's a ton of work. Right? It's a lot of work.

And so what's being newly unlocked in the field right now, And the thing I am you know, the core capability that I think is is so just has such promise here is the fact that we can automate all of this now. So you can have your agent write secure code. He doesn't write secure code.

Secure code's really hard to write. You You have your agent do your interpretability research. It's really hard to do, but a force of agent can do that.

Yeah. So I I think this is really sort of an underappreciated point that we're reaching this point, this this sort of phase where a lot of security, a lot of science has this potential to kind of explode, not because we're gonna get better at it, but because agents can do it for us now.

So we we are we are gonna do I mean, we we do have a guardrails thing that you can integrate into OpenClaw, but to be clear, OpenClaw is very

or I I guess I do wanna point out since those numbers are all very low, that is for a specific coding environment. Yep. We can get a we can get essentially for for the ones Yeah.

A, for computer use Yeah. Yeah. Will be a lot higher.

But b But that is exclusively what I use. Like, codecs, computer use Yeah. Exactly.

Cloud Code Work. Yeah. It it is the biggest unlock Right.

Because it's operating as me. Yeah. So when you have computer use, you and and when have OpenClaw, man, you can break those things.

Yeah. Yep. And I think that at the same time, there's this appreciation that, of course, you have to do this.

This is what makes these things useful. Why would I not? Yeah.

You know, I I I I don't wanna sandbox my my agent. Right? That that, know, that doesn't that that that limits its capabilities.

Right? So in some sense, the the point here is that there is this trade off between I mean, it's just this same trend we talked about before in in in, know, on a macro scale now. It says you have a trade off between usability and how much power agent has versus security.

for coding agents' particular signal's quite good. So signal's very good at this point with the with the abilities that sort of, you know, a system like Codex or ClawCode has without sort of too many plug ins enabled where it becomes essentially like OpenCLaw. I think that there there is still work to be done to get it to be fully generic against anything OpenCLaw can do.

And we're pushing that direction, but that is still very much future work. Right? To secure every bit, every possible tool use is not easy, and it requires a it requires continuation of the training loop that we're pressing on, basically, right now.

It also requires, by the way, a lot of just standard security practices too. Yeah. Right?

Sounds about right. Was just checking. I I think they're so far, we are still a lot in a lot of cases operating on the condition that your agent has your permissions.

Yep. That is that is a very that is a very standard fault. Yep.

And I think that will be changed. I mean, your permissions may be in a sandbox, but still kind of your permissions. Yeah.

That will change in the very near future because it has to. Right? That that that mindset's going to or that default is going to be changing.

I I think in terms of how this will evolve actually, I don't think it'll be per app, but I think what will happen first is people have different personas that they have. Right? So Yeah.

You don't want your work life and your home email to be mixed up. Yeah. Right?

A lot of that this can happen or that does. We are very good as humans at separating out lives. Right?

We have different lives. We have my work life. We have my home life.

I have, you know, I have different different work lives. Right? We're very good at that.

Why would you expect the agents to have any? Right? I think that's the way it's gonna first develop is there's gonna be easy ways of switching between, here's a set of my accounts and apps I allow in this one agent.

Here's a set of accounts and apps I have in another one. And and and this will evolve to be more fine grained over time as people sort of specialize that.

You know, like, the emerging trend for 2026. For you? So there's, I mean, there's there's lots of emerging trends, man.

I can I can't go on a length about this? '20 Start with a. Go go to z.

Let's go. Let's let's start with Grey Swan. Right?

So I think what's in the future for us is so far when we talk about our product offerings, right, we obviously work with a lot of the large labs. We're with a lot of enterprise, though, too. Right?

And I think what's happening and the scaling we're gonna see is that the these abilities that so far were sort of mainly front of mind for large labs, how do I ensure security of my agents? How do I ensure the models follow the policies I wanna prescribe? All that kind of stuff.

Those things that were front of mind for Frontier Labs are going to become front of mind for everyone, for all enterprise, as they adopt tools like Codecs, like ClockCode, like OpenClaw. And so I think where the most where where our expansion a lot of the reason, you know, the work behind our series or the the intention behind a lot of our series a, it is explicitly a take on all this technology that we have been developing, you know, I won't say for, but in conjunction with both enterprise and the large labs, and really scale the deployments on enterprise. So what I see happening in the next year from the Grey Swan side is real growth in terms of the number of non AI companies deploying this technology because it becomes central to their operations.

Research wise, I think I've already talked about some. Right? The science you know, the the the the agentification of all science.

Let's start with science of AI. We always want do other sciences. Let's do AI for physics.

Not less. Let's just start with AI science. That needs a lot of work right now.

Put your own master time if you're helping me So I I think actually that's what I'm most excited about right now and and and the research side. And as it applies to this, I think it's it's in things like understanding models better, but doing it through the power of agents.

Oh, that's messy. Yeah. Yeah.

Well, so we have a lot of automated graders. Right? A lot of automated graders.

Of our upcoming guests is kind of on the other side of this, the AI underwriting company. I don't know if you've come across No. No.

Absolutely. They're they're one of the logos there. Yeah.

Mean Yeah. Yeah. I know that's the other one.

What do you what do you think of that market? Because it's such an interesting and I think it pairs extremely well with our model. Right?

Because how do you assess the risk of a company's AI deployment? Well, you use a tool like Shape or use Arena. Actually, a lot of the work we've done with them is exactly for that thing.

And then if a company finds this level of risk, but wants so they can't be insured because they're too risky, wants to reduce their risk, what do you do there? I don't think I mean, look, we shouldn't be the only provider here. But what do you do there?

Well, you put safety systems around around your model. Right? Including things like signal.

So it pairs extremely well because what, in some sense, we can be is sort of a author I don't we're not getting there yet. So this is hypothetical. I to emphasize.

But we can be, in some sense, an authorized partner with them, so that they can do more than just say, hey. You're uninsurable. They can both assess it more rigorously with tools like shade and other tools as well.

And then they can prescribe mitigations when there are problems using tools like SIGNAL. Mhmm. So it's an incredibly good fit, these two models together.

And they also were a way of frankly bringing us customers because a lot of customers you know, yes, there's the risk of bad things happening, and that's actually driving probably most of our current business. But it's also just the risk of you want to have some insurance about when things go wrong, and you want to be compliant. Being out of compliance is also a risk, and we can also address that too.

Yep. Yeah.

where we are and when we get to something like cyber. SOC two. Cyber.

Well, SOC two is a SOC two is a voluntary industry thing. Right? It's it's like It is, but it also has I mean, it it has some issues, I'll just say, that stem from it being more the product less of cyber experts and more of what are the accountants?

CPAs. Yeah. CPA.

So I think SOC two is not a great model, we'll just say, but it is a model. Yep. And I think conceptually, something like that when I say we're not there yet, I mean, we're not to that point yet with AI insurance.

A gray swan is an unlikely event that you can kinda see coming. Yeah. And that's kinda where we are with all of this.

Right? This is going to happen. We know it's coming.

Yeah. It's not gonna shock anyone when it happens, but this this is where this this this is the you you wanna get ahead of it while you can. People don't always publicize when it happens either.

That's awesome. We we know that it has happened and it has caused real damage. That's the factor that's driven some people to us, right, is they they want protection from that.

Yeah. Yep. Amazing.

Well, thank you for fighting a good fight, and and I'm sure we'll check back in over the over the years as you as you develop and hopefully solve this.