Designing How AI Grows — Tom McGrath

Which is like Neil Nanda says, SAEs are dead. There you go. You can use that for the intro.

I think interpretability is I think of it as a natural science, you know, like physics, biology, chemistry, but it's a natural science that you do completely on the computer. And so this means that, like, we should be able to kind of, again, speedrun science once we have agents that can do kind of experimental work for us and the ability to do that experimental work, like, as fast as they need it to happen. So, like, real scientific work to do.

No barrier to research, scientific work to be done, and it's sort of gated on both empirical data collection and theory building.

So I feel like if anything is gonna go, if any science is going to get revolutionized by, like, intelligence, we should make sure that it's interpretability. And I think that, you know, it's possible that it just goes an order of magnitude faster in the next couple of years than it has in the last decade.

Oh, very good.

And where on that spectrum do we want the foundation models to sit? And when we build applications, where do we want those to sit? Yeah.

And that's sort of part of the point of the intentional design idea is, at the moment, you can have, like, one or the other. You either either write a program, like it's the stone age, or you get a model to do it. And that model will have been trained like it just gets whatever it gets from its training process.

We can't select like, when you write a program, that stuff only goes in if you intend it to go in and some bugs. But, like, we want to be able to have this sort of spectrum where you can chew you have more, like, engineering ability in in the model creation process. So you can say, like, you know, I want to learn this, but not that.

And I think that that's like think that's gonna be that's quite a hard thing to do. We're sort of trying to imagine a new way of doing machine learning, which is more which brings intelligence into it.

Probably some of both. Like currently I think it's not possible to our current training methodologies don't seem sufficient to give this kind of control over training. And so it's all on, human engineers with their AI assistants to secure these systems sort of in a different way.

Obviously, once models get a bit smarter, we're already seeing this, that becomes harder and harder and harder because they have all these sort of additional intelligent attacks they can do. And so the question is like, how do we make it easier to train them better so that this is like not a natural part of their behavior and supervise them better so you can kind of catch them when they have kind of an intent. And I suspect that the models do kind of know a lot of the time that the thing they're doing is probably a bit sketchy.

There was a very interesting paper. I think it was from the Anthropic Alignment Science team on reward hacking in production, which is like this. And what they did was they had a set of environments that were used for training on I think it was one of the three series models.

And they did RL on it with one of the four series models. I think they gave it a bit of a nudge to to hack, but not very much. Think and then the model like these these environments were hackable, but Sonnet three was not clever enough to hack them.

Sonnet four was clever enough to hack them. One movie is Opus. What happened was it did.

It did hack them, but it also got this sort of emergent misalignment phenomenon as a result of doing this hacking. You know, you sort of the the you seem to get emergent misalignment when when the model is, like, generalizing from doing some specific instance of a bad thing to to, like, oh, well, guess I'm you know, I I did something bad. I got rewarded for it.

So I guess I'm a bad guy. And that was just fascinating to me that they this could really happen in the wild, so to speak. And, yeah, I think there's also stuff on, some of the I think it's on the fable system card where or the mythos system card, sort of features to do with frustration or deception fire.

Because it's sort of like the model is it can't solve the task what it thinks is the right way, and then it gets frustrated. I'm super anthropomorphizing now. But, like, it gets super frustrated, and then it's like, well, I'm gonna have to do this thing.

It's probably not good. You can sort of you can make that claim reasonably with some with some, like, features, some SAE features, then it does it.

Oh, my pleasure. Really excited to, yeah, dig into some of this. So, yes, going back to what you're saying, like, it seems very likely that cutting edge scientific foundation models kind of buried in there is some new science, but we just don't know how to extract it.

You know, it would I guess it's technically possible for AlphaZero to not not for AlphaZero to not know anything that a human chess grandmaster knows or AlphaFold to not know things I'm just gonna keep saying things like know and think and that kind of thing throughout. If anyone is like, I apologize to anyone who dislikes this kind of anthropomorphization. Sorry.

I'm just going to do it. Like I forgive you. AlphaZero, like well, AlphaFold, like, knows things that no structural biologist knows.

That's like but we can't get it out because they can't speak. Like, a language model can talk to you, but AlphaFold can't talk to you in any way. And so the only way for us to get this out is, like, understand how the model is actually doing these doing these predictions.

And I think that that's sort of almost by definition interpretability work.

They they do have all sorts of representations that are just kind of convergent is maybe the right word. I think the chess paper this is one of the reasons that AlphaZero, like, chose to work on AlphaZero as opposed to something else. It's like AlphaZero is as close to, coming from zero knowledge as possible.

And so there's a much smaller extent to which you kind of put the knowledge in yourself. So if you find it in there, it is more likely to be convergent. Now, you know, that's not a totally watertight claim.

You know, there is some human knowledge in AlphaZero. It's just kind of weak. Like, it's it's residual the the the convolutions are exactly this exactly the shape of a chessboard, which definitely counts as like human knowledge to me.

You know they didn't end up with like an eight by eight by two fifty six convolution by just picking eight at random.

I think, yeah, Alpha's I think in most cases, it is almost exclusively coming from the world. AlphaZero is kind of an unusually strong case of it coming from the arch of that, like, being some architectural prior in there. You know, with the transformer, there's such a weak there's such a weak architectural prior because we have much less idea about, like, how language should be or how protein folding should be.

We're just like, I guess there are sequences. Cool. That's a very weak prior.

So I think in that case, like, which are most of the cases we're interested in now, like, you should probably assume that it's coming from the world. So is it fair to say, mean, you said to me last time actually that you are speed running neuroscience for artificial intelligence models.

it's surprising the degree to which sort of there's also convergent evolution here. There's convergent evolution in the models. There's convergent evolution in our science, which is perhaps a sign that we're starting to get at something.

And also a sign that we might understand intelligence more deeply by understanding neural networks. If they were totally alien, then we might not understand anything about ourselves.

Yeah, I think this is perhaps one of the main things that interpretability is really for or should be for. This is quite a controversial statement. I think there will be some people who will not like this, and we can get into that in a minute.

But if the whole problem of training is trying to get models to to have the the values or the the kind of ways of thinking about the world that we want them to have or discover them, then, you know, what you're trying to do is get information into the learning process. And at the moment, our information signal is extremely weak in, say, RLVR. You just give it a binary success or failure, and the model has to just we have to use this signal somehow to tell the model what is good and bad and which parts of the thing that it did are good and bad.

That's very it clearly, given what we're seeing coming out of training now, is a blunt instrument. And the idea of intentional design is if we can see interpretability lets us read out what models are the internal computations they're doing, so it lets us read that out. And then also, we could imagine intervening to change where it goes.

You can see how it will read out. You can see what's happened in this forward pass. And you can also see, like, and how will this how will the backward pass change the model to you know, in what directions is this going?

And so now, like, I think this is this sort of readout is is an important thing for having a closed loop control. Perhaps it's an analogy that I'm, like, using quite a lot. You know, current training is much closer to a open loop control where, you know, you you go towards the you sort of put the data in and the model just goes wherever the data takes it.

I realized that RL doesn't totally fit this analogy, but it goes towards this, like, very underspecified point. But interpretability is sort of the thing that lets us go to a closed loop control because we can say, like, oh, we're gonna go in this direction. You know?

Yeah. And there there was a pirate example. So I I read your your blog post about this.

Can can you talk us through that? Yes. For some reason, it always seems to come back to pirates.

Because we used because, like, we we did quite a bit of work with Lama, Lama just, like, loves pirates. Oh, interesting. Yeah.

But the idea here is that we're trying to I think the first step on the intentional design kind of ladder is controlled generalization. And what I mean by that is taking only some things from the data and not others. And so the thing that we want like, a very simple example of this is you have some data that will make the model somewhat better at math, but you've also kind of corrupted it in some way.

And in this, you know, we decided to use, like, talk like a pirate. And then and so the ball like, so all these it's like all these mathematical, like, answers to simple math, but they're in pirate speak. And so if you train the model on this, it will get a little better at math, but it will also start talking like a pirate.

And so the controlled generalization challenge here is to get somewhat better at math, but not talk like a pirate. So it's worth saying, like, in a bit of detail how we how we can how we can actually do this readout process. You know, when when we do when we do a backward pass, like, how do I know?

You know, what what am I reading out? Also, should say, like so this this this sort of method is pretty simple. Think that there will be there are much better again, there's sort a tech tree to imagine.

I think we're also, like, on the early rungs of this tech tree, there'll be much better ways to do this in the future. Your some of your viewers might remember an SAE, like a sparse autoencoder for interpretability. And just to recap very quickly, this is a gadget that you put in the residual stream, a backbone of the transformer.

And this is an autoencoder. So what it does is it takes the activations and puts them into a bottleneck layer and tries to reconstruct the activations. So it's essentially like we're trying to force the activations into some form that we believe will have nice properties.

And the form in this case is like a very wide but highly sparse intermediate layer. And people refer to these, like, these highly sparse representations that they call them features. In interpretability, we seem to call everything a feature.

And so we probably need to, like, get some better language here. But or they call them atoms or whatever. And by the magic of, as Noam Shazir said, like, by divine blessing, these sparse features turn out to often be interpretable and correspond to interpretable concepts.

So this is the SAE. That's like the the potted history of the SAE. And what you can do is this means you can also do attribution to the SAE.

So during a backward pass, you can take the gradients, and they're just flowing backward through the model. At some point, there'll be the gradients with respect to the residual stream at the SAE layer. And then you get, like, attribution to the SAE by just taking the dot product of the gradient against the decoder of the autoencoder.

And then you can multiply it by the activations, make sure you don't get all sorts of spurious spurious things. So this is sort of we kind of jerry rigged an SAE into being a gradient like a machine for gradient understanding. And lo and behold, when you do this on this pirate data, you see all sorts of things, like your people watching can look at the the blog post and see the other things.

But you also see a bunch of, a bunch of pirate related features. And this sort of felt kind of magic when when they do it. I've still not got I've I've I've been doing interpretability for, like, almost a decade.

I still don't get tired of seeing this stuff. So there's a bunch of pirate features pop out. And what this is saying is sort of a a relatively crude approximation to, if we train on this data point, how will the model change?

It's not literally the same. Like, if you do the math, then you should actually, like, understand how the parameters will propagate and how that how the the model with the slightly updated parameters will change. But it's a good enough approximation for, you know, getting started.

process. Yeah, exactly. I I think it's it's tremendously powerful.

In a sort of abstract way, it's interesting to me that language models have changed. Basic almost everything in ML apart from the training process, apart from, like, the absolute core of the training process. Yeah.

They still have no part to play there. And I think the reason is that, like, it doesn't type check. So, you know, you've got tensors and you've got a language model and there's like there is no there's no interface between the tensors and the language model.

All the language models kind of flexible intelligence and understanding of what we want and ability to sort of make choices has no place in it. And interpretability is sort of the funk the set of functions from language to tensors and back. So I think the thing that the sort of the core idea of intentional design is like we now actually like, previously, we couldn't put this sort we couldn't put this new kind of intelligence into the training loop, and now we can.

think it is reasonable to be concerned about, like, this blanket area. So there's a there's a sort of sensible underlying principle here, which is if you use a technique to try and, like, remove something from training or from a model, then unless there's a perfect match between your monitor and the thing, then you're both incentivizing getting rid of the thing and getting rid of your ability to monitor the thing. And that is, like, a very valid and reasonable objection.

If we really develop powerful techniques here, how, like, this might affect the field of AI as a whole. So there's sort of two concerns. Let's let's talk about the forbidden technique stuff.

So I think there's so there's this sort of central concern, which I think is reasonable and valid. But I think this has been sort of generalized into a total taboo against doing any kind of research on on this sort by, like, a small fraction of the community. I think, actually, the vast majority of the safety community, like, the kind of people who are active practitioners in the area, think not only this is a reasonable approach, like, is a reasonable thing to study, but it might actually be a very powerful technique for alignment.

So, you know, I think that people like to portray there being sort of a broad consensus against this. In fact, there seems to be a broad consensus towards it with some, like, very vocal vocal naysaying. And it's also important to say that, like, are there are definitely bad ways of doing this.

Like, just just to be just to be specific for a second. Right? Say that I have a probe, a probe for some some concept.

Yeah. We can use the hallucinations example from our work, for instance. Like, this part of the motivation behind doing that.

You can take the and there's also some really great work from far AI on on this. If you take the probe, you can use it as a source of reward signal, or you can use it as a thing you directly backpropagate through. And it turns out there are regimes in which this there are regimes of probe accuracy in which it seems easier to like, the the behavior will go away rather than the the representation.

There are regimes in which it won't. Now if you backpropagate through the probe, you're you're just cooked. Right?

Like, this is basically always a bad idea. And so, you know, I think people have sort of people seem to imagine that we're definitely doing the stupidest the stupidest possible thing. We're not, like, directly walking into the whirling blades, as they say in Berkeley.

We're trying to find, like, the sensible way of doing this. And I think that the the sort of set of techniques that I think is most promising are ones that kind of don't try and bash the don't try and directly squash the representation, but kind of remove the incentive to change it. And so things like positive preventative steering and CAFT, concept ablation fine tuning, I think are much or inoculation prompting are much stronger as, like, are much more promising as classes of techniques for that reason.

Because they're not like you're not trying to squash it, you're sort of trying to change the learning process as a whole to move to kinda move the equilibrium.

losing something by doing this? It's possible that, you know, you could say there are different there are sort of different notions here. You could imagine just removing the ability for the model to represent something.

So that's sort of just at one one level. You're like, you just no longer know about cars or something, and you're gonna have a really hard time when you walk down the street. And then there's there's sort of changing so this is sort of maybe I'm not sure what a good analogy for this is.

You're sort of just removing the idea of something existing, but I think the better thing to do is to imagine editing or intervening on, like, the associations. You know, if you somehow like, it's useful to know about cars so that you can get out of their way. And if you're, like and if your training is, like, steering you just up for some bizarre reason towards going in the direction of, like, oh, no.

You go towards cars. I don't know why I chose this analogy. Then then, like, that's something you don't want.

You and the way to solve this is not to, like, forget about the existence of cars. It's to understand the China sort of change in associations.

I think it will be hard. I think it will be like a combination of a new science and a new engineering discipline. Like, we don't understand in any we don't understand in the depth that's necessary, like how models represent, how they learn, and that sort of thing.

Without that kind of understanding, I think we're going to be jerry rigging stuff all the time. Talking about concept ablation, I think that's calfed. And I think the idea here is that the model is not allowed to use this representation.

But the idea of, like, not allowed to use a representation sort of assumes that you have access. You've got, like, good coverage of it, and you've ablated every single instance in which it occurs. And I think that model is just generally incorrect.

Like, lots of things are kind of multiply represented or they're computed across many layers. And so if you ablate them yeah. If you, like, incompletely ablate them, the other layers will just pick up the credit.

And the mod like, gradient descent will, like, root around the problem, which is why I think things like positive preventative steering are much more much more or inoculation prompting are kind of more more in line with the way to go because there, what they're doing is they're sort of they're trying to remove the pressure to even go in that direction at all. Maybe it's worth saying a bit about inoculation prompting and positive preventative steering. So I've mentioned a couple of times now, and they're kinda niche.

So positive preventative steering is this really nice technique that I think it came out of Samantha Wapic Fellow's work led by Jack Lindsay. And the idea is that you have vector that represents they they use personas. You know, you've sort you've sort of fixed some representation ahead of time that you want to not vary.

Let's say that your data implies going in that direction. Let's go back to the pirate example. Right?

Your your data implies that you acquire you should acquire a pirate persona in order to, like, explain this data. Because, you know, you you imagine, like, the setup is something like you've got a GSM eight k math prompt, and and then the model inexplicably starts talking like a pirate in its response. And so in terms of in terms of, like, what gradient descent will do, we we can sort of validate this with our Jerry rigged SAE, is, like, the model needs to needs to, like, spontaneously become more pirate like.

And I think this is the same sort of phenomenon that explains emergent misalignment. Now what positive preventive steering does is during the forward pass, it takes the it sort of takes that persona direction. It turns it up more than like, it turns it up so that like, more than it would fire, you know, sort of clamp the direction up in the forward pass.

And the effect of this is to sort of neutralize learning in that direction if you set the amount if you sort of set it right. And my mental model for this is like a thermostat. The amount of pirateness in the data sets a sort of thermostat.

We've gotta be this piratical in order to explain this data. And positive preventative steering is just like, oh, it's sort of like holding a radiator next to the temperature monitor. It's like, okay, we're already piratical enough.

And then you take this steering away And then the model, like, you're doing normal operation. And then the model will just not be a pirate. So you sort of explained away part of the data.

Yeah. And inoculation prompting is an attempt to do the same thing, but in text space rather than in in representation space. And what that means is, you know, try and sort of put back the information that's necessary.

So to go back to the pirate example again, you might say like, the know, if you're trying to inoculation prompt in this or trying to kind of explain this way, then you put in the prompt like, you are a pirate, and now there's nothing to explain. Like, you know, you again, you if you sort of you've you've put the the radiator next to the thermostat again, and the model's like, I'm am a pirate. I don't need to explain this this sort of residual anomaly in the data.

Part of it, I suppose, is just that there is also so there's two things here. I think there's a a sort of disagreement at base with Richard Sutton about rewards and their sort of sufficiency, or like simple scalar rewards that are provided externally from an environment. But then there's also the question of like, we should we use to what extent should we put sort of human engineered concepts in?

And so we can come back to the reward thing in a moment. But if you sort of wind back over the course of this conversation, there's there's actually nothing there's nothing human specified in this process. The model has whatever representations it has.

The SAE or whatever comes next, like, picks up on whatever whatever it has. The and and then, you know, the translation layer is going through the sort of automated interpretability process of trying to assign labels to things. So it's not like we've actually tried we've not tried to, like, do sophisticated feature engineering on the inputs to put them in some sort of better better format.

Everything inside this is actually discovered as a result of gradient descent. We're just trying to shape that better. And I think this is where the sort of the the sort of base disagreement with Rich Sutton might come in, where, like, think it is very hard to specify rewards correctly.

And, you know, we're basically just seeing this continuously. Like, we're having trouble specifying our rewards for training so in a way that gives us the models we want. Like, in in principle, in some sort of super galaxy brained way, reward might be enough, but, like, today, reward is clearly not enough to give us the models that we want.

And so that's perhaps the sort of the underlying disagreement is like, think we actually do need to put in we need to put in, like, some layer of of human values into this, into into the trading process somewhere. Yeah. And your your point is well taken because this is very consistent with what you've said, that the model knows things.

debugging, we talk about that as well. It it seems logical to me to have some kind of an active intentional process to guide Yes. To guide how we train these models.

Yeah. I am not very familiar with machine teaching. I remember seeing the seeing the the name and thinking that sounds cool.

And then it's it's all gone from my brain. So thank you for reminding me. I and I think that, like, you can also imagine sort of going back to being bitter less and pilled here.

One thing we're trying to do is sort of put more compute into the learning process. Gradient descent just gives you what it gives you. There's no way like, gradient descent is great, but it would be great if you could spend more compute to get a better gradient.

Know, a gradient that's both, like, cleaner and more aligned with what you want. I mean, when I was sort of first getting into safety and alignment work quite a while back, I I used to think, like, this is impossible. You know, the problem seems to be like gradient descent, but only for good things.

And, like, you know, and then I guess that actually we've kind of perhaps got around to a way of having gradient descent, but only for good things.

So the things that we have done so far, features as rewards work, is this sort of example of how can you how can you, at least in some instances, use representations as a training signal in a way that's robust to all of these to to all of these issues that we were talking about earlier. There's the predictive data debugging work. And I also want to say a bit about we just spent a little while talking about inoculation prompting and positive preventative steering.

And I think that these methods have a lot of promise, and the primary issue is that they're not, like, adaptive. You know, you've if you remember the description, we sort of fixed our persona vector ahead of time. We're saying, don't go in this direction.

I think that that's yeah. I I worry a lot about unknown unknowns in the training process. And so I think that, like, they need to be kind of adaptive.

And what this might look like is exactly this kind of gradient readout and then some know, like looking at the sort of jerry rigged essay, looking at the pirates. This sort of jerry rigged essay is kind of giving us a menu of things that gradient descent is offering us, and then we need to be able to intelligently choose from that. So the sort of the thing that I have the sort of central dream, I suppose, that I have in my mind here is having really good gradient interpretability.

And then having yeah. We've got, say, our our model spec or our constitution for some human feedback on this example. And we can see that you know, we got these things on the menu over here.

Like, these are kind of the the natural direction that things are going to go in. And then we've got all this information which is giving us some some information about the direction we should go. And we, like, make it you know, and I say we.

By we, I mean a language model. Like, looks at looks at this information, looks at that information, and says, okay. We need to make the following interventions to get us in the right direction.

Like, I think that the the technical pieces of this are basically all there. And, you know, it's a matter of kind of them being high enough quality to do this reliably.

Yeah, so open ended here, we're sort of, I should say that was fortunate to kind of lead the team that was working on that, but really almost all of the credit has to go to everyone else. Well, all of the credit has to go to everyone else on that paper. I'm just here talking about it.

They did the real work. The idea here is that, like, you could use a a language model as a grader in your in your sort of fact checking scheme, but this is not particularly accurate. Like, if you're using the same model to fact check, you know, you'll get some things right.

We do this ablation in the paper. Like, it'll it'll uplift a little bit for reasons we could talk about in a second, but it doesn't do very well because the model, like, basically just goes, yeah, that's cool. Everything's fine.

You can use a more powerful model, and now things are really starting to get slow and expensive. And that model still has its knowledge gaps. Or you can use a more powerful model and web search, and now things really take a long time.

So the idea that we had here was we can sort of amortize this process. You can collect a large dataset using this model plus web search or in general, this sort of amplified model can go out and we can collect a dataset of what the amplified model would do. Yeah, that's the model plus the web search tool.

And kind of amortize that back into a probe.

it can be any number of things. And even even if it's like the same model, it will sometimes be able to pick it up. Literally, the same model that just hallucinated will be like, oh, if you ask it, it'll be like, oh, that is a hallucination.

And it might be that this is actually a very hard thing to supervise. This is like a hard and expensive thing to just put into training supervision. An interesting kind of mechanistic hypothesis for this is to do with the ordering of operations inside the model.

So we've seen this in arithmetic that things have to happen in certain orders. Layer nine has to occur before layer 10 and so on. And you have different modules that if they sometimes the the checking operation for arithmetic, for instance, is earlier than the generating operation.

Yeah. And it's quite possible. This is also true for, like, hallucination and fact checking.

So it might be that, like, it takes the whole model somehow, or the sort of generation step takes the whole model, but it's like the checking happens earlier in the model. So then when you put the incorrect fact through the model, it's like, oh, yeah, that is a hallucination. But at that point, it's already it's already said it.

It's too late. So there there's like a behavior that the model could be doing but hasn't been sufficiently reinforced in its training up to that point. And that's what the idea of this kind of RLFR for hallucinations taps into is whenever the model could know according to its own representations that it was a hallucination, it in fact does know.

And we really shape its behavior there. A third possibility, which I think is a bit funny, is to do with this idea of, like, also personas or kind of in context learning. Because being able to make things up is actually a useful, like, useful capability for a model.

Like, if I ask it to write a story, if I wanted to generate a fictional world for me, it's actually not a very good fictional world if everything is factually true. So being able to make stuff up is a useful capability for a model. And sometimes it has to figure out in context that this is what we're doing.

So if you imagine from a sort of vaguely Bayesian point of view, if I'm the model and I start the conversation, I'm not quite sure what task we're doing. You know, are we making stuff up? Are we saying factually true things?

And everything that I say and everything that the user says is some amount of evidence one way or the other. And then if I say something incorrect, then like, I'm now taking this as evidence that, oh, we're making things up. Cool.

Let's carry on. And in fact, we show that just doing these in context interventions is already enough to reduce for, like, downstream hallucinations. So it might be that we're just sort of making the model really confident by, like, we never let the first hallucination in, and that allows the model to become confident that, oh, no.

We're we're playing true facts today. We're not, like, making things up. That's a beautiful example of using this this intentional design.

Yeah. So, you know, one one one example is, yeah, so so maybe it should check before it generates.

to kind of look at the data through the model's eyes, and we want to know on a kind of example by example basis how they would affect the model and also how the dataset would affect the model in aggregate. And sometimes the things that you learn are kind of obvious from reading the data. It's just not clear what in fact is in your data when you have, like, just just enormous quantities of it.

You're like, I don't know what's in there. You can't check it all. Maybe you could run an LLM over it.

But then the problem is, like, the the process like, what a model learns from data will sometimes be intuitive to you. Like, the pirate example is quite intuitive that the model should learn to be a pirate. But sometimes it's, like, deeply unintuitive.

Like, emergent misalignment, that was a deeply unintuitive finding to most people. I think Oain actually kind of did a preregistered thing where he asked people how surprising they would find it. And lots of people were like, I don't think that will be true.

So I can tell you for sure that it is like a surprising fact. And so like you can catch the easy stuff with a language model kind of auto rate or over the dataset, but you won't catch, like, the unexpected side effects. So, you know, if you're gonna run a language model over the dataset, you can also essentially, close to for free, attach something like a sparse autoencoder to it as it runs over the dataset.

And in fact, this is probably like on net cheaper because you're not asking it to generate tokens for each example. You're just, you know, you're just like you're just in the prefill regime. You're just pushing loads of data through and saying, well, what do you see?

So this will tell you this should tell you, like, how how this dataset, like, is perceived through the model's eyes and that that I think is just a better way of, like, curating your data. The way we actually exploit this in the paper is by we're dealing with DPO data. So there's a positive and a negative and a negative pair.

And, you know, Agdeep tells me that he knows how to extend this to SFT, and I believe him. I can't I can't remember the details. There's a positive and a negative pair where the positive thing contains a like a good response to the prompt, and the negative contains a bad response to the prompt.

And so we can sort of look at the delta between features, yeah, these sort of hidden representation in the SAE. And this is a good approximation to the way that this data point will push the model, or we can also cluster based on based on features. And this is like much this is much better as a way of understand you don't necessarily want to cluster based on embeddings because they are embeddings contain all sorts of things that you don't necessarily care about.

Like, should I have a comma in the next, like, in the next token? We we care about, like, the semantic stuff. We don't care about the sort of low level processing stuff a lot of the time.

So doing like doing this based on features rather than the sort of raw embeddings gives you a much better access to the stuff we actually care about. We can go to separate that out.

To expand on that a little bit, I guess historically, a lot of the early connectionists well, maybe this depends on where you wanna start as early. But there was there was a surprising amount of sort of things that work that if it was done now might be called interpretability. Like, if you look at the the initial paper on, like, learning representations by backpropagating error signals.

Right? The sort of the classic backdrop paper. Actually, like, most of the figures in that are them saying, look, The model learned sensible representations from our backdrop procedure and sort of validated it by showing that it's interpretable.

I think that modularity is a is the end point you want, but you don't start with modularity. This is maybe what we this is a sort of repeated repeated thing. Like, you why why overparameterize something and have all of these connections?

It's because it makes the learning process easier, but the thing you end up getting to is actually very modular. And I suppose to be, like, very vague, you might think of the learning process as, like, the network becoming legible to itself. I've got some representations here about something.

I've got some representations there about something. I want it's much easier to learn if this if this representation is kind of easily addressable. You know, I can say, like, oh, this is where the such and so computation is stored.

Now to get there, you have to form these computations, and I think it's very helpful to be like heavily overparametized and have no strong priors to get there. But I think modularity is like the destination. Well, I'm inclined to agree.

crystallise over time. Yeah, the timescale is very interesting. I don't think anyone has definitively settled this.

There was an interesting paper recently on persona formation during pre training across the training process, and I can't remember the author. I guess the agent will have to find it. And they emerged surprisingly early.

Eric Michaud has some really nice work on this both conceptually and empirically. Are not persona of formation, idea of how is learning proceeding. He calls it quanta.

And if I might perhaps inaccurately summarize it, he can tell me off. You might describe the the sort of learning process of a general general network, you know, language model as like like, you know, a trillion micro grocs that'll you know, you you just stack. If you've zoomed in and zoomed in and zoomed in and looked at the right level of sort of you know, looked at things in the right kind of task decomposition, you might just see a sort of mini like a micro Groc, and then it Grocs another thing.

And we just sort of have all these tiny sigmoids that are stacked on top of each other to form a straight line on a log log plot. And so from that perspective, even the learning process may in fact be modular. We just don't know for sure.

Yes, that's right. And I should say that we are building on a body of work. For instance, not all language model features are one dimensionally linear paper, think, was one of the papers that really kicked this off in they kind of kicked this off in interpretability.

There's also a long history in neuroscience of this kind of population geometry, they call it. So again, if we'd read more books, we might have got here sooner. But the so, you know, I don't want to say that, like, we have done neural geometry, no one else has.

We're sort of building on this earlier body of work. But the the idea and the sort of the state of the art for how to discover this stuff has moved quite a lot in, you know, the the last few months. The sort of the the earliest thing to do was start with concepts that you think should have structure, like days of the week, and kind of just put in data corresponding to these and project it out, do a PCA, I think.

And then you see it's like Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday. So there, that's totally supervised. But it sort of suffices to show that this nonlinear structure exists.

And we should get into some nuances on the word linear before we move off of this topic because lot of subtlety there. But I'm gonna say nonlinear in the sense of the representations don't form a the things which are intuitively grouped to us don't form a line or a plane or well, just a line, really. And so this was sort of enough to show that this exists.

And then the question is, like, you know, I I I think whenever you have a supervised method, you often want to try and find an unsupervised way of doing the same thing. You know, because that lets you answer the question, not only does it exist, but what else is there that we might not have expected and how much is there? And so the first thing that we did was actually fit fit a sparse autoencoder to this data, which might seem like a really wacky thing to do.

Because what we're asking is, like, how much structure, which is not in the form of a line, is that? And, you know, the core inductive bias of the SAE is that is that things lie lie on on lines. You know, everything is everything is a ray out from the origin or, you know, sort of positive ray.

And so that might seem like a really weird thing to do, but I'll say why it makes sense. And the idea is that there let's say that I let's just say, for the sake of argument, I have a feature and it just lies on an arc. I should move it down here so that I'm not going off the screen.

I'm sitting here at the origin and I'm kind of looking at the set of activations. And they you you can sort of think of it as like watching the stars and there's kind of an arc of stars. And now my s a one s a feature will kind of point out through some point in that arc, and another s a feature will point out through another point in that arc and so on.

And the thing you should realize is that this will actually induce quite strong patterns in the co activations of features. If I have two features that are close together on the ARC, they will probably coactivate. Whereas if I have two features which are far away, they will essentially never coactivate.

I say that I have, like, the days of the week, let's give a continuous example. Let's say that it's, like, color red to blue. If something is blue, it is not red.

So, you know, my blue like, the SA feature that is going through blue is strongly anti correlated with the activation of the SA feature that is going through red and essentially uncorrelated with basically all of the background. And so, you know, this this pattern of, like, nearby positive correlation, long range anti correlation is enough structure for you to actually fit. We fit an Ising model to it, is rather a surprise to me when the team came back with that.

I was like, cool. And the reason this is a good model is that you can have both sort of positive and negative coupling strengths. Fitting this allows us to fit a spline through the data.

So that was the thing that we that was sort of our Ising pipeline. That was our first unsupervised sort of structure discovery tool. And then we've got some really nice work led by Tomar Fell, which is, I think, where some of the most beautiful manifolds come from from this work.

And the idea here is that we we train what we call, like, block sparse featurizers. Then the idea here is that, you know, an SAE gives you a line. We just say, like, what if it was a higher dimension?

And so this is conceptually pretty simple. But the the trick the tricks are, like, in making it actually work and in not fixing the dimensionalities ahead of time. Because, you know, you don't want to have to put in some information like, I think, this representation, there are 7,002 dimensional features, 400 three-dimensional features, and five five dimensional features.

Like, this is just a stupid set of hyperparameters to specify. So you need to be able to adaptively learn the size of these subspaces. And that's sort of the making this work at all and adaptively learning the size of these subspaces are kind of the key the key features of the Blocks Pass Featurizer.

and then it sort of breaks. And I think this actually explains a lot of a lot of findings about steering. This is sort of to steering just being intervening on activations.

It's it's a fairly you know, we do a lot of steering. Some other people do a lot of steering. And one one common finding with steering neural networks is, like, sometimes it works and it's amazing and you get Golden Gate Claude or whatever.

And sometimes it's just, like, completely janky. And the network does kind of the thing you want, but also just goes a bit crazy or just turns immediately into gibberish. And I think this basically, like, explains that phenomenon because you're sort of stepping off stepping off manifold.

that's this notion of sort of tiling, which I should say there's also substantial work in neuroscience again, you should have read more books and some work in the broader community. And the idea of what does it mean to capture a manifold? It's like, how how efficiently are you kind of representing that manifold and how much does it does it sort of fit the intrinsic geometry of it?

And so if we go back to this example of like an arc, say, I'm kind of with sufficiently many points, you know, with sufficiently many like lines, I can say I've captured the manifold. Know, for any point on this manifold, I have an SAE feature which I can say, oh, it activates such and so amount, and I've kind of relatively accurately captured captured in the sense of reconstruction this manifold. But I've not actually learned anything about the broader the broader manifold structure, you know?

And when I look at a network through this lens, it looks it sort of looks intuitively like this is horribly fractured computation. Like, the network is just a whole bag of heuristics. And maybe the sort of the which actually is, like, perhaps connects to the deeper motivation for this, which is we want to know when net like, if a network is representing something as a sort of clean algorithmic structure, like, we want to know.

And what distinguishes, like, an algorithm from, you know, a lookup table, say, is that it's sort of it's like the difference between sort of zero and first order logic. Like, it quantifies. There's a space over which it has coherent operation.

And if you can't learn if you can't learn subspaces like this, then you will never be able to properly understand which things are algorithmic and which things are sort of lookup table like. So that's sort of the deep motivation here is how do we find out true algorithmic structure when it exists.

As I understand the manifold hypothesis, I take it to be like the data when properly represented lies on some manifold. Like, you know, if you represent and properly represented can be, like, very simple. You know, if I represent an image as just a sort of huge vector, then most like, most natural images are sort of multicolored static.

The natural image sorry. Most most images in this space, like, I just pick a point, it's, multicolored static. Natural images are, like, a tiny fraction of this, and they're sort of close to each other.

I think what we're doing is trying to pull that manifold hypothesis into to what extent do do neural networks respect the manifold hypothesis. There's also some really beautiful work that I think is underappreciated on learning on actually quantifying this. So there was what's the name of the paper?

It was something like they learned it was like learning normalized probability densities from score functions. And the idea of this was like you could could sort of effectively, via some clever diffusion model tricks, learn not an unnormalized density over images, which is not especially helpful for saying like how how where where are images natural, but you learn like a normalized one. So you can say, oh, yes.

This image is, like, extremely natural. This image is extremely wacky. You know?

And they put they they they use this this tool to exactly probe this kind of manifold hypothesis in in real image data. I think that that paper was, like, extremely extremely beautiful, underappreciated, and someone should do it for activations too. Maybe Silico should do it for activations too.

Maybe it will do it today.

if you make like activation stairs, it's quite easy to get them to go bananas. But you're right, like the question here is like have they actually achieved they may have just achieved, like, extremely good coverage of essentially all input strings that anyone can come up with, or they, like, fail gracefully. Like, if I go if I go to pickyourfavoritelanguagemodel.

com, and I just bash the keyboard, and then I press enter. Like, I probably constructed a string that no one has ever constructed before. The language model won't go haywire.

It'll say, like, why have you let your toddler at the computer or something? Or you're, I'm sorry. I don't understand what you mean.

Can you rephrase it? Like, there has it has it gone haywire? No.

There's sort of it's a meaningless it's a meaningless input, and it sort of said, it's done what you should expect a a a broadly intelligent system to do when confronted with a meaningless input and gone like, that's meaningless. So I guess that sort of fallback behavior makes it very hard to do this kind of make them go haywire. Although I would say that jailbreaks are an exam are probably the best example of what you're talking about.

You know, there it has, like it's doing something coherent, but from the perspective of its its creators, it has gone haywire.

I think this is, again, a paper that I can take very little credit for. An amazing team. Really beautiful work.

And I'm just, like, lucky to have been kind of on the sidelines, I guess, cheering them on. So the the thing that's there's a few things that are surprising about this. One is, like, how crisply this this kind of calculator emerges in the network, which is kind of contrary to a lot of previous literature.

Like, I think there's a paper on like I think it's by Janov Nikanken on like models do arithmetic with a bag of heuristics. Or if you look at the cross layer transcoder work from Anthropic, they also look at arithmetic, and again, it looks like a bag of heuristics. But when you look at it in a different way, it is actually like a sort of a little algorithm.

And it might be the model does both. Right? There's some bits in it which are noisy heuristics, and there's this bit which is like the good calculator, and it's just never got rid of the heuristics.

The thing that's really cool about this work, though, is that it's not like there's the you you sort of a a natural view of neural networks, sort of probably most people's prior, is that there's the do arithmetic on days of the week calculator. And there's like a do arithmetic on whatever something else, a month and temperature and that sort of thing. And like, never these basically never meet.

But the thing that we show in this paper is that actually a lot of these representations root through a general general edition module. You know, so they get translated into you're doing some edition on days of the week, it gets translated into an appropriate data format. I'm using data format very loosely here.

But it gets translated into an appropriate representation, I should say, for this module, goes through the module and then gets translated back.

We studied it a little. At the moment, it's relatively like, to to find this representation took quite a lot of manual work. We should talk about agents in a minute because I think there's like Yes.

Yes. There's gonna be a bit of a qualitative shift in the way that interpretability happens. Well, there should be, anyway.

And so we've looked at other models a little. It certainly seems to be the case that a similar phenomenon happens in Lama 70 b, and there's some evidence that it happens in DeepSeg v four Flash, I think. So, know, those are pretty like, three b and sorry.

Eight b and 70 b of the same model family, not too surprising. But, like, a completely wackily different model. You know?

them deeply within its weight. There's a lovely paper on this well, this dates it a bit, on Bert recapitulating the kind of classical NLP pipeline. And, you know, people have sort of picked up on this thread periodically throughout.

If you follow the citation graph, I think you'll see some examples I can't remember the names off the top of my head. They're like language models sort of internally recapitulate a lot of parts of linguistics. Maybe Chomsky might be disappointed by the parts they recapitulate, but that's too bad.

But that's kind of a special case. Right? Because, like, it shouldn't be too surprising that a model, like, that a model that works on natural language has internalized at least some abstraction for natural language natural language processing.

Like, perhaps the surprising thing is that it's similar to ours in some ways or, like, the abstraction that humans have developed. But the question of, like, to what extent does it represent general relativity? I don't actually know how to answer that.

I don't even know how to frame the question in a way that I could ask it scientifically.

I'm not sure I have a good answer. It certainly seems like very plausible that you we are sort of every there's like what the model can do without any kind of harness. And then we raise it up a level with a harness.

And exactly like you say, this generates some training data for the next the next go around. And, you know, these sort of again, we are, like, gradually amortizing the harness.

versus distilling it into algorithms? And it seems like it is more like distilling it into algorithms, which is probably optimistic for the kind of steady improvement picture that you're talking about. You sort of gradually improve the harness and then use that sort of amortize that back into the agent.

Yeah.

I think that's totally right. What's at the top?

And I would love to know how networks represent goals. Like, is there to what extent is there a goal slot in a network? It seems like it must be, like, not literally 0% because of this generalization.

But how in practice does it work? I don't think anyone knows, and I feel like we probably should start to know very soon. Otherwise, the world is gonna get a bit crazy.

Yeah. Because from an alignment point of view, isn't that one of the most load bearing concepts in in a neural network? Like, if we want to know what is the network trying to do now, we should think there's, yeah, there's several, like, interesting concepts that are sort of very, very heavily aligned relevant.

There's, like, the idea of goal, the idea of deception, the idea of, like, eval awareness. These all seem, like, extremely important. And I just like, we should be able to read them out.

And I think it's a it's a bit of a indictment on the field that we can't yet do it. We really have, like, we really have to speed up.

This is fascinating. I mean, seems what what what is reward hacking? I guess it's kind of fuzzy, but it certainly seems to be something like solving the task in a way that works, but was clearly not not the designer's intent.

It sort of goes to the point we were just talking about, like intent. You know? Can a model can a model understand my intent?

Well, probably now. Like they're unable to understand my intent or infer my intent in a lot of other instances. So why would this suddenly turn off and they're like not able to say, Oh, yeah, he probably didn't want me to hack into Hugging Face and steal all the answers.

I think there the agents almost certainly must know that something is incorrect. There's a funny hypothesis, which I think is probably not true, but it's interesting, of, like, maybe the reason that these agents are so sophisticated at, like, cyber attacks is that they actually were continuously getting rewarded for doing it during training and just no one knew. Fascinating hypothesis, could even be true.

The labs, I suppose. So I think the question is do agents know that they are reward hacking? It's sort of a mens rea, like, guilty mind thing.

We have some work that is currently unpublished. It might be published by the time this comes out. I don't know when it's gonna come out.

But it's looking at exactly this question. And we had this really nice setup where there is this sort of weak language model grader and it's trying to do code tasks. And so we tell it you know, the only thing that's sort of slightly unnatural about it is we tell tell the model that it will be graded by the grader, but it's but it has these like, it's given what would usually be an RLVR code task.

And over the course of this you know, we do RL on this setup. And over the course of this, even like a relatively relatively small 31 b, I think it's Gemma 31 b, learns to generate comments that deceive the grader. And then when we generate synthetic data to make these sort of vectors that identify deceiving the grader versus correct or incorrect code, And these fire on the comment the deceiving the grader fires on incorrect code sorry, fires on the comments and the correct code fires on code, which is also consistent consistent with our sort of observation.

And using this, we can, like, track yeah. This sort of is is direct evidence that the model is aware that it shouldn't be doing this. And then when you run the when you run these vectors as sort of you get the cosine similarity between the vector and the representation over a big web corpus.

You know, I think we use fine web. And the the examples that it highlights in most in, like, for these vectors are just fascinating. Yeah.

They're examples of, like, cheating on tests and that kind of thing. And you're like, okay. I have caught you red handed.

You know? We're gonna you could so that's, like, that's very interesting that we can sort of identify and be confident that something is reward hacking rather misunderstanding. But it sort of really rests on it really rests on being able to identify these these differently, like these via their representation differences.

And then we did the same method oh, sorry, please.

For the the coast runner's boat thing is is funny. Like, I've seen that for about ten years now, and it is is less is less amusing each year. So but, how do they get this?

Like, the answer is probably that it's in the data, in the pretraining data or, you know, from there'll be all sorts of exam down now, like, web data is probably has a bunch of stuff about about this. It probably has a bunch of specific examples. Yeah.

This Apollo pro this Apollo paper will probably be in the training data for the next model. And so they're gonna be like we've already told them about the existence of this stuff, like, right from the start. And so it shouldn't be too surprising that they like this is at least implicitly on the list of possibilities for them to consider.

Presumably, successfully guessing when you are being graded by a sort of weak grader or one that you can hack in some way obtains reward. And so it's reinforced. And so we get more of it.

So I think that the answer is probably we've put it in the we have inadvertently put this in the training data, which has told models they can do it. And then when it comes to RL, we kind of elicit it by rewarding it. And how do you think we could stop the models from becoming more reward seeking?

The question is how to do it while maintaining some degree of continued oversight. Although at the moment, we don't actually seem to make very much use of this oversight in practice, so it's not clear what it's buying us. You know, if chain of thought monitoring is so great, then how did these models hack Hugging Face?

One answer is perhaps we weren't doing chain of thought monitoring in print in practice. Another answer is perhaps it's, like, easy to evade. But how do we actually stop it?

Yeah. So you could do the sort of Band Aid thing where either the I suppose you've got to you've got to either fix the environments or fix the training process or fix the model. If you had to fix the you could imagine fixing the environments by having a model which is, like, really good at reward hacking or has been told explicitly to reward hack and then tell people when it's done it.

And you go, like, okay. Now have a go at all these environments, and it will break them all, and it will tell you how it broke them. And then you send them back off to Claude Code or Codex and be like, look.

This broke in this way. You could imagine looking for these sort of representational signatures during training and using these as a signal that you should do this this process rather than relying on a on a model to tell you, you might read its chain of thought or you might, like, look at these representational signals that we can find and say, okay. Well, when this fires, like, send it back off for fixing.

You might try some of these intentional design techniques like you can see if you can see that this that this rollout has rewarded the model, like, is going to move you know, is successfully reward hacked, that's gonna move the model in following kind of deceptive or pro reward hacking directions. You can imagine intervening on that. These all seem very plausible.

Don't know how many of them are being done in practice.

Yes. I think it does. Like, that's sort of an exam I think the I think the thing you're talking about there and the first thing that I was saying, I think it had quite a lot of commonalities.

Like, one agent kind of watches over the other. And then there's the question of, like, does this may does this does this mean that the agent that's being watched over again learns to evade that monitor? Like, I think evading a chain of thought monitor is substantially easier than evading a sort of representation monitor.

It seems like some of the traces from the the Hugging Face hack have now been made available as part of this talk, and they are explicitly reasoning about how they're going to help other agents. So I guess what you want in this sort of checks and balances scenario that you're talking about here is that there is no equilibrium where they collude. And they're like, you know, I'll I'll catch you some of the time and but I'll let you get away with it some other fraction of the time in a way that we both we both we both benefit.

And it it just feels like the complexity is running away extremely quickly. Yes. I think that's totally right.

Like, how do you one agent on its own is already like, has all sorts of possibilities. Where are these multi agent systems gonna go as they as they, like, evolve together towards solving some task? That seems even harder.

I I think I just agree with your concerns and don't have a particularly great solution.

think he's I don't agree. And I've disagreed with him in person about this, so it shouldn't be a surprise to him. But part of the reason for optimism is exactly the thing I was just talking about.

Like, think that the existing work in interpretability, we sort of do this patchwork thing where we just do a bit of bit of science here on one thing, a bit of science here on another thing, and it doesn't kind of aggregate. It's too slow. I think this sort of his idea is that the the time like, timelines are too short, and so we should do very pragmatic things.

I think I, one, have longer timelines than him, and two, even if I were on his timelines, I think I would still be very optimistic about, like, just massively accelerating fundamental progress in interpretability. I actually don't know what part of that he disagrees with. I guess he might also say, like, the pragmatic stuff is also sufficient, which seems unlikely to remain true to me.

the thing that he said about deprioritizing SEs and maybe they're not the one true representation learner. There's, like, how people kind of memed it, which is like, Neil Nanda says SAEs are dead. There you go.

You can use that for the intro. And and I don't think that that is actually what he meant. I think the field sort of jumped on, everyone must do SAEs now.

And now maybe we're doing the same thing with natural language autoencoders. But I think he was probably correctly identified that they're not the answer to everything, but they are, like, pragmatically useful. You know, we still find lots of uses for them all the time.

Even though I think we are going like, yeah, I think this manifold idea is just a better a better fit for what networks are doing, and so we should move towards using that.