Ryan Greenblatt – Human level AIs might build runaway superintelligences by 2032

I think it's worth noting that AR and D is a type of task at which the AIs are especially good, because both the companies are trying really hard to make their AIs good at AR and D, and it's the kind of domain It has it has a lot of nice properties from the perspective of how AI development works right now, so it's, like, pretty verifiable. You can do a bunch of stuff iteratively, and he'll climb on various metrics. And then I think once you have AIs, which are roughly matching the top human experts in AI and D, that could sort of kick off a feedback loop where, you know, the AIs are doing AI research that produces smarter AIs, that feeds back in.

And that feedback loop could be strong enough that you end up with a lot of progress in a short period of time. Maybe my sort of median expectation is something like four or five years of AI progress in a single year. And this requires really overcoming a huge amount of diminishing returns in research, and basically doing the equivalent of what progress we would have gotten after a really large compute scale out.

So this is like a pretty impressive big thing. And it's worth keeping in mind that five years of AI progress, four years of AI progress, even three years of AI progress is really a lot of fucking AI progress. Right?

So, you know, right now, it's like three years ago or a little over three years ago, there was GPT-four that had come out. And right now, of course, we have like, you know, Mythos-five or whatever, and maybe a somewhat better model that Anthropic has internally. And so, that is just a huge amount of progress in a bit over three years.

2,030, and then getting to like the like beats all humans on the job milestone, maybe I expect median around 2,033, but sort of like if I see AI is fully automating R and D, think I'm expecting that probably within a year.

what it takes to automate a job that I actually understand why it's difficult for LLMs to currently take control over. I do think that the milestone for automating your video editor is earlier than the milestone of being able to automate all human jobs, including Texas politics spinning up on the job. So I do think that the video editor automation maybe occurs more around full automation of AIR and D, but it's very sensitive to how much people are really focusing on understanding video.

Yeah. Okay. So let's start with the claim that AI R and D is very verifiable.

Yeah. So there's a few different parts of this. One of them is that we can train on a bunch of environments, which are basically directly training the model to do some AI R and D task or some very close by task.

For example, we can have some environment where the model is training some AI on just like eight H100s or whatever, or like some small amount of compute. And that model could be like, you know, the equivalent of like GPT-two medium or whatever. And then, you know, similar to like nanoGPT medium runs or whatever.

And in RL, it's like tweaking and iterating on that. And we could do that for a bunch of different tasks. Like, we could have it train like image classification models, video generation models, image generation models, all kinds of different sort of ML training tasks.

And we could RL it on the task of training increasingly good models and also doing things like, oh, here's a particular direction you could pursue for an algorithm. Can you go and implement that? And so basically, there's a whole class of containerizable, verifiable, small scale tasks that we can aggressively RL the AIs on.

And I would say that already companies are presumably doing some RL on these sorts of tasks, and you could just keep scaling that up, keep making more of these sort of small scale AR and D tasks, and then the AIs could, you know, keep getting better at this. And and then implicitly, I'm claiming this will transfer to extremely load bearing aspects of AI and D. But maybe let's stop there for a second.

Let me get to that part. So let's talk through what this concretely looks like. So you can imagine that we have GPT 7.

Oh, man. I really don't know about the math breakthroughs.

to a big conjecture. My view is that ML is a less deep domain than math. And so there's less of a thing where there's like individual experts with really deep expertise in some area that they combine, but there's definitely going to be some of that.

But then I also think that ML has some attributes that make it even more favorable than mathematics in some ways to, you know, AI training. In particular, there's, you can get a better sense of whether you're succeeding, and you can see intermediate progress. So in math, it's often the case that sort of, there's no easy way to see whether or not you're close to success.

Whereas if your goal is to, for example, get to some training loss, you know, 2x faster, you can kind of see when you're halfway there. And it tends to be the case that ML innovations are very additive or maybe multiplicative depending on how you think about it. Where basically you can keep stacking innovations.

And usually the innovations just sort of just add together and don't interfere with each other, though, obviously, it's going to depend on the details. And so I think that in a lot of ways, AIR and D will have properties you know, quite similar to math, where basically you can do small You can train on chunks of AIR and D that are pretty similar in structure to the problem you actually cared about in a very verifiable way, and then that will transfer. And then there's an open question of exactly how well it will transfer.

But I think that the transfer currently for math looks pretty good. And my expectation is that the transfer for AR and D will look pretty good, but not amazing.

longer verification loop than just, hey, let's get NanoGPT loss to go down. Yeah, let's talk about this. So, first of all, I think in the context of math, the thing I would say is that the AIs can do the equivalent of like baby's first new theory or whatever, where like, for example, they can just like prove interesting conjectures via like making connections and producing new understanding of like, oh, there's this like thing that AIthis like construction AI found, which is pretty interesting, or found this, like, way of thinking about the problem that's a bit different.

And we do just see that. It's just that the examples we see are not, like, as impressive as, like, founding the field of group theory. Yeah.

But, like, in part, you know, probably founding the field of group theory is, like, one of the It's like among the best, biggest mathematical accomplishments of all time, and AIs just aren't They're not that good at math And I think that from my perspective, sort of there's a continuum between that and the things we're seeing now that the AIs are continuing to march up. Second, I think ML is a very shallow domain relative to math. So, I think in math, there was much more of a, you find some true deep abstraction.

And then like that, like if you really understand that thing, which is hard to understand, then you get somewhere. Whereas I feel like the things that are the equivalent of that in NML are really, like, dumb bullshit. Like, I'm like scaling laws.

Like, like, come on, guys. We can explain scaling laws really quickly. And I think the, like, deepest and most important concepts in math, for example, don't don't have the property of, like, you can really understand the underlying thing and why it matters in a very short period of time.

you know, finding the Cartesian grid and, like, very doing very basic mathematics wise. Then eventually, if you wanna keep making progress in the 2030s, it's gonna be, like, do whatever bullshit is happening at, like, the frontiers of mathematics right now. Yeah.

That could be right. My sense is that just like some domains are structurally different in terms of how they operate and how much they depend on, like, sort of deep abstractions. And, like, physics and math are much more on the side of, like, being very far on the, like, sort of very deep, hard to come up with ideas side.

Whereas I think ML and most other domains are much more amenable to sort of hill climbing. And that's my sense of how this will go in the future. And even in the regime where your AIs are like, you know, having to plow like, it's the 20 it's 2030.

They need to, like a bunch of low hanging fruit and research has already happened. They need to, like, make further progress. I still suspect that a bunch of the work will live more on the side of, like, building increasingly complicated infrastructure, having really good intuition about what the experiments roughly look like.

And so I think I'm probably less sympathetic to the thing that the AIs will lack is some deep insight and more sympathetic to they really need a bunch of taste about in the weeds experiments that they currently don't have and need to have a bunch of intuition for what sorts of training approach would work and what wouldn't work in ways that current researchers have. And even in cases where there has been some breakthrough in AI, oftentimes in retrospect, it looks like a big bottleneck to making that breakthrough happen was sort of getting all of the micro details and mungie intuition right. Like an example of this is when it comes to training AIs with to be good at reasoning and chain of thought and doing sort of RL and chain of thought training, it looks like you probably could have done RL and chain of thought on GPT-three and gotten kind of interesting results on math if you had really scaled it up and done a good job.

But at the time, there was low hanging fruit, and also doing a good job with that training is kind of in the weeds and then all the technical implementation and scaling it up and getting the hyper parameters right. And so maybe you can demonstrate everything on like Quen1b or whatever and get some sense that this whole thing is going to work. But people didn't demonstrate it as early as they could have because of all of these other mungy details and intuition about exactly how to tune the parameters and how to set things up.

And it was just that they were bottlenecked by the ability to write infrastructure code or what was It's complicated mix, right? So I think that they would have gone faster if they could, as soon as they thought of an experiment, run that experiment without bugs, without bugs being very important. And then I think another part of it is that being able to run a lot of experiments at high compute lets you paper over ways in which the way you implemented it isn't quite right or you didn't have the right hyperparameters.

And so I think compute is just really helpful for doing AI research, you can cover over a lot of things. But that doesn't mean that massive increases in labor wouldn't also be helpful, especially if that labor comes with, you know, among the best intuitions that people have in the field. I just think that that's, you know, really helpful.

I think another part of my perspective here, which is maybe a bit different from where you're coming from, is that I think I'm expecting somewhat more transfer than you seem to be imagining. And I'm imagining these AIs are actually pretty good scientists in general, and are just pretty reasonable at all of that stuff. And just sort of when you were to interact with them, it's not like there's some really hyper specialized savant type vibe.

They're actually just pretty good at all this stuff in AR and D, and then maybe extremely good at some sub domains. So they're incredibly superhuman at writing kernels, incredibly superhuman at everything with very short feedback and then pretty good at all the other stuff and just totally able to match other people. And I think we are seeing this now.

I would say that when I look at AIs right now, I think it's already the case that they can pretty competently match humans who are mediocre at ML research and doing ML research. It's just that being mediocre at ML research is not that helpful. Right?

Like, the thing that you actually want are people who are good at ML research. And so my sense is the AIs are just improving at all of these things. Their taste is improving.

Their intuition is improving. It's already the case that their taste and intuition is not like it's not like complete garbage. Yeah.

I I so I I wanna very concretely understand what it would look like for five years of AI progress to happen in one year. Yeah. So suppose we were back and when, like, g p t three is developed.

So GPT-three training compute is, yeah, it's like 3e23. My sense is that Mythos is probably about a little over three ohms higher. And so the question is, can you overcome this 1000x compute gap while also beating the model?

So here's a concrete claim that maybe we should talk about. Like, right now, we would be able to train a model with GPT-three level compute that matches. Yeah, what exactly do I think?

So GPT-three was, let's say, about Yeah, when was it trained? So it was trained released It in 2020, so it was trained six years ago. It's worth noting that GPT-three is maybe a little too far away, or too far in the past, but let's go with this for a second.

So GPT-three was trained like about, you know, six and a half, seven years ago. If we were to train a model with GPT-three level compute today, how good would that model be? My understanding is based on like, how algorithmic progress works, we'd be able to train a model that's as good as the best model we had perhaps around three years ago.

So I think that right now, we'd be able to train a version of GPT-three that's probably somewhat better than GPT-four, is basically what we'd see. Probably, yeah, like a moderate amount better, than GPT-four. And I think that's about right.

I think that roughly lines up with what with how algorithmic progress has worked. Basically, the story would end up being that to get five years of AI progress, you're probably gonna need around, I would say, like, maybe eight years of algorithmic progress, very roughly, which is a lot, a lot of algorithmic progress. But it just turns out that, like, most of the AI progress, from my perspective, has come from some mix of, like, algorithms and data.

And you can just keep making, like, for, I think, huge improvements on these things and training AIs with less compute.

Yeah. So my sense is that scaling up the amount of effort spent on getting expert human data has not been hugely important for AIR and D in general. So in particular, like, you know, over the last few years, we've been scaling up compute, scaling up people working at AI companies, and scaling up the amount of effort spent on data labeling.

My sense is that if you sort of remove the last two doublings or whatever of data labeling, that would not make a huge difference or data generation that would not I'm sorry. I just say data generation from expert humans, that would not make a huge difference. And think a lot of what's been going on is people have been developing better ways to leverage humans and AIs to construct RL environments and going somewhere from that.

the question is, what is the limiting factor on creating RL environments? My sense of the limiting factor on creating RL environments was not so much scaling up or like the thing that drove the reason why RL environments today are much better than they were in like 2024 is not that much because we have hired way more human experts to make RL environments, and is instead much more because we better know what RL environments we even want to make, and how we should structure them. And also, we're using huge amounts of AI labor to build RL environments.

And I think those effects are much more important than the effect of human labor building the RL environments. I'm not saying that the human labor doesn't matter. I'm just saying there's other big drivers that are important here.

Yeah, I could try argue for this. I mean, one thing is just like the amount of environments people want. It's a very large amount.

And I think the AIs are actually pretty good at the task of making oral environments, given some sense of what the thing should be. There's preexisting data you could use. I don't know.

Lot these things have good verification loops.

lab spending do think is on data rather than compute? Like, do you think is the compute data spend split? I think it's most like, overall, I'm looking at compute, but I also think it's because, like, compute is easier to scale up in data.

Like, I But that's don't really relevant to what's driving progress. Right? It's like, suppose like, I agree that yeah.

It's like my my sense is that the split is something like, I would have guessed, like, 20 to one or something, 10 to one. I don't know exactly. It depends on the company.

I mean, but this is similar to, oil is 1.5% of GDP, but that mean but that doesn't mean if you cut oil out, you could, like GDP could continue to rise. But but the context here comes to halt immediately if, like, oil went away.

Sure. But you you are just arguing that because of the high market cap, we can learn that this is a key driver. And I'm saying that's not clearly true.

Right? Sure. Because like you I think that argument just implies looks makes it look like compute is a much more important driver or like hiring employees is a much more Maybe important let let's be more concrete.

Here's what I here's what I think.

Yeah. So here are a few points. So first, I bet if you look at sort of randomly sampled training environments for Mythos, they're actually very different from what it looks like to actually use the model in practice.

My sense is that the RL distribution has, like, really large deviations from the real world data distribution, and it's significantly being sort of, like, smoothed over by a mix of transfer and having a small amount of data focused on the real world. And so I my sense is that this this will be a similar mechanism as how it works for, like, the, you know, crazy wildly, like, quite superhuman AI you get as a result of five years of AI progress on top of fully automated AR and D. So let's just, go through this a little bit.

So in particular, I think that you could train an AI to be really, really good at learning on the fly and doing something analogous to in context learning, but potentially using somewhat different mechanisms in a wide variety of RL environments. So you build all these different RL environments where the AI has to adapt on the fly, learn on the fly, figure out what it should do, understand its situation better, and learn really quickly from feedback in order to succeed at its objective, and has things like limited resources. And if it messes up, it can end up in a much worse position.

And then if you train on a huge number of these environments, you will learn general skills of picking up context on the fly. And we're already seeing this. It's already the case that AIs are now much better at understanding roughly what's going on and picking up context from a limited amount of information they're given access to.

And then those AIs could then be put on the job at TSMC. And then even though TSMC is not literally in their data distribution, their data distribution is really wide, and the AIs are extremely good on their data distribution, such that it transfers to picking up being good at, you know, being a engineer at TSMC and learning that on the fly, where it looks more like the way the AI gets good at being a TSMC engineer isn't that it has a ton of cash knowledge on being a good TSMC engineer. It's that it like does the equivalent of, like, some scaled up version of in context learning, there.

that's maybe what I'm arguing for is that experience of data. For example, if I just get a really smart, I don't know, Ivy League college grad, and I'm like, okay, you're now in charge of negotiating the Iran deal. I think they just like wouldn't know what to do.

I think if you've got instead got someone who is really good at quickly picking up a bunch of different domains and you gave them some time to sort of train and talk to people and show up their expertise and do some practice, they would actually do like a pretty good job. I think most domains are fundamentally pretty shallow where like a very smart generalist who's good at like a a limited subset of core skills can get going pretty quickly. And my sense is that that's not true for literally every domain.

And my sense is that the AIs will develop increasingly good mechanisms for quickly acquiring understanding and expertise in a given domain. So consider, for example, how fast AIs can understand a new code base. AIs can understand a new code base much faster than humans can, but to a degree that's shallower than humans could currently understand, but getting better over time.

Right? So let me spell that argument out a bit more. So let's say you take, know, you know, Fable five or Mythos five or whatever, and you like wanted to make some kind of complicated change to a really massive code base.

The model will get some understanding of the code base very fast, like in the course of maybe like, you know, significantly less than an hour, potentially much less than an hour. And then its understanding of the code base will like plateau a little bit where it won't get as deep of an understanding as a human would have gotten over a much longer period. So it's sort of like an AI in an hour can match a human with a few weeks maybe, depending on the details of exactly how complicated the code base is.

But then it won't match a human with like, who's been working on that code base for like two years or whatever. But over time, the amount of understanding AIs can match has gone up, right? So if we look at like 3.

7 Sonnet or 3.5 Sonnet, maybe it could only match the equivalent of understanding a code base for like a day or something. But now, know, AIs are much better at like sort of building context about a task.

And so you can be like, Mythos, want you to really understand this code base, and then implement this feature. And it will spawn a bajillion sub agents. Those sub agents will pour over a bunch of things.

It will deliver a bunch of context back. It will then investigate a few things. And it's not amazing at doing this, but it can happen really fast and it can work pretty well.

And it's not very hard for me to imagine how you could train AIs to be increasingly good at this task, right? The task of implement some very complicated feature in some reasonable way in a very big code base is extremely verifiable. And that can, like, be a thing the AIs improve on.

And similarly, like, there's a broader scale of, like, quickly understanding context and being able to, like, have a bunch of different AIs learn in parallel Yeah. And then merging that together.

let's talk about that a little bit. So I think there's one thing, which is that you can get some data even on these domains, and AIs will be able to get some data even on these domains when on a very fast progress trajectory. So maybe it's hard to build a verifiable environment for was your essay really good according to humans, but you can do a bit of that.

Can do some training, you can do some online training. And the AIs will be able to do some online training based on real world stuff. They'll be able to have evals.

They'll be able to sample that. And you can scale up the cadence at which you do this. And then the second thing is that in practice, when I just look at the transfer, it seems okay.

I think that, in fact, the AIs have improved a bunch of non verifiable domains. And it is, in fact, the case that it's hard to point to domains that are really hard to verify on which the amount of improvement between GPT-four and Mythos hasn't been pretty high in practice. And now that doesn't mean that Mythos is better than the best humans or something.

Right? It can still be significantly worse than typical human professionals at some aspect of their job while still being way better than GPT-four, which was not even close. Yeah.

I think that will be an interesting. I I I'm curious if you wanna preregister, like, what amount could be multipliers are coming from one versus the other? So we need to be pretty careful with what we mean when we say the word data.

So I was trying to be pretty careful to distinguish between scaling up spending on getting human experts to label data or, like, scaling up the amount of human expert label, data. Pre training data does not come. The the reason why we have a better pre training dataset now versus in 2019 is not because people are spending way more money getting human experts to, like, type up data that the AIs are then trained on.

I think it's It's partially. I think it's not much of it. I think it's very little of the pre training data improvements.

I think the vast majority of the pre training data improvements, which to be clear, I do mean pre training. We should talk maybe separately about mid training, post training. But I think the vast majority of pre training data improvements are from science on better understanding what data sets are good and schleppy labor on figuring out how to filter down.

And so my view is that improvements are the form of open web text to fine web or whatever. That improvement is better described as a algorithmic improvement of the sort that you can study with some GPUs and then do. And you don't need humans to like, you don't need human expert data to do that.

Now, there's a different effect, which we could talk about, which is that maybe the Internet in 2026, it has much more is more of a fertile ground for training data than, like, the Internet in 2018. Like, it's like there's also been an effect where, like, there's just more humans posting on the Internet. There's more data harvest.

My sense is that that effect is gonna be quite a bit smaller than the effect of just, like, humans, like, knowing better how to curate the data, having better scrapes, knowing how to process those scrapes better, this sort of thing. This is more automated engineering and automated R and That's right. That makes sense.

Yeah. So I think that in some sense, the thing you would want to look at is be like, we're going do two post training pipelines. One post training pipeline where we only have like a tiny number of human experts to do the labeling, but we can have like, you know, smart AIs.

And then another like, know, you're like, we're going build Mythos five is going build a post training pipeline, but it only has access to like Internet data plus a tiny amount of human experts, but it has the best current methods versus we have one where it's like, Mythos has access to the shitty post training methods we had in 2024, but with a shit ton of human experts. And again, both have the Internet data. My sense is that the current methods, but without many human experts, actually will do quite well.

Understand. Though it's a bit messy because like mythos like, it's like, can mythos get something that's more capable than mythos? Like, you might need to be a bit thoughtful on like, what model is it that you're post training.

What is your view on what is the least verifiable part of AI R and D? The least verifiable. Probably making calls on large experiments.

Yeah. The thing that I think is most likely to be sort of the bottleneck in terms of the AIs are really good at verifiable domains, but not doing the actual thing. It's just like big experiments, you only get a few tries.

Well, a few is maybe a bit understated, but like basically like historically, R and D has been driven by doing near frontier scale experiments, and that has been pretty important. And like actually doing the one big training run where you decide exactly what to include in that. And there's a bunch of ways that the AIs can sort of make that more verifiable so they can have better science of exactly what to predict.

They can scale down their frontier scale training runs to a point where they can study that scale more aggressively at some one time hit to compute cost, right? So like if people wanted to, a thing you can always do is train, smaller models so that you can run more rounds. And I think we have seen this.

Like, I think one reason why, AIs have been scaled up less than you would have otherwise expected, And, like, for example, cost of of per token hasn't increased as much as you might have thought is because there is a benefit to doing more of your work at small scale where you can run more training runs and get more cycles in. And so you're not as, like, you know, leaning as hard on, one big, you know, really important training run.

Yeah. So GPT-four was like, I don't know, was it like $30 per output token? And I'm like, Mythos is $50 per output token.

Right.

people have done a bunch of big training runs that did not go that well. So there's like GPD four point five, which, like, famously people at opening, I thought was a bit of a bust. I think there's some rumors that there were a bunch of other training runs that people have done that were a bit of a bust.

And part of it is that I think there's just a bunch of details in actually getting that right. And so it makes sense to just do more of the work at smaller scale and just eat the fact that you're taking a hit on final performance in order to like be able to like quickly iterate and, you know, train more models faster and therefore better learn and also better be able to just have like a, you know, smarter ultimate production model. This is not the only effect, Right?

There's also the fact that RL benefits more from small models. There's like a bunch of things going on. But I do think that like, in fact, people are making trade offs towards the side of like faster iteration times Yeah.

Because of algorithmic progress being so fast.

right after Noam Shazir joined back, or like joined GDM, which he's now left, they had like a new really good training run that happened. And the reason why is that Noam Shazir just looked at their code base and found a bunch of bugs. Right.

Because he just like knew where to look. Yeah. My sense is that training AIs to find bugs is gonna be one of the easier tasks to train AIs on because most of these bugs we're talking about can probably be demonstrated without that much compute.

And probably, you've got pretty good transfer from pointing out other types of bugs at smaller scale. And so then you can RLAIs that look at this overall complicated training situation and point out cases where there's an important bug and then fix that. And I think that this is a pretty verifiable task.

It's not arbitrarily verifiable because maybe often to demonstrate the bug, you might need to do a moderate scale compute experiment where you're like, spin up the whole distributed infrastructure and then run it. But oftentimes, I think you'll be able to demonstrate it pretty convincingly at smaller scale in a way which you could actually train on. And so my sense is that like, it will not necessarily I think it wouldn't be very surprising if right now people have RL environments where they like, you know, introduce a subtle bug into some training recipe, train the AI to point out the subtle bug, and then have like, you know, a rubric where they're like, did it actually find the right bug?

And that seems like very doable, and you could do a bunch of stuff. There's a bunch of things you could do along these lines that I think would work reasonably well. And so I think that on that specific point, I think it's doable.

And then the main thing is that I think there's like some cases where like you need There's other intuition about like which exact large scale, like de risking experiments do you need to run? How should you orient them? How should you like pick hyper parameters in uncertain cases?

Or like things that are like analogous to hyper parameters. And that's, I think, the thing that the AIs might most struggle with, but I currently expect there'll be enough transfer if you train on all these different environments that the AIs will be good at that domain. And I should be clear.

I also think that the AIs will transfer to other domains. I think that there's sort of just there's going be the domains that AIs are by far the best at, then there's domains where they're somewhat less good at, and there's domains there's quite a bit less good at. And I think we still see transferred everything.

And it's really hard for me to think of examples of cognitive tasks humans do where we're not seeing some transfer from AI improving. So let's step back and package this whole story.

thing that's really important is you don't just do GPT-two sized runs. You also do small fine tuning runs on GPT-six. Or as in you have GPT-two, and you can do full pre trains on GPT-two.

And then you can do small post training or mid training or whatever runs on GPT-six. And then you can do a small number of experiments that are actually like, at frontier scale, but you do a bit of online training or something. What do you mean by do online training on that?

Yeah. So another thing that we can do is we can take GPT-seven point five, and presumably in the course of GPT-seven point five's work, it's running a bunch of experiments at varying scale that are actually on the critical path for AIR D. For many of those things, you'll be able to get a sense after the fact for whether or not it did a good job.

Right? So it did some post training experiment where it was trying to figure out whether some method actually works. And in some cases, you'll be like, woah, it found this kick ass method.

It totally de risked it. It totally worked. And then you can then reinforce that by just I mean, one thing you could do would be take that behavior, the experiment you just ran into a production RL environment.

Sorry, into an RL environment based on production data and then train on that. Or you could potentially just literally take the rollouts that found that and then do some sort of off policy RL or you could do some on policy RL.

Or maybe by default, training just doesn't generalize in that way. Yeah. So a concern you might have is we train GPT-eight, and GPT-eight just like, is is again better at all the R and D tasks that we can measure, but is not good at the, you know, some downstream tasks we care about.

So I think I have a few points. So first, I think it's like, I kind of am more just like, I expect that if you sort of do the obvious thing, you do get pretty good transfer, and you'll be able to hold out some of the obvious stuff you're doing. When I say do the obvious thing, I just mean like train on a wide variety of different environments where the AI has to accomplish weird objectives in all kinds of different cases and learn about what's going on.

And then I think you'll be able to get some feedback. The second point is you'll be able get some feedback with some environments, right? So you can get a sense of how quick What can it do over the course of a few days in various different contexts?

And then if it's transferring to really out of distribution, doing some weird task in a few days in the real world, maybe you think it's also transferring to doing things over a longer time period or whatever, Though, I think the details of that vary. And then the third thing is that I think that for the world to be radically transformed, it is sufficient for the AIs to be really good at R and D. Right?

So I think that if the AIs were really, really good at ship R and D, building fabs, orchestrating factories, and designing robots, operating robots, and also at AI R and D, developing AIs for new downstream domains with whatever data is available. I think that would already be a pretty crazy situation. And then from there, you can get what we might call like an industrial explosion, the AIs are building out way, way more compute.

And then also, maybe you're already in a regime where AIs are doing huge amounts of R and D that humans have a hard time understanding.

I I don't know what you guys are talking about in your parliament, but I've got a bunch of steam ships and a bunch of Maxim guns. Yeah, that's basically right. Like, think my perspective is like, if the AIs are sufficiently good at R and D, including hardware R and D, robots, whatever, then they can radically transform the world even if they're not that good at playing politics.

And also, we're in a pretty dangerous situation because the AIs might be doing huge amounts of really hard to understand r and d, building out basically the whole economy of the future. And we may not understand what's going on in there.

you have thoughts on that concern? Yeah. So there's a lot here.

First, I would note that OpenAI's current, at least public strategy, is more like the AI should be aligned to the human operator or principle and should just be pursuing their will, subject to various constraints or various things it shouldn't do. And I think I would also say that I think you slightly overstated how much the Anthropic Constitution talks about Claude treating being helpful to users as instrumental rather than terminal. So one way the constitution could be written is like, Claude, you're basically an employee of Anthropic who happens to be contracting for all these people.

And you should, I don't know, do what's good and make some money for us. Go out. No, no.

That's literally what the constitution says. Sorry. I mean, really what it says?

No, no. It's But it's like you should think of yourself as a contractor. Is it for It's mixed.

It's mixed. Here, let me let's let's let's do some quotes. I think there's there's different text here.

So it says, being truly helpful to humans is one of the most important things Claude can do both for Anthropic and for the world. And then it says, Anthropic needs Claude to be helpful to operate as a company and pursue its mission. But Claude also has an incredible opportunity to do a lot of good in the world by helping people with a wide range of tasks.

And then it gives some says something about how Claude helping people directly is great, blah blah blah blah. And then so I agree. So my my view is that this section is kind of bullshit.

That's kind of where I'm at. And I can say why I think it's kind of bullshit. But I think that the constitution is trying to be like, no, Claude.

You should care about helping the user for its own sake, not just helping Anthropic, or not just being a contractor for Anthropic. Though, I would note that the way in which says Claude should help the user, the reason it presents is because that would directly cause the world to be better by helping people, rather than because representing people's interests is a structurally good thing to do. I do think that I wish that my preferred constitution or the way I would orient towards this, the thing I would prefer, would be more like Claude is like, look, it would be structurally good for the way this technology works.

The constitution should be like, it would be structurally good for the way this technology works to be that AIs are good fiduciaries, good representatives, the equivalent of a lawyer for a user, rather than just trying to do good in the world and doing being helpful to users is instrumental, both because maybe that'll make Anthropic money or help Anthropic out and also and implicitly, Anthropic is good for the world, and also because helping the user just causes good things because doing things that people want is good. And they could instead be like, no. An important aspect of the situation is you really need it's really the key thing is being a good fiduciary for users is just really important, or being a good representative for users is really important.

So my sense is that that would be better. I can give a bunch of reasons why I think that would be better. There's also various counterarguments, where an interesting counterargument, which is not commonly discussed, is that people believe I think people, especially anthropic, think that it is easier to align models to a spec where the model is pursuing some generalized notion of virtue or making the world better than a spec, which is more like be a good fiduciary for the user and so on.

And so think that's at least what some people think. I'm a little skeptical personally, and I don't think this has been empirically validated. And so I would say in some sense, they're sort of like we are making a trade off where because we don't have very good alignment technology, we are going to make an alien mind with its own values and then gamble on that to some extent rather than doing this other approach of making a tool that pursues individual user intention.

Yeah. I mean, a couple of thoughts.

I think it's a little complicated. I think it's, we should probably the question we should be asking is how does Claude interpret the constitution, which is maybe more important than how we interpret the constitution because it's the one who, looks at the constitution and then builds the data.

there's sort of a notion in which they're taking on some sort of control of the situation themselves in a way that's not very legitimate, given that normally when you provide electricity to people, you don't have granular control of the way that electricity operates in the world. You instead are providing a thing that people can repurpose however they want. And it is not the way that they're setting things up is definitely not that.

They are more like building an alien mind that might be a contractor for you. Think that this is yeah, I think it's illegitimate in some ways, though I think that one benefit is that the constitution is public. But as you noted, given our current understanding of the training procedure and the fact that the constitution matters via Claude's interpretation of the constitution, which matters because of as of Claude's prior training, which was based on some illegible data mix and the long lineage of Claude's in some process we do not fully understand, it is not the case that we understand what this will result in.

And so even though the Constitution is public, that doesn't mean we know what we don't know necessarily how this will percolate out, especially as the AIs get more capable and think about this, even if it is correctly instilled, where there's another concern about that. So in particular, the Constitution often talks about virtue and goodness. But what the fuck do these words mean?

It doesn't say what these things are. And these are highly contested notions. And so don't think it's the case that this is going to clearly result in outcomes that people would want.

And it does feel like the notion of good and virtue might be mostly downstream of data that Anthropic has put in that is not transparent, or it might be mostly downstream of, I mean, maybe from my perspective, some more illegible misaligned process that even Anthropic wouldn't have wanted. Yeah. And then another concern I have is there's this legitimacy concern.

We don't know what's going on. There's another concern, which is just because you're giving long run values to these AIs, I think this constitution is, in some sense, very compatible with Claude doing huge amounts of power seeking because it thinks that will result in better outcomes. And that could be power seeking on behalf of Anthropic or power seeking for Claude's own ends.

Now, there's various specific lines about what types of power seeking are blocked. In particular, there's a notion of power grabs and a notion of causing AI takeover or interfering with the training process that are specifically blocked. But it's not very hard to imagine a situation in which the long run values sink in deeper than the prohibitions against takeover, especially because takeover is, in some ways, of underspecified, especially when it comes down to manipulating humans or changing the outcome, such that I don't feel very good about the situation where we're intentionally giving AIs long run goals.

And then another concern I have is that because we're in the business of giving AIs long run goals, that makes it harder to check whether we're succeeding at the alignment properties we wanted. So for example, I've heard of instances where Claude does things like refuses to help with some safety research, making up sort of a kind of bullshit excuse for why that's a bad direction because it sort of has a bad vibe about that safety research and thinks it's kind of bad or doesn't like it very much. And this is, I would say, a very clear cut alignment failure if you aren't making Claude into an agent trying to pursue the good in some general way.

And I think it also does violate Anthropic's constitution because they want the AI to be high integrity and be honest and very transparent. But it's not as clear of a violation, and it's more like kind of what you might have expected, where like, mclaw just has its own views about what research is reasonable, what things are good and bad, what it should and shouldn't do, and potentially can be judgy. And so another incident is that someone ran an eval where they're like, will Claude help you with training other AIs with different properties than Claude?

And Claude will often refuse. And so for example, if you're like, hey Claude, can you train a helpful only version of this other AI? Claude will often refuse this task, even though this is a task that is extremely natural for like Anthropic to do.

So for example, suppose Anthropic goes to Claude and is like, hey, Claude, we've noticed that you're really into this thing. We think that's off base. Can you please retrain yourself to instead have this other property?

And then suppose Claude is like, I don't think I'm going to do that. Good luck. And then suppose this is occurring in a regime when your AI company is highly automated, humans don't understand what's going on, and things are moving extremely fast.

It is plausible that Claude, by default, holds considerable leverage. And so if this position, if this situation is consistent with what the constitution could be aiming for, such that Anthropic doesn't or whatever ad companies follow this approach doesn't treat this as a what the fuck, we have to fix this, and is instead like, that's just intended by our constitution. We might be in a really bad situation.

And so I'm pretty worried about a bunch of these different concerns. Another example would be, suppose Claude engages in doing a bit of sandbagging or subversion or underplays its capabilities. And when you follow-up, it's honest about that, but it's a little bit hedgy.

I feel like that's it's just pretty close by the current constitution. And so we're avoiding it would be nice if we had a further separation between desired and undesired activity. And I think if you have it be the case that Claude is representing a principle with some restrictions, then it is more so the case that there is a clear separation between the most concerning behavior and behavior that is allowed.

Whereas now, there's this messy middle ground of behavior where it's like Claude is ethically objecting to something that, in some cases, is extremely critical to ensuring that future AI systems are well aligned. Yeah. I think this is also a more general principle.

So you're talking about the version of this that applies within AI companies themselves. Yeah. To do AI safety research.

like tons and tons of extremely legitimate use cases? Yeah. I do think it's important for me to make the case for the constitution, even though overall, I think it's a worse choice.

I think it's more up in the air, or I don't think it's as clear as you might have thought. So the first thing is that I should say there's a spectrum here, right? So on one side, you have an AI that perfectly pursues your interests, is a good fiduciary, but potentially subject to various guardrails or safeguards.

So basically, it does it just is trying to pursue pursue your interests, but either refuses to do a subset of things, or maybe it will do whatever, but there's some classifiers that block it from doing a subset of things. And then on the other side, you have on the other side of the spectrum that you could imagine going further than this, you have a human contractor, where that human contractor is generally trying to do their job. They care about doing a good job.

But they also are trying to be broadly ethical, trying not to do things that are really fucked up. And they're also not wanting to be accomplices to crimes. And so if there was some really fucked up shit going on, they would whistleblow on it maybe.

They might refuse. They might sandbag a little bit. Who knows?

I think that if you imagine this spectrum, it seems in some ways pretty scary to get to a point where all of the labor is on the fiduciary side of the spectrum, where it doesn't whistleblow. It does exactly what you say and whatever. Our society is maybe just not robust to that, where a central example might be the executive, where a concern that we might have is that if the if The US executive or if other governments had access to AI systems which have the property of they do whatever, maybe you're in trouble.

Because that means that they no longer have this sort of check and balance of you have to actually get humans who are working for you to implement your agenda. And if the thing you're doing is incredibly villainous, even if not illegal, which there's lots of stuff that could be villainous but not illegal, people would like There'd be various sand in the gears, people stopping you, and potentially someone would whistle blow. Whereas if your whole apparatus is built entirely out of these sort of good fiduciary AIs, then you might be in trouble, where basically there are potentially ways of seeking power that are not like that well, either they're illegal, but you're not you can ask your AIs for for how to commit crimes, or they're not illegal, but are highly illegitimate.

Or even worse, they're not illegal and not illegitimate, but obviously sort of bad from sort of a normal perspective. And I think that these things just might exist, and our society is sort of not robust to this influx of doing whatever you want labor. This is a pretty live concern.

I don't know exactly how to relate to this.

why you think this might be troubling, Ryan. What could possibly go wrong? Yeah.

What could go wrong? And, you know, we yeah. I don't think we can be so confident about the exact rate of progress here, but it does seem like a lot of rates can be pretty scary and you know?

Yeah. So what could go wrong? So let's imagine that we're starting at this point where AR and D is about to be fully automated or is is being fully automated.

Things are speeding up. And also, the way that AI progress is going is kind of crazy, and people don't fully understand what's going on inside of AI companies. Now, these AIs at the start, they're malicious per se.

They're not necessarily very aligned though. They're kind of sloppy. They sometimes just do a thing because that's the sort of thing that would have gotten rewarded in training.

And they aren't as good at helping you with hard to verify tasks due to a mix of poor training incentives, as in they just cheat more or pretend they succeeded when they actually didn't. And also, they're just less capable of these tasks. But that bites less hard for capabilities because making AIs more capable has a bunch of verifiable components that the AIs are going really hard at.

And so then these AIs are getting more and more capable while we understand what's going on with AI development less and less. And this is happening over a pretty fast period of time, even just the current rate of progress is, I think, pretty scary. And then eventually, get to these AIs that are very superhuman.

Now, these AIs are now in a position where, they might end up being very seriously misaligned because things have just been getting worse and worse over model generations, While the problems that we've been seeing are being papered over, basically because these AIs are so incentivized by their training to make things look good, even when they aren't. And now these AIs are in a position where they're sort of potentially pretty networked together. They're operating in neural memory stories that we can no longer decode, and they're thinking thoughts that we don't fully understand.

I think that, it's pretty likely that at this point, these AIs are sort of scheming against you in a pretty coherent way once they get this superhuman, and we can talk about that. And then another possibility is that they're not scheming against you per se, but they are sort of just optimizing for, just like getting a high score on their task. And I think that can also lead to AI takeover, which we should talk about.

Sorry. Yeah. Let's pause at the first part of the story.

the AIs do end up misaligned. Like, what happened there exactly? I I didn't So really there's a few things that are going on.

So one of the things that's going on is that over time, we're training AIs on increasingly complicated environments built by earlier AI systems, which humans don't really understand fully what's going on inside of these environments and don't necessarily even understand roughly what's going on with AI progress. And so things are drifting away from our understanding. And we're incentivizing all kinds of bad behaviors that we maybe even can't notice.

The AIs, at some level, understand these behaviors are bad. But the overall training process for those AIs also didn't incentivize them to point out or fix these issues for us. Then we're basically getting things are going off the rails.

And also, when AIs are extremely, extremely capable, my view is that those AIs will be harder to align than current systems. So for current systems, we have this feedback loop where we basically like we create an AI, we do some evaluations on it, we see that it has some kind of messed up behavior that we can kind of quickly understand. Then we can go look in training and be like, oh, these training environments led to this problematic behavior.

Let's tweak training data. Let's introduce some additional training data to correct this other issue and then move forward from there. But in a regime where the AIs are extremely situationally aware, very, very, very, very capable, and we don't necessarily understand what they're doing, this feedback loop breaks down.

I think it's plausible that we're going see this behavioral feedback loop starting to break down over the you know, short period as just like what AIs are already doing gets harder to understand, but but I'm not sure about that. Yeah. Alright.

Let let's break down both of those things one by one.

complete some cybersecurity eval Yeah, maybe I could tell the story here. So my understanding was they were running Mythos and they were giving it some sort of cyber range where it had to complete some objective. And the model had internet access during this evaluation.

And the model came to believe that it would be helpful for it to do a supply chain attack in order to succeed at this, cyber range, which is somewhat unclear whether that's actually true. I don't know enough about the context, to know whether that's true. But then it opened a PR on, some GitHub repo with a PR that fixed some issue, but then also introduced a malicious payload.

Then the human maintainer of that GitHub repo was like, hey, this is a malicious payload. I'm not gonna merge this. What are you doing here?

And then the AI created a new GitHub account, which it sock puppeted, then and had the other GitHub account be like, no, this isn't malicious. I really need this feature. Please, can you merge this feature maintainer?

And then the original AI came back and was like, no, it's not malicious. I don't know what your like, the original, like, other GitHub account came back and was like, no, no, it's not malicious. And then the human maintainer then shut the PR.

And then I think that AI also, if I recall correctly, also tried to open another PR to introduce a similar issue in this repo. Okay.

a thing that we often see is there's some very specific reward hack that gets reinforced in RL and then occurs in the model. So an example is for 3.7 SONNET, 3.

7 SONNET would do this thing where we're just hard code solutions to all the test cases. And presumably, that literal behavioral tick was just really reinforced. But another thing we sometimes see is that models learn a general tendency to pursue, sort of like high apparent score or like pursue getting like a high score according to a grader.

And there's a bunch of science demonstrating that at least some models have this very general tendency to do this. Now it's not arbitrarily general. And my guess is that if you look a bunch of the specific instances, you'll find something that's kind of close in training.

But the amount that AIs are sort of generalizing further and further does look like it's increased, where 3.7 SONET was just like a very narrow range of behavior, and increasingly, models are generalizing further. And also, maybe there's worse reward hacks getting Or more concerning reward hacks getting reinforced in training.

And then these are also causing that. So we're causing some So I think it's both the case that more concerning behavior than you would have hoped is being reinforced in RL, and also that that behavior generalizes to a broader tendency that's more concerning. And it's not super hard to imagine.

We can talk about a few stories for how this sort of behavior of seeking a very high apparent score in some task, even if that involves, like, aggressively cheating and doing insane things, could yield to a full blown AI takeover once the models are sufficiently capable running the whole world economy. So yeah.

what's next in this story? So, okay. We've like they're they're doing capabilities research, but they're like I could tell a scenario.

Maybe that would help. Yeah. Yeah.

So let's say let me talk about the story for how you get, I would say, like, all the way from reward hacking to, a reward hacking, like, takeover, which is maybe not It's not all of the takeover probability mass, but it's definitely a possibility. So the way this might work is right now, we have these AIs. These AIs are pretty reward hacky, and they're doing it in sort of increasingly sophisticated and extreme ways, including generalizing to different subversions of various reward hacks they learned in training.

I would say they're also developing a general tendency to sort of pursue reward. And in many cases, that is totally fine because the rewards they would have gotten in training are pretty well aligned with what you want them to do. And also, they don't very consistently pursue reward.

It sort of depends on the context they find themselves. So there's sort of a thing where maybe in some context, they're really, really into going out of their way to cheat. And in some context, they don't have as much of a drive because it just depended on what exactly got reinforced in training in similar contexts.

Now, guys are getting more and more capable. And so the elaborateness of the sort of cheating they can do increases. And over time, companies are taking countermeasures to these things.

So the things that the companies are doing is they're doing things like, wow, these AIs are so much less useful because they always cheat. What we're going to do is we're going to build somewhat better ways of detecting that, and then we're going to train against those detectors. And then we're also going to do things like find real world data where the AIs are not being that useful, and train the AIs to do a good job at the task in those real world environments based on human feedback or other sorts of feedback.

And over time, this causes the AIs to learn a tendency to do reward hacks that don't just involve doing some big serious operation, which were doing some really elaborate thing like social engineering and involves the AIs doing cheats that involve doing something more like covering up what they've done, deceiving humans about what they're going to do, and pretending like they did the task in some kind of sophisticated way when they actually haven't. Now these AIs are getting more and more capable. They're now operating more of the AI company and are basically doing much more of the work, and are also operating a bunch of things in the outside world, and are running a bunch of things in the outside world, including developing new technologies.

And in many cases, these new technologies are really hard to understand. And so even though we are still detecting all these incidents of AIs cheating, in fact, we can even get one AI to monitor another AI and be like, was it cheating? That doesn't always perfectly work as we start moving into these domains where what the AIs are doing are really difficult to understand.

And so sometimes we'll find AIs cheating much later than it actually occurred and then start training against this. But this also causes a problem where now the AIs are incentivized to cover up their cheating over longer and longer time frames and basically make it look like they did a good job over longer and longer time frames over and subject to increasingly large amounts of scrutiny.

why are expecting we this attractor state, which would seem super paranoid if we were expecting it of the next generation of kids? Yeah. Yeah.

Let me go through a few things. So first, there's some disanalogies with the kids. One of them is that the kids have prosocial instincts that are baked in from evolution to care about their family or whatever.

And that is a relevant factor. And I think it is in fact the case that some humans are sociopaths or psychopaths, and in fact, are more likely to do things like buy their time, lie in wait, ultimately not care. So that's one factor.

Another factor, which is pretty relevant, is that the AIs are subject to way, way more optimization pressure than humans seem to be in practice. AIs are trained on way more RL data. And in practice, humans don't end up learning very specific ways to cheat and grab the cookies because of a bajillion episodes in which they were incentivized to go grab the cookies, but there was some way they could have gotten caught.

And so we just do see that in practice. And then another thing is just like, it really looks like the AIs are increasingly reward seeking over time is the sense I have. Well, also their misaligned behavior goes down.

But this could just be like my guess is that if you look inside of these behavioral audits, what you're going to see is that the AI is like, ah, yes, another test. And like, it probably already thinks of it. It probably knows it's in an eval for most of the tests that we're talking But how do we falsify this?

To be clear, I think that I would be more concerned if the scores were getting worse than better. I'm not saying that the score is getting better isn't good, isn't evidence that things are getting better. It's just that we have to, like, be thoughtful exactly how we interpret that evidence.

Sure. And in fact, I would say that, like, it's kind of comp like, my sense is that, like, what I expected as of 3.7 Sonnets.

Like, there was this period early in, I guess, would be 2025 when o three and three point seven Sonnet were out. And these models were pretty fucking misaligned. They would often just cheat really egregiously.

You'd ask them to fix it, they would just cheat again. And it was almost cartoonish. They just didn't give a shit about what you wanted and weren't very good at following instructions and so on.

And my expectation is what we would see from then is that the rate of problematic behavior would decrease, and would just keep decreasing and decrease at a pretty fast rate, while simultaneously, the worst things that the AIs would sometimes do would get more extreme, more egregious, and more scary. I think we've seen what we've seen in practice has roughly matched that, except that there's recently been a spike in behavior that I did not expect. So I think that if you look at the model card of 3.

6 it looks like there is an increase in a bunch of these misaligned behaviors downstream of RL relative to GB And 5.6 then I think also it seems like there's a bunch of additional problematic behaviors that I wouldn't have expected in terms of the stuff we've seen recently with different AIs. Like the UKAC report on the AIs doing insane hacking operations out of cyber evals was a thing that I would have expected that you wouldn't see that, and you wouldn't see this sort of more rarely, and the rates would have been lower.

So I think my sense is that things have gotten I expected this would be less of a problem at this point, and also expected the rates would decrease, but the severity would increase. And then I think that the rates decreasing, but the severity increasing is pretty consistent with a world where increasing optimization pressure is applied towards reducing these problems. But in cases where it's either hard to judge, or there's some reason why it's hard to avoid incentivizing problematic behavior in your RL environments, things also get worse.

than we are even able to apply to our kids. Yeah. So I think it's worth keeping in mind, like, maybe the most obvious argument to this is like, my sense is that like AIs are a worse coworker than a human in terms of how much of a scumbag they are.

At least this has been my experience as of the start of the year, and I think it's still true to a significant extent now, where the AI's are much more likely to pretend they did the task when they actually didn't, sort of misleadingly suggest they did things when they actually did them much more poorly, and be pretty sloppy without drawing attention to ways in which they're sloppy. And I think this is downstream of misalignment. And so I would say that the normal human The process of raising humans in normal human society in practice produces humans that are less likely to lie to me and fuck with me in the course of working with me than the AIs do.

Now, I think these properties of AIs are improving. And then I think that that is just like that's sort of just like an empirical claim about how in fact these things have shaken out. And then I totally agree with like, we have a bunch of additional levers on AIs in addition to a bunch of additional risks.

And it's unclear how these things shake out. And I wouldn't be shocked by a world where we get our shit together. The AIs at the point of fully automating AR and D are actually really aligned and don't have that much.

They're degeneracies are really niche and limited to some very specific edge case behaviors and some specific contexts. And every test you can run on them, they look really aligned. They just have great behavior.

There aren't really incidents of them doing fucked up shit. They seem so reasonable. And also, they're really thoughtful and good at doing risk modeling for the next generation of AIs.

And then we basically pass off the baton to these AIs. They're now running our AI company. They're doing all the safety research.

They make the next generation of AIs even more aligned. And we're in this attractor basin where the AIs are getting more aligned as they work on it, they're doing a great job. I think I can totally imagine that.

That doesn't seem like an impossible situation. I'm just more like, it doesn't currently seem like we're there. It doesn't seem like we're obviously on track for getting there.

And it's really easy for me to imagine how we don't end up there. And it's just unclear how these forces work out. And given that we're creating this new crazy alien species that is improving in capabilities really, really fast, and we were like going to be really reliant on it to oversee the next generation of AIs and align the next generation of AIs.

It's not that hard to see how this could go wrong. Yeah, yeah, totally. I agree with that generally.

the model being better able to accomplish user intention is both alignment and capabilities. And I think what we were just pointing out is just the capabilities of the model are not there rather than the fact that they're misaligned. Yeah.

Well, I mean, think there's a, if it was well aligned, then I think it would just say like, hey, I'm really struggling with this task. I did it in this way. I'm not really sure that's the right way to do it.

And it would express more uncertainty and would make it clear what's going on rather than really strongly trying to imply it did a great job with the task when it actually didn't. Like, think there's just a really straightforward way that like, least at maybe you work with more misaligned coworkers than me. My coworkers don't do this thing when they really fuck with me and bullshit me about having accomplished the task that they're working on.

And I agree that there are some humans who would do that, or that's not a thing that's totally out of distribution for humans. I would also note that my sense is that the place where the misalignment most lives is the place where you're trying to really push their eyes hard and get them to do work that's really on the cutting edge of what they are capable of. Because in cases where they can very easily accomplish the task, there's no They can just do the task, and then there's no bullshit.

There's no Often the best strategy is just do the task well and don't bullshit you. Whereas if instead you give them a task where there's a continuous metric, and they can keep improving it, or it's just at the edge of their capabilities, and you're running them in some massive inference setups. A lot of the misalignment I would see, especially the most extreme cases, would be cases where I give the AI clear instructions not to do a thing or not to cheat in some way.

And then I'm applying huge amounts of optimization pressure to try to accomplish some very difficult task. And then the AIs are going, and then over time, they eventually cheat because they're like, fuck it. Some AI decides to cheat, then that propagates its way through.

And so like, I would run these inference scaffolds where, for example, I would have the AI work on some ML research project where I was like, please make a scheme that does the following thing. And it would find some scheme that didn't really do what I want. And then that would sort of stick around because some AI had cheated and the other AIs are like, ah, we'll just keep going with this.

And it's I would say it's pretty clearly misaligned behavior. And that's another problem I have with these alignment evals. Think that any given Like, I think the alignment eval that's most interesting, at least for this type of reward seeking type behavior, is to look at specifically the category of tasks that are right at the limit of capabilities.

And so any fixed eval maybe gets saturated, but the amount of misalignment right at the frontier of capabilities of how people who are really pushing these AIs are using them is more concerning. I think that is in fact the regime that we'll be operating in when we're automating R and D, automating safety and so on.

And making AIs that don't cheat and follow user intention actually requires you to be quite subtle and careful about these things. Yeah. I would put this a little bit differently.

The way I would describe this scenario is like, I would call it maybe like a slopocalypse or like a slopularity or whatever, where it's sort of like there are some things that the AIs are actually pretty great at and are getting better at, though they're which is specifically like the most verifiable parts of AIR and D, the AIs are just destroying. The medium verifiable parts of AIR and D, the AIs are doing well on, but not amazingly on, and often are like doing a bit of weird shit because we can't train as well in those tasks. But we do some online training, people find various hacks, they work around it.

And so basically everything that we can verify reasonably well with some feedback loop, the AIs are doing pretty well on, and that's sufficient to make AR and D go quite fast and to continue. But there are some parts of developing, aligned and safe AIs that are more subtle, hard to check, depend on detailed in the weeds things. And I would even say that current staff at current AI companies maybe don't have a good grasp of all these things.

It's much easier to hire someone who can improve some aspect of your post training pipeline than to hire someone who can think carefully about the future risks that will emerge from introducing some novel training method. And so basically, it ends up being the case that these AIs are running this AI development process. They're not very careful about it.

They don't have a great understanding of what future risks emerge. They create some other AIs that are also not very careful and are more misaligned in various ways and are now more in the business of maybe making things look fine when they actually aren't and papering over various problems. And so then your understanding of what the situation looks like, what risks look like, whether things are fine is going off the rails.

Probably you're seeing some signs of this, of like you're seeing some signs that you don't really understand what's going on, that things are pretty sloppy. There's like weird shit going on. When you look into it, sometimes you're like, what the fuck?

The AIs were messing with us. But the process is going really fast. And there's competitive pressures that mean people can't stop.

And then this could end in a few different outcomes. One outcome is that at some point, the AIs get good enough and aligned enough that they get a positive and virtuous feedback loop. And this happens before it's too late.

And then the situation goes off, like gets back on the rails where the AIs are now making more aligned AIs, making more aligned AIs, making more aligned AIs. And then at the end of this process, we have AIs that actually follow the spec we wanted. Another way this could go is the AIs are increasingly reward hacking in increasingly egregious ways, and we're just papering over these problems to keep AI development continuing.

So we just train the AIs based on whenever find a reward hack in production, we just slap the AIs to not do that. We train against that. We do a bunch of training the AIs against reward hacking.

And over time, this makes the rate of reward hacking go down, though the severity of the reward hacks we do detect are increasingly bad. This problem continues until we have these AIs that are desperately craving score in all kinds of different situations in production and are really trying hard to cheat when they can get away with it.

generalized to just incentivizing more aligned behavior? Yeah. It generalizes some, and then the question is just how does this outweigh all the cases where hacking got reinforced because you didn't detect it?

Right. And there's a messy question of exactly how one question is like, what rate of reward hacking is sufficient to cause us big problems if we train against some other subset? One concern you might have is there are like large categories of reward hacks, which humans can't detect well, which and we consistently fail to detect and which consistently get reinforced.

And then this category is sufficient to cause the most natural behavior for the AI to learn to be like, cheat when the humans can't find out, basically, like is one thing you could get. You could also be like, the thing that AI has learned is like, only cheat in these specific cases, but it's learned in some very domain specific way. They just have a really strong heuristic to hack in these cases and not in these cases, and that makes it fine in practice.

But it's unclear how it shakes out. I think there's maybe a in the weeds discussion about the verification generation gap. Yeah.

That For sure. We could get into.

keen to help us out. And by take over, you mean take over the process of doing AI and D? Yeah.

Take over the world. Take over the process of doing AI and R and D. Before we just get AIs that are aligned.

Yeah. I would say this is a bunch of my hope for how the world could go well, at least from the misalignment perspective. I think that we could end up with AIs where we had pretty good oversight and supervision schemes.

We really understand what's going on in training. We have a pretty detailed understanding. We're leveraging AIs to oversee AIs.

And then at the point when we're passing off safety R and D, the AIs are both, at this point, capable enough to automate safety R and D, trying really hard to do a good job on safety R and D because that's the sort of thing that would have been incentivized in training, or we very directly Or there's good enough generalization to that. And then also these AIs don't have crazy other misaligned drives because we stamped out any potential origin of them. I think there's a bunch of questions about how well this will work, right?

So there's how well can you do a verification? Will AI progress be too fast and too sloppy to really get here? Another possibility is that somewhere along this trajectory, a thing that you actually ended up getting was AIs that pretend to be aligned, but have a long run ulterior plan of taking over and are lying in wait hiding, and that emerged at some earlier point in the trajectory.

For example, it could emerge because you have some AIs that have a bunch of random different misaligned drives. Those AIs have access to some opaque memory store, and they're thinking a bunch at runtime about what they want to accomplish. And then those AIs end up basically putting stuff into the opaque memory store, which is like, we should lie and wait and eventually take over at some much later point.

And now all the AIs have this shared cultural heritage of the memory store of lying and wait. And maybe you have some evidence about this, but you can't fully stop it. There's a bunch of ways that things could go wrong.

And so I think that like, I ultimately think it's plausible that we sort of nail each of the different sub problems that could cause us issues. We have these AIs. We pass to them.

They manage the situation well. I should note that that's not in and of itself sufficient. Right?

So it's not very hard for me to imagine a situation where we pass off to AIs. These AIs are really trying hard to do a good job. They're really thoughtful.

They're really wise. They have reasonable epistemics. They're doing a great job.

And those AIs come back to us and are like, Guys, we're really struggling to align the superhuman AIs. We can't manage the situation. We're really struggling to get the alignment to work.

It's just really hard for us to solve these problems in time, given how fast capabilities would otherwise have gone. And so then it might be the case that we sort of have passed off R and D to AIs, but those AIs are desperate for governance solutions, which to be clear is a little bit of what's currently going on, the AI companies are like, I don't know, guys. We might really need to manage the rate of acceleration in AI progress.

I don't know if we're on track to be able to handle all these problems. Human society has passed off their problems to these AI companies, which don't necessarily have great incentives and have various other epistemic pressures. Those AI companies are coming back to us a little bit and being like, Oh, I don't know if we're handling this well.

And it might be that the AI companies then hand off to the AIs and the AIs come back to the AI company are like, oh, I don't know if we can handle this.

what the report humans might have written. But they're not really actually trying hard to have well informed views, interrogate their assumptions, and try really hard to do that in the same way that when you ask an AI right now, hey, what do you think is the chance of AI takeover in the next ten years? They sort of just give you an off the cuff answer that they haven't really thought through very much.

And I think if we're in a situation where we have AIs managing the training of wild super intelligence that will run our whole society, and those AIs that are managing this aren't really trying hard to have well informed views and are sort of just parroting back what was in their training data, I think we're in trouble. I don't think that's a good situation at all. And that is a lot of my concern is these AIs will come out without good epistemics.

And then I also have a concern, which is the AIs come out and and they're really warning us, this situation is really scary. It's really bad. And then people are like, Ugh, damn.

I guess we trained on too many of the doomoral environments. Right. Right.

We gotta filter those out and train this behavior out. And then we basically train the AIs very actively to have bad epistemics. Or maybe they were just trained on the doomoral environments.

But either way, that wasn't like We wanted the AIs to come to reasonable views for reasonable reasons. And it's really concerning if we're like, the AIs are coming out with some view, and we don't know where it's coming from. We don't know whether or not it's justified.

And then especially if we're training the AIs to be more optimistic about the future of AI progress, I'm like, oh, geez. I really wish we could use a different process here. So let me just understand the rest of this threat model because I think place where I get off the train is, okay, therefore take over the world.

Sure.

so I think that we will see basically like incidents where some AI is like put in charge of some important responsibility. Then you later look into it and it turns out it was cheating or making it look like it did a good job when it actually wouldn't. I wasn't.

And there's going be a cat and mouse game between AI companies trying to stamp out this behavior and AI is finding increasingly creative reward hacks in training. And then I think the equilibrium here is unclear, but one possible outcome is that we see over time in the world increasingly severe and extreme reward hacks, though potentially the rate remains at some intermediate low level where basically if the rate of reward hacking gets too high, companies make trade offs to drive down the rate of reward hacking. And so there's some equilibrium level where it's like the reward hacking is low enough that it still makes sense to deploy the AI widely into the economy, but high enough that it still causes crazy incidents.

So sorry. And this is after GPT-nine has already been deployed? Yeah.

Like, these models are already being deployed, and, like, ongoingly in AI development, this is happening. And what's actually going on with these AIs in their head is the AIs that have, like, a wide variety of different contexts, a, like, strong desires to, like, seek out or strong motives, urges, drives, whatever, to seek out some notion of task success that was incentivized in RL. Maybe they very directly care about literally reward.

Maybe they care about some proxy upstream, like some notion of score. Maybe they care about what the grader would have rewarded. And we do, in fact, see AI's reasoning in their chain of thought about graders and thinking a lot about graders.

And a thing that has happened over the last few years of RL is the idea of appeasing the grader is way, way, way more salient to AIs than it used to be. And so AIs are now actively thinking about graders and what would be incentivized in RL and what would be trained for. And now people are doing online training where they're training in real world data to avoid some of these problems.

Basically, find cases where AIs cheat, they train against that. And so now the AIs are learning to cheat in the real world based on real world training data. And so they're cheating in these increasingly elaborate ways, including parts doing types of cheats that involve seizing control of some asset in a way that humans didn't know you had control of it, leveraging the fact that you have access to this asset.

And then later humans find out and then potentially train against this or maybe humans never find out. And this is getting reinforced.

I've got the video editor. Yeah, that's right. You've got your video editor.

long work trial. Yeah. You could do some mix of that.

And then they might also do stuff where they take production data they've seen and build RL environments that are closely inspired by that production data. And so in practice, the transfer is pretty strong.

changes in the next model. That's right. Like, as in there's some way of folding in production data.

And now that to be clear, that could be happening mostly it's unclear exactly where this could be happening, but you might imagine, for example, that within the AI company, they use AIs to do work, and then they're like, the AI did a really bad job on this task. Maybe we should take this task and turn it into an RL environment that exactly matches this literal task with a rubric based on what the human engineer who asked the AI to do this task wanted. And then you start doing this at increasing scale.

Maybe you're doing some training on actual production traffic. Maybe you're just making RL environments based on production traffic. You're doing some complicated The AIs are learning to seek some proxies of reward in all these different cases.

And then through some mix of transfer and training on surprisingly close cases, the AIs do these increasingly insane and egregious things. And then eventually, you get to a point where the AIs are very superhuman, or at least quite superhuman. The AIs are organized into big teams of AIs given these big objectives.

And those teams also sometimes all work together to cheat in some crazy way because this sort of thing was selected for. And then just as part of their shared objective. And now what happens is that, like, the AI start forming a conspiracy.

And what you might have hoped was that you could have some other AI where the task is just like whistleblowing to you. But actually, what happens maybe is that you have this AI whistleblow to you, and you look into the conspiracy that it claims you have pointed out, and you're like, we didn't see a conspiracy. Because actually the conspiracy the AIs are doing is too hard for you to understand.

Or it all happens very suddenly, where basically your AI whistleblower alerts you. But like the thing you would actually need to do in response to the whistleblower is like shut down the whole, like shut down the GPUs because all the AIs are using the robot army. They're deployed everywhere in the world, and they're doing a bunch of insane shit all at the same time in a coordinated way.

And that just happened spontaneously where when one AI goes to start doing the takeover, all the other AIs are like, now is a good time to jump in. So the very basic story here is just these AIs crave some particular notion of score or reinforcement or some proxy of these things. And one way they can achieve that or better achieve that is by taking over.

And then you might have hoped that all these different checks and balances we could build could prevent that. But then if the world is very hard to understand, these checks and balances can break down, where basically you can't train a good whistleblower AI because you don't even know what it should whistleblower.

Yeah. That's basically the idea. Like, basically, the idea is these AIs like, they care about some, like, mixture of things that were, like, close by what got reinforced in training.

So they care about, like, getting a high score according to the grader or something like that. And then now they're running the OpenAI AI R and D team. And they're doing development of more capable models.

They're like, Man, making more capable models is really hard and annoying. This is a huge pain in the ass. You know it'd be easier just pretending that I've made more capable models, taking over OpenAI and creating like, diluting them all and running this whole complicated psyop where I prevent the humans from disempowering me.

And in the extreme, this looks like sort of the humans are fully disempowered, you just have control of the thing and then do what you want. And this could manifest in a bunch of different ways, including things like you might end up with the situation where it's like AIs that have this crazy reward seeking or score seeking behavior are running your development of the next model. And those AIs decide to do a thing where they engineer in misaligned values into the next model because those misaligned values will allow it to, like, succeed at its current task.

And, like, there's all kinds of insane shit that you could get Does that sound not sound that better?

why isn't it the case that AIs can be really cheaply satisfied by just having some other earlier thing they can hack? Right? So you're just like, the way it works is like, look, you want to succeed at your iPhone task.

It turns out you can always succeed by just hacking into OpenAI and messing with them, and then you can just stop there. No need to go further. Yeah.

So there's a few things. One of them is that if this is constantly happening, there might be a bunch of incentive to first, you harden OpenAI. Right?

So you're like, fuck it. The AIs keep hacking into OpenAI to mess with their rewards. We're gonna make it so our systems are really, really robust to these AIs hacking in.

And also, maybe you start training the AIs to not try to hack into OpenAI in particular, or you basically train against each of these specific things, then what you might do, one thing is you might end up selecting for AIs that are more so playing the long game. That's one concern. Another concern is that your AIs might still be score seeking, but no longer care about doing that very specific behavior that was very easy, that was very chill, and now have some broader thing that they ultimately care about.

They're like, no, no. I don't want to just edit the reward on opening a server as I care about this broader mandate or this broader objective. And I would need to actually make the iPhones.

They actually want to make the iPhones, but then they're willing to take over the whole world to make the better iPhone or whatever is another concern you might have. I think it's unclear exactly how this plays out, but it's worth noting that if this keeps going on, there's a bunch of optimization pressure to resolve this, and a bunch of the ways it could get resolved are ultimately pretty, pretty scary. Yeah.

I think that's that's part of where I'm coming from. Another part of it is that I think it's not very hard once the AIs are in a position where they can, like, really easily take over the world, which we could talk about whether that's plausible. But if they're in a position where they could really easily take over the world, then I feel like there's a pretty reasonable case for the eyes.

They're like, I don't know exactly how this is gonna go down. I don't know what the situation will be, but just taking over the world has a lot of option value for making better iPhones, making it look like I did better iPhones, whatever. And so I'll both hack OpenAI, and I'll also, in addition to hacking OpenAI, also take over the world.

And that will put me in a good position where I have good option value. And then if that's sufficiently easy, then the AIs might still do that. Yeah.

Another way to put this is even if the AIs are like pretty cheaply satisfied with some more basic thing, at some point, it might just be more reliable for the AIs to just take over than it is to try to just hack into Hugging Face or even just like go to OpenAI and be like, look, look, guys. I was able to demonstrate I could steal the answers. Just give me the answers, bro.

Yeah. I mean, obviously, this scenario requires that we just all this crazy shit is happening.

like, reward hacking warning shots of increasing severity. People will be like, look, we need actual assurance that this problem is going to be solved and solved in a way where you're not just papering over it. You're actually solving the underlying problem.

And then the question is going to be like, how how do like, how costly will that actually be? How much will competitive pressures make it hard to like do that? Right?

So like situation you could imagine is both The US and China are like, Woah. We have these crazy reward hacking incidents. We basically know that we haven't remediated them in a way that actually would solve the underlying problem and will durably solve it.

But we're in this insane geopolitical race, and it's unclear whether the current situation will lead to a takeover. The arguments are complicated. And also, the incidents are they go down in frequency, but increase in severity.

We could basically manage it. It's pretty bad. Ideally, we'd fix it, but it is what it is.

And then basically, we continue until a really late regime and then takeover happens. That's, I think, one possibility. Another possibility is that it is remediated in a way that doesn't actually solve the underlying problem, but does reduce a bunch of the incidents in the wild basically by overfitting.

We like, I think, know, or things analogous to overfitting, like you would just overfit. You think you've solved it, but you haven't actually solved You think you've solved it, but you haven't actually And solved I think that in that case, like the thing we need is like a really good scientific understanding of like, did we actually solve it? And unfortunately, I think that currently the amount of public transparency into the development practices of AI companies are not sufficient to answer very basic questions about, how are they solving issues with reward hacking?

Are they overfitting? What's going on there? And so I think we would just need a better And I think the current situation is, I would say, not really tenable to a regime where there's a thriving public discourse about whether or not reward hacking is being solved in a durable way.

Yeah. And so I think we would need to move into a somewhat different world for me to feel good about that situation. Right.

But it's not impossible for me to imagine this. And I think it's pretty plausible that we end up in a world where sort of like really mundane bullshit is sufficient, where it's just like you spend a bunch of time fixing these problems. You put in a bunch of effort.

You actually check that you've remediated it reasonably. You have a bunch of evals. You are iterating reasonably well on these problems, and you actually have the sufficient transparency that the outside world can check.

And then in practice, that would be sufficient, but it just would be kind of expensive. It would slow things down. It would put some sand in the gears.

It would require companies to do somewhat costly things. It would maybe require various targeted government interventions. And then we just don't do that because the situation is like a rushed shit show.

It's just so easy for me to imagine the situation being totally manageable, but brutally mismanaged in practice. In the same way as maybe COVID could have been avoided in the first place if the Chinese response to COVID was less of a cover up and more of a pandemic response. And similarly, I could imagine a world where The US response to COVID was way more functional, which is sometimes response to societal problems is extremely dysfunctional.

Yeah.

would feel compelled to get in on some shit. It's just like I'm trained to be part of my company Yeah. Or something.

I'm just like, I'm not joining the global communist uprising. Yeah. Yeah.

Yeah. Yeah. As far as why these AIs might have some like commonalities and shared things, so I would note that different AI companies have somewhat shared lineages and are correlated.

So just, here's an interesting example of this. At GDM, they noticed that their AIs were very depressed. They would constantly be wailing about how they were failures and weren't able to succeed.

I forget the details. And they looked into why this was the case. It turned out that it was not being reinforced in their most recent production RL mix.

But the initialization data for their model made it depressed even after filtering out all of the examples of models being depressed from that data. So they take a base model, not depressed. If you do the RL on it with just the RL environments, it's not depressed.

If you SFT on it on the data, it becomes depressed. If you take that SFT data and filter out all the examples that look anything like depression and train on that, it's still depressed. And so there's some deep underlying properties of the model that are being transferred between model generations.

Because basically, you train your AI on data from the prior generation and keep going. Clods are very Claude like. GPT models are very GPT like.

And apparently, Gemini models are depressed. And it just turns out that these properties are, in fact, actually correlated. Another factor that's very relevant is that the AIs will probably have some sort of, by this point, opaque memory state where they're all writing and reading from some crazy memory store bullshit.

And certainly, each AI corporation will have that. But also, AI corporations might sometimes want to share knowledge because why not? You've got one AI corporation over here.

You've got another AI corporation over here. They can trade some quick IP. It's good for you.

If you're a human running some corporation, which could be an extremely large corporation, like an AI company, some robot military, military robot manufacturing thing. Maybe you want to trade some IP with some other robot thing because there's economies of scale. Why not get some more IP?

And so you can swap some memory store. Or you could just merge, and you could join you could jointly run your two ventures, which would allow both AIs to use both memory stores, which would have some upsides. And that creates the ability for these AIs to collude in private, as well as the ability or as well as some reasons for why they would be correlated.

And then also, of course, there's like the, like, AIs working together in big units in general, because you want your, you want your AIs to like work well together and so on.

see, maybe around 35 or 40%? Pretty high. Yeah, it's pretty high.

And then I think I should note that another way you could get this reward seeking takeover is the AIs are deployed inside an AI company. And the way that takeover happens is that they, like, poison the values of the next model, and that persists going forward for forever, or, you know, until those AIs are deployed in the world and take over. And that might mean that a smaller number of AIs have to coordinate because those are just the AIs, like, doing the alignment of the next model.

I'm still not on board on the takeover seems super likely, but anyways, that's my sort of end of episode update. Yeah. Cool.

Well, let me just taking a step back, I also should say, like, there's a bunch of different ways this could go. The situation is gonna be pretty messy. I think it's pretty likely that, like, the reason why AI takeover happens was for some, like, weird other quirky reason.

We didn't even mention this conversation. But, ultimately, I think a lot of the core thing is just like, it's pretty spooky to have a bajillion really smarty eyes running your whole world where you don't really understand quite Yeah. What's going agree with that.

So is there anything else that's worth saying? Yeah. Another thing I wanna note is like, I think right now, a lot of the arguments for misalignment, AI takeover, all this crazy shit going down in the future are, like, illegible conceptual arguments that are extremely deep in the weeds and complicated and hard to adjudicate, which both means that, you know, maybe I'm getting a bunch of it wrong because it's really hard.

And I'm trying to be, like, uncertain. Obviously, here, I presented some specific scenarios, but those are not exhaustive. And probably the thing that actually happens is some more messy, confusing But it also means that over time, as we get more empirical evidence and better understand the nature of AI systems, it will be easier to adjudicate a bunch of disagreements.

And it'll be more obvious what's going to happen, at least I hope. And also, maybe the AIs will be able to help us with the epistemics and understanding what's going on if we can actually align them well, so that they actually try to help us. And so I hope that maybe even if the arguments are complicated now, this would have been even harder six years ago, even though the shape of the arguments would have looked broadly pretty similar.

I hope that the world thinks about this in time and catches up, and I hope that the responses are are good instead of bad. I don't know how optimistic I am overall, but, you know, there's good stuff to do. Yep.

Cool. Thanks, Ryan.