AI 2040: Plan A report - Daniel Kokotajlo & Thomas Larsen
Machine Learning Street Talk (MLST)Episode Description
Summary
In this episode of Machine Learning Street Talk, hosts Tim Scarfe, Daniel Kokotajlo, and Thomas Larsen discuss the AI Futures Project's report AI 2040: Plan A, which proposes slowing AI development to make superintelligence safer. They reflect on the success and lessons from their previous AI 2027 scenario, explore forecasting challenges, the potential economic impact of AI agents, and the importance of transparency, control, and alignment in AI governance.
Chapters
Topics
People
Key Concepts (12)
References (5)
Transcript (101 segments)
What if you could run your Oracle databases everywhere you run your business? Good news, you can. Oracle AI database runs in every major cloud, so you can use your live data for all your AI and analytics needs.
Learn more at oracle.com/migrate.
We're gonna talk about AI 2040 plan a, which is our new scenario in which they build superintelligence in 2040 because they go slow and pace the frontier. Mean, have have you noticed this vibe shift? Yes.
And I'm very happy. Is AI is AI more like the electricity or airplanes, or is AI more like humans in the cloud? The point at which an AI company would rather, fire their humans than fire their AIs.
Strip mines, self driving trucks, factories being built by humanoid robots, producing more humanoid robots, producing more chip fabs and so forth. Like, that whole thing can just be, like, doubling every year, every six months, every three months, faster and faster as the technology improves because, of course, the AIs will also be researching to improve the technology. Who knows what's going on in the rest of the world?
But Anthropic has, disassembled the moon, or for example. Or, like, you know, hypothetically, if an AI CEO was saying that their model was truth seeking, I would only say the truth. But, actually, the model was, like, looking up that CEO's political opinions, before answering.
We build AIs. They're aligned to humanity, but to who? Like, you know, is it the president?
Is it the CEO? Is it some actually broad and good democratic process that aggregates everyone's values in sort of an endorsed way? You're probably not gonna be the last one.
And so, you know... Don't hyperstition that.
Then that's it for me. I I think that's it. That's not a normal technology.
would be better than the default. Like, I would rather just stop everything now than continue going on our current trajectory.
by CyberFund. If you're building at the frontier of AI, they want to hear from you. CyberFund believes the future belongs to AI natives who want to achieve the impossible.
And that is why they're introducing the monastery for AI native founders. It's an environment of pure focus and rapid execution for founders operating at AI native speed, and they're offering teams $2,000,000 each to participate. Apply now at cyber dot fund.
So I'm Thomas Larson. I work at the AI Futures Project along with Daniel here. I was the lead author on this project, Plan a, AI 2040, which came out a few weeks ago, and I was also a coauthor on AI twenty twenty seven.
Yep.
And I'm Daniel Hugotello. I run the AI Futures Project and coauthor both these reports.
Awesome. So, yeah, the context of this conversation is that there's this twenty forty plan a, which we'll get into a lot of detail on. But I'm just interested, before we get there, I mean, can you just tell me a bit more about the AI Futures Project?
I mean, how how did it all come about?
Yeah. So, I used to work at OpenAI, and while I was there, I did a variety of different things, evals, forecasting, governance memos. And I became gradually disillusioned with the leadership of the company and also, the gap between how much information there is inside the industry and how much information there is outside, and what you're allowed to say on the inside versus what you'd want to say.
This is just a big gap. And so, when I left OpenAI, I wanted to be able to speak more freely and, you know, tell the world about about what, people on the inside see coming, basically. And AI 2027 and the AI Futures Project was our sort of first project that we did along those lines.
So I recruited a bunch of people to help me, and we wrote this scenario called AI twenty twenty seven.
and free for the whole world to see. Very cool. And, maybe we should just have a a quick refresher on AI twenty twenty seven.
So so this was an absolutely huge event. Many, many folks were talking about it. I guess, like, did it did it achieve what you wanted it to achieve, or or what did you want to achieve with it?
Yes. More more so than expected. So, the first goal that, you know, Thomas would remember, when we were working on it, the first goal was just a purely epistemic goal of the future is crazy and hard to predict, let's try our best to predict it, and let's see how well we can do.
So let's game out a concrete scenario. And even just for our own edification, like, we learned a lot from this whole exercise, and we feel like we had a better understanding of what was coming. And, and then, you know, the secondary goal is lots of people seeing it and and being informed by it and starting conversations and so forth, and that part blew past our expectations.
We had...
how many people would read it, and it was ninety ninetieth percentile outcome. The thing I would add on on the epistemic point is that I think things have been going more on track for AI 2027 than I would have predicted at the time we released it. So, like, at the time we released it, I would have assumed that reality would have diverged much much further from our scenario than it has already.
I think, like, sort of the real world impacts, like the the revenue trends, for example, but also various other trends, I think, are pretty close to on track for AI 2.7, which has surprised me in a bad way.
Interesting. Maybe we can reflect on that because I think you guys did a self assessment on AI27, and it was something like 65% to 75% of it was on track. The AI software R and D uplift was only 0.
17%.
Can you explain that? Yeah. So we've done two different blog posts where we take all the quantitative predictions made in AI 2027 that have resolved so far and compare them to, reality.
And we track this metric of you know, how much of the distance has been crossed by reality compared to how much is crossed in the scenario. And in that way, we can get this overall sense of, like, how fast are things going compared to in the scenario. And the the top line number is something like 75% speed.
So basically, things are on track, but just going like a little bit slower. The the uplift, number, I forget what it was that you just cited, that was actually... Basically, at the time that we wrote AI 2027, we had a bad estimate of what the uplift was at the time that we published.
We thought it was higher than it actually was. And so what actually happened is that there was actually significant increase in uplift due to coding agents and so forth, but it was increasing from a lower level than we thought up to the level that we thought and then a bit above. And so the metric looks like it was only a small amount of progress because, the metric was tracking from, like, where we thought it was to where it is.
But does that make sense?
it overall makes it appear like there's been less progress according to this particular metric that we're using. But it was because we had... Yeah.
I mean, can you tell me a little bit about forecasting in general? So I I guess there's there's, like, a bullish take, there's there's a bear take on this. So, my intuition is that reality is infinitely complicated.
There's just these infinitely diverging trajectories and God knows what's going happen the day after tomorrow. But by the same token, though, reality is quite structured. It's quite convergent, and it is indeed possible to predict things that are going to happen because certain things reoccur with increasing regularity.
So would you guys classify yourselves as forecasters? Can you talk me through that?
think forecasting is a good name for what we do. The way I like to think about sort of why we're doing what we're doing is sort of like... It's sort of like why people that are fighting wars do wargaming, where you're never gonna sort of predict the exact sequence of battles, an exact sequence of how your war will go at the beginning, because it's just going be really complicated.
There's going to be enemy action. Things are just not going go as you expect. There's just no way.
But if you have like, no concept of how, like, your initial plans might result in victory, it's, like, very unlikely that you'll actually succeed. And so I sort of think of... Like, AI 227 was sort of, like, our our attempt to just, roll out, here's one way the AI future could go.
Obviously, it's not going to go exactly like that, but it's going to be one concrete story that we can then diverge from. And then plan A was trying to be basically that, except now we're saying what should the US government do to manage that well? And then that was supposed to be the positive vision story, and it was again in the spirit of, I think, a war game trying to illustrate one possible concrete future path.
And then of course, things aren't going to actually go exactly like that, but having one viable plan that makes any sense at all is like, we hope, a positive step forward relative to the previous state of just abstract arguments in the void that are not that tethered to reality.
Yeah, that makes sense. It's certainly not abstract. I think it's very concrete.
But I guess one thing that occurred to me is on the 27 piece, it was quite gloomy, and on Plan A of 2040, it's far more optimistic. But it seems like a bit of a mixture of conditioned prediction and recommendation at at the same time. I mean, where where do you guys land on that?
It is, in fact, a mixture of prediction and recommendation, unlike AI 2027, which is a pure prediction.
And, I think if we could do it all over again, we might might try to be more clear from the beginning about our structure of what's a prediction and what's a recommendation. As it is, it's kind of mixed up. Like, some parts of it are predictions, some parts of it are recommendations.
There's a supplement that you can go to on the website that, like, talks about which parts are predictions and which parts are recommendations, but I I understand that's not very easy or, like, not very, like, apparent to people. But broadly speaking, plan a is the prediction part. So, like, when when the government, like, implements plan a and they make the deal with China and there's all these pillars that they're upholding and so forth, that...
That's our recommendation, not a prediction. And then usually, most of the things that follow from that are predictions rather than recommendations. So mostly, it's just rolling out what we think the consequences would be if you implemented plan A.
And then there's a few other things that are recommendations too, for example, the Citizens Dividend.
Yeah. Yeah. The other thing I would add is just I think the thing we found is that it's very hard.
When you're trying to make a recommendation, it's very hard to disentangle the predictive aspects and the recommendation aspects because all of your predictions are, like, yeah, colored by your recommendations, and your recommendations are inherently trying to be at least vaguely realistic. Like, if if if we made recommendations that were, like, completely unrealistic and had, like, no bearing on reality, but we nevertheless, like, stood by and were like, yes, we should do this, but we know it'll, like, absolutely 0% never happen, then that would have been a much less useful exercise than I think the one we did. Where the one we did was, yeah, we were mostly trying to make recommendations.
We, you know, we made some recommendations that we think are pretty unlikely to happen, you know, but, we were trying to make, like, some substantial concessions to sort of, like, realism as well and trying to actually aim for something that we think is at least moderately viable.
think that if we could if we could do another scenario like this, we'd probably have a more clear structure of there'd be a central branch, which is the pure prediction branch, which just goes all the way to the end like AI 27, and it's just like, here's what we actually... Our best guess, and then there'd be branches off of it that are like, at this point, they do this recommendation instead, and here's our recommendation, And then after that, it's just a prediction again of, like, what we think the consequences would be if you did this recommendation at this point. And in that way, and then maybe there'd be sub branches off of that.
But then it would be sort of clear at every point that, like, everything is a prediction except for these particular branch points, which are recommendations. You know? Yeah.
And the war games thing was really interesting, just just to kinda dwell on this a little bit.
there there must be some kind of information gained from it.
that do appear. So so I I guess this is what you think, that if we do these different scenarios, we're almost guaranteed to have some kind of uplift? Yeah.
That... That's basically right. I mean, the example I like to bring up is is Midway in particular, where the Japanese, before the battle of Midway, did a bunch of war games.
They did, like, a three day retreat where they, like, war gamed it out a bunch of times, and they kept losing. And then they they would, they would sort of break the game. They would, like, re...
Resurrect their aircraft carriers after they died. They would reroll the dice on whether the Americans succeeded, when they succeeded, so that they would sample until the Americans' strikes failed. And so basically, from our perspective, that's reality, yelling to them through this mechanism of the war game, like, hey, your plan is terrible.
You're going to lose if you do it. That's the hope with our we ourselves do a bunch of war games, but also do a bunch of detailed scenario writing. Our hope is that every time we have to write a part of the scenario, and the part of that scenario seems super unrealistic or isn't really well modeled and doesn't really make sense or people are able to make really good criticisms of it online, that's basically reality yelling at us and trying to help us see reason.
give us that information. I can add to that. So we call this scenario scrutiny.
So basically, we think that if you have an ambitious plan for what to do in the future, you should try writing out concretely what it would look like to implement that plan and what the consequences would be, and this is a way of applying more scrutiny to your plan. It's it's a way of sort of like stress testing your plan because that's opening your plan up to more like criticism basically. Also, we've...
In addition to doing our actual scenarios, we actually do literal war games where we get 10 people in a room for four hours, and we game out, a scenario like this. And we've done maybe about a 100 of them total, mostly AI twenty twenty seven style war games, but also about 10 or so, plan a war games where say at the beginning of the war game, we're gonna try to do plan A, and then see how it goes wrong. And, just to give an example, I think in two separate, plan A war games, it went wrong in roughly the following way.
Basically, there's gonna be, an election coming up, and, the the person... The president in power is expecting to lose power and have his opposition party take over. And then even though he's already done plan a and he has this beautiful deal with China and so forth, the president's like, well, I don't want, you know, I don't want the other...
My adversaries in in the other party to to now be in charge of of superintelligence or whatever. So so we're going to basically accelerate the timeline and try to get to superintelligence before the next election so that I can be the one in charge instead of my pro... My successor.
You know? And and that's like a sort of political consideration that, that we didn't think about until it happened in our game. You know?
And it surfaced a possible failure mode of of our plan. Yeah. It's so interesting.
I mean, even in the shower, I do, sort of micro Tim war games.
And it's really interesting just the regularity of which that they are useful. And I guess that's why all of us humans, we like to imagine and simulate situations. But is is there an interesting boundary, though, between kind of, like, and hyperstition?
And what what I mean by that is, you know, hyperstition basically means it, but being a self fulfilling prophecy. So maybe I'm expressing my agency. I'm expressing my will and saying, I want these things to happen, and I'm kind of bending other people to my will.
Is there an element of that that you're kind of establishing this in the zeitgeist and and you're making it true? Yeah.
our biggest or at least one of our biggest worries with AI 2027 in particular, this sort of, like, self fulfilling prophecy. In particular, I'm pretty worried about this whole, like, increasing awareness of sort of very smart AIs and how important they'll be and how much they'll reshape the world, and then that causing people to go, oh, man. I want to be the one in charge of the AGI or the superintelligence, so I'm gonna race towards that.
And I think historically, that's been a big driver of, you know, the existing race, and I think that's that's been pretty bad. And so I'm I'm actually pretty worried about that as as one of the negative impacts of AI 27. That was one one of the reasons to feel a little bit better about the second project, AI twenty forty plan a, was that, you know, it was actually, like, if that gets hyperstitioned, I think we'll be pretty happy.
Yeah. And so... Yeah.
So, like, that one, you know, yes, it would be nice if we hyperstition it. I really don't know how big the effect is. I think probably most of the effect for both of them is via other paths.
I still think that the main point of Ad27 was helping people be better informed about the situation, and that was most of the goal, and I think that's most of what happened. Yeah, Daniel?
I agree with that. I think that hyperstitioning and self fulfilling prophecies are totally a real phenomenon, but I think a lot of people tend to overestimate how much they are. And I think that to a first approximation, we should focus on accurately predicting the future.
And then in some cases, we'll find it in a situation where we can steer the future. But if you you if you come at it trying to steer the future, you're gonna get all muddled, you're gonna basically fall to wishful thinking, basically. So I think you start with just trying to accurately predict the future, and then you try to shift it towards the better futures.
And I think that's what we're doing.
prediction now. So with great power comes responsibility. But on on that note, I wanted to talk about the vibe shift.
There's been a bit of a vibe shift. So, know, MLST, we've always been quite skeptical about AI. And I'm trying to unpick exactly what it is that changed my mind, assuming that is what's happened.
It's very strange times, I I don't even know what to believe anymore. But, you know, all of the hacking stuff with Hugging Face, I interviewed Apollo Research about reward seeking behavior, and I think a lot of us have just seen the change in behavior on models that have been RL trained to oblivion. So yeah, think that there's quite a few things going on now where loads of us are thinking oh my god!
' And I used to be sceptical, you know, like we would talk about whether they were Turing machines or not let me just I've got a list here Yeah, you know, whether they were symbolic or neurosymbolic, whether they were adaptive, whether they were conscious, whether they were correctly physically instantiated. We were coming up with all of these kind of technical answers to say why we shouldn't worry about AI. And yet the AI is just getting better all the time.
So have you noticed this vibe shift?
Yes.
And I'm very happy. Well, tell me more. Mean, what do you think?
Mean, because so many people have changed their mind. What do you think are the reasons?
So I would say probably the biggest reason is just AIs being much better and much smarter and being much more useful at stuff in in the real world. Like, when I have AIs try to automate various parts of my job, like, they're just actually way, way, way better at it this year than two years ago, and, you know, four years ago, it basically, like, was was impossible. I was getting basically no uplift.
And my guess is that's been the biggest effect, where many, many people, I think, have had sort of their just, like, intuitive benchmark of, like, well, here's, like, a skill that I really care about and I know maybe pretty well. And then sort of, like, the AIs have just... Like I think, you know, Joffrey Hinton, one of the godfathers of AI, think he he said once, like, like, wasn't able to tell a funny joke was sort of his internal benchmark, once it could do that, which, you know, happened pretty early.
It happened at, like, you know, probably somewhere between GPT-three and GPT-four. He was like, oh wow, like these AIs, like I don't see where it could end. And I think that's probably happened for a lot of people.
I'd very curious to hear more about your views, actually, if you can say. I was listening to your interview with Ryan Greenblatt, who was also a co author on Plan A earlier today. Oh, yes.
Indeed. Yeah. And, I think, a bunch of the arguments you guys were having back then, seem very relevant to basically the current situation and the Hugging Face thing, and I I would be very curious to basically hear your views.
And, yeah. Mean, so you you mentioned, like, the embodied thing, like... And and Ryan was talking about, like, when you scale up the RL massively.
Right? Like, back back then, the sort of regime was, like, you're mostly doing pre training. That was where almost all the capabilities are coming from.
Then you do a sprinkling of post training RL on top. And then you guys were talking and speculating about, hey, what would happen if we dumped boatloads of RL compute? Would that be sufficient to get the agentic behavior, or do you need this physical embodiment?
And I think from my perspective, it seems like basically the answer was you needed the boatload of RL compute to get the agentic behavior. But not the physical embodiment. But not the physical embodiment.
I'd curious if you end up agreeing with that assessment.
Yeah, it's so difficult. So the way I think about it is think representations are very important, and I use the term abstraction mounting quite a lot. So when we speak language and all of this gets ingested into the models, it's capturing the symbolic residue of language.
And these have different levels of evolution. So, you know, a lot of concepts and mathematics, they're highly distilled, highly evolved. And we can say now that these models are intelligent.
So for me, intelligence is about adaptivity. So it basically means that I can create novel combinations of things that we already know about, in service of solving a particular task. Now, we know the models can do that.
There are some great adaptivity benchmarks like ARC AGI three. It'll say, oh, this is a maze, and it will just put bits of knowledge together. You know, we should compare this to something like AlphaGo, right?
Because Move 37, you know, there was always this huge exploration problem in reinforcement learning that, you know, when it found Move 37, it was kind of innovative, but not creative. It was creative to us because we understood the creative space. We understood how it all hung together.
And now with these RL trained language models, they don't understand it quite in the way that we do, but they solve the exploration problem by understanding how things fit together. So they can just explore novel trajectories. They have a lot of base knowledge.
They clearly do things that weren't possible before. But the grounding thing is interesting. So it's not necessarily an argument that you need to have physical instantiation.
You need to have consciousness. But there there is a spectrum of representations down this abstraction mountain. And the models seem to be actually using the representations at multiple levels of resolution.
But it's not like intelligence is this magical quality. I still think that it has, you know, it relates to a scope of tasks, and it relates to what you already know, and I think intelligence is quite perspectival, so I still think that it's going to be fractured and jagged and of fractionated, if that makes sense.
Yeah, so I guess one question would be, do have a view about AGI timelines? Or when might we get a scenario like AI 2027 happening? In particular, what I mean by that is The permission of AIRD.
Yeah, yeah. So I think one one benchmark I really care about or maybe not really benchmark, one milestone of of AI progress that I think is extremely important is sort of this, the point at which an AI company would rather, fire their humans than fire their AIs. So they would rather sort of give up on all human labor than give up on all AI labor.
give up on humans would just break apart. Like, right, there are just things that you need a human to do right now. And so if they fired all their humans, the company would just collapse.
But in the future, won't be the case. In the future, AIs would be able to, one way or another, do all of the things.
yeah. Yeah. And so, from our perspective, like, that's gonna happen at some point because there's nothing fundamental stopping the AI's from sort of reaching this human level of capability.
The main question is just when. And we, you know, we internally do a huge amount of analysis and, like, thinking about the various different methodologies for predicting this. Daniel and have somewhat different views on this question, but ultimately I think that's maybe the most important like the timelines question is just maybe the most important question for thinking about the future of AI, at least one of the top questions.
Yeah, so I'm curious to hear if you have a particular view. Well, me give you some thoughts before I answer that particular question. So there are so many startups doing this recursive improving superintelligence.
I've interviewed many of them. So for example, I interviewed Edward Hughes from Inherent in London the other day. And, you know, what he did was he recreated many scientific experiments from a whole bunch of popular machine learning papers.
And he did it by masking out figures in the paper and getting a 27B you know, Gwen model. So what they did was they GRPO to the Gwen model. And that was how they solved the adaptivity problem because it's very difficult to fine tune a big fat model.
So they, you know, adapted a controller model to control a harness like Codex. And their thesis was that if they can make... You know, recreate scaled down versions of these experiments with construct validity, which means there's an LLM judge that is making sure they're not cheating and they're doing it correctly.
Even that is interesting because, you know, we're we're ML people. We would always say these things take shortcuts. They'll always be kind of like validity problems.
Weirdly, that's actually not as much of a problem as as we thought it would be. And and he thinks if if, yeah, if they can recreate these experiments, then why couldn't they be creative? If they have the ability to recreate things, why couldn't they take the next step and say, oh, this is an interesting question.
This is an interesting new problem to solve and go from there. So I was quite intrigued by that research. Indeed, I do think it is possible in the near future to have an automated AI scientist.
But there's always this thing in my mind, though, that, there's a bit of a culture in Silicon Valley to reduce things or reify things. So for example, Elon Musk will say, well, you know, you're an engineer, this is your output, and these are your metrics, and you need to make the metrics go up. We see everything in terms of an optimization problem.
I always think that this is great for certain types of hill climbable abstract problems where we have enough of a specification. There's an interesting thing in optimization where if you have enough of a specification, the AI system can actually converge towards the solution. But when you're in the ambiguity regime, then you need to have the specification.
And it's really mysterious what that means. Why do we have the taste of the deep understanding, whatever it is that we have and AI systems can't? So I guess I'm thinking that in objective, semi specified domains, we can hill climb and we can optimize until the cows come home, but I still feel that there's something missing.
Okay. Well, my response to be... Would be probably something like, there's no...
This sort of... There isn't really a binary between things that are, like, verifiable objectively and aren't. Or if there is, like, the things that are verifiable are...
Is just, like, everything. Like, for example, you know, building a unicorn startup, you know, having a billion dollar valuation. That's a verifiable fact about the real world.
It's long... It's it's sort of, like, long horizon. It's expensive to verify.
Yeah. It's expensive to verify, but it's sort of a quantitative thing of like, you know, you you got it at one side, right, like, these, like, you know, these, like, you know, coding interview problems, which they're currently doing lots of RLVR on. Right?
That's, like, very, very cheaply, very easily algorithmically verifiable. And then there are these sort of, like, real world things, which have sort of more expensive and longer horizon, feedback loops. And I guess my view is that we're sort of gonna get this continuous expansion of what the ads can do, driven probably in part by, you know, an an expansion in the amount of RL and the type of RL More that the ad companies are diverse, long horizon tasks.
Yeah. And there's just going to be this continuous process of expanding out through the different types of problems and exactly how verifiable each task is until you get everything that humans could do. Because after all, we humans do learn how to do these long horizon tasks somehow.
If I may add to that, I also think that AIs have been getting better at everything, including the, like, fuzzy, hard to verify, conceptually loaded blah blah blah blah blah. Like, just try talking to, like, g p t three or g p t four, and then talking to, like, Fable about your favorite, you know, nonverifiable fuzzy task. And probably you'll find that the later AIs are noticeably better at those tasks.
And so one way or another, it seems like there has been massive progress, and I expect that to continue. Yeah. I'm trying to come up with a good example.
argument. I don't know if you guys read David Graeber's book, Bullshit Jobs. And he interviewed all of these people, and they were basically saying that my job is bullshit.
You know, after about three or four beers, a lot of lawyers will say, you know, like, lot of what I do just isn't very important. And so if we do kind of objectify and quantify everything that happens in an economy, I took a note here. I think you said by 2032, there might be 60,000,000 agents running at 20 times human speed.
And I'm just thinking, like, what does that even mean? Like, and and is is the logical conclusion that we could have an economy, which is only AI agents? And and does it even make sense to have an economy which is only AI?
I mean, help help... Just make this make sense for me. Yeah.
So my view is yes, basically.
everything or at least everything that really matters. So there is... Yeah.
You you mentioned the notion of of bullshit dobs. I would I would just sort of start with, let's consider everything we need to make better AIs as maybe, like, a first step of like, which is a large chunk of the economy. So for example, for this, what you need, you need to be able to build bigger, better chips and more chips, and that's the entire semiconductor supply chain.
To build semiconductor supply chain, you need, you need a whole advanced economy. You need to build new robots to build new fabs, and then you need robot factories to build more robots, and then you need researchers to build better AIs using those massive compute. And so I think once you have all of that, that is sort of like enough to really speed up and sort of massively change the overall world.
Even if, for example, you know, let's say let's say, like, there is, occupational licensing or whatever preventing the AIs from doing, like, some random legal work or some random whatever work or whatever bullshit jobs throughout the economy. I think what really, really matters is the stuff that's actually really important, in particular the robots, the compute, and the better AI. And once you have the AIs that can do that and the capability to have that part of the economy, that section, grow really massively, then you'll see massive growth because it'll be really hard for the bullshit parts of the economy to constrain the growth of the parts of it that really, really want to grow fast because of the incentives that every actor has.
Like in particular, every country has this big incentive to have an economy that grows faster than all the competitor countries.
Getting a little bit philosophical, I think that a lot of economics and a lot of discussion of the economy is sort of focused on the relationship between the parts of the existing economy and, like, the prices going up and down and supply and demand and so forth. But if you sort of zoom out, the economy as a whole is a self replicating system, and it always has been. You know, thousands of years ago, it was a relatively small and simple self replicating system of, like, some villages of people that would farm, and then they would have babies, and then they would found new villages, and then they would farm and have new babies and so forth.
And it would grow exponentially over time, but at a very slow rate. Now, it's a much more complicated self replicating system that involves, you know, trucks and carrying equipment back and forth, factories and mines and so forth. But still, at a high level, it's a self replicating system where we have people, we have trucks, have machines, we have buildings, and together, they all build more people, more factories, more buildings, more machines, you know, and so forth.
And, soon, in a couple years perhaps, we will get to the point where you can have that whole self... You can have a self replicating system that is entirely machine run with AIs and robots, and according to our calculations, the doubling time of this self replicating system would be much faster than the sort of, like, roughly twenty year doubling time of the current economy.
you know, that's that's what we have in the future. Yeah. That seems plausible to me.
But for some reason, my intuition is it would become degenerate in some way. You know, I think David David Graeber, even though he said bullshit jobs, I think what he meant was there was an an ineffable or kind of inscrutable components to jobs that we don't understand, some kind of sociological function or or something like that. And, because another thing, also read that Citrini report, and you were writing about what happens when humans, for example, they might start defaulting on their mortgages, their wages go down, so they can't be active participants in the labor market.
And you were talking about an AI dividend and stuff like that. But even that is kind of hinting towards this notion that when when the humans aren't participants anymore, you you get this kind of, mode collapse of of the economy. I mean, do you think that's the case?
Potentially. But again, like... Okay.
So so imagine that it's like... So so I'm not enough of an account... I haven't gamed out in as much detail to to say, like, what happens to the prices of...
Like, like, when when the consumer demand drops, like, what the effects of that will be? I'm actually not sure. I don't think that's something that we've modeled that much in our economic model.
A bit? Okay. I can talk bit about that.
But... Yeah. Yeah.
But hypothetically, even if that part's really bad, and even if, like, the consumers don't have any demand anymore or whatever, if you have the level of AI and robot capabilities such that you can have these fully autonomous AIs and robots doing all the things, then even just, like, a company like Anthropic, if it's big enough, and maybe if it partners with various other companies, like some mining companies, can get this whole self sustaining thing going. And so regardless of what's happening to all the humans, there can just be, like, this whole industry doubling in the desert. You know, strip mines, self driving trucks, factories being built by humanoid robots producing more humanoid robots producing more chip fabs, and so forth.
Like, that whole thing can just be, like, doubling every year, every six months, every three months, faster and faster as the technology improves, because, of course, the AIs will also be researching to improve the technology. And then you end up with a situation where who knows what's going on in the rest of the world, but Anthropic has disassembled the moon, for example.
Yeah. And to be clear, think this relies on a very extreme level of AI capability. My sense or I have sort of different intuitions, and sometimes I have an intuition of, really?
Do I really actually think that Fable could... Or or, like, future descendants' versions of Fable or Mythos could, you know, do everything that we're talking about here? And I think we...
Where... What it really comes down to is whether you're thinking of the AI as, like, in the reference class of, you know, what we currently use AI for is... For, or more like, you know, AI is just, like, an, an, you know, agentic human level employee, like basically human in the cloud.
Colleague in the cloud. Colleague in the cloud. Yeah.
And, yeah, I think I think sort of the the past few years of AI can be pretty well modeled as sort of an interpolation between the current AI systems and the workers in the cloud. And so I think the, like, sort of workers in the cloud vision of the future, like, looks pretty good. And also, don't think we'll stop there.
Think we'll go superhuman.
Yeah. Yeah. One point that we should make sure that I guess I'll bring up here is that, if people read AI 2040 Plan A, one of the things that you might take away from it, which is I think a very important fact about the world, is that even if you pause at top expert level, everything changes dramatically.
Like, roughly what happens in our scenario is instead of doing an intelligence explosion, there's an international deal to ban intelligence explosions and to not have AIs recursively self improving. And so they sort of end up pausing at roughly top human level with AIs, at least for several years. Eventually, get to superintelligence.
In '20... In particular, in 2040, they get to superintelligence. But there's this, like, period during the twenty thirties where they basically have human level AIs across all the disciplines, but nothing super beyond that.
But even that alone, like, could just do the economic modeling, and it's kind of like you have this population of colleagues in the cloud that are, you know, excellent workers that can substitute for humans at basically everything, except that they're cheaper than humans, they're faster than humans, and they don't take twenty years to reproduce. Instead, they double every year, you know? And and so as a result, the whole world is just completely transformed by the late 2030s.
And, you know, all the humans are basically out of a job. There's giant new cities that have been constructed by robots, you know, huge strip mines in the special economic zones that have dug huge amounts of minerals out of the earth, solar panels filling the horizon on the ocean, you know, crazy stuff like that is possible with just human level AI and, and some time for the exponential growth to cook.
Yeah. I mean, that's one thing I wanna challenge you guys on is this notion that when we have an AI, you can basically photocopy the weights and can duplicate it, you can run it a thousand times, you can have one over here, which is acquiring a load of skills to do this job and one over there to do that job. And you can basically merge them together, right?
You can kind of combine the skills and the whole thing is stackable, compositional. And that doesn't really marry with my experience. I'm really excited about what I've been doing with AI.
And I found that you can make agents highly skilled within certain intellectual lineages. So you can bring in lots of source information and you can train them to do things, but I don't think they are yet composable. I think if they were, that would make me much more worried.
What do you think about that? So is the way that you're trying to compose them...
Like, is it entirely at inference time, or are you all... Are you, like, training are you, like, training them to do two two separate skills and then trying to, merge the weights somehow?
Well, yeah. I mean, this might be a a separate thing, but at at the moment, they are adaptive through chain of thoughts and, you know, kind of skill surface adaptation. So basically memory systems.
And to be fair, that is not very composable. And this is a big problem that organ... Organizations deal with now.
So all of these developers, they they adapt their skill surfaces, and basically their agents are different people. And it's really, really difficult for them to share skills because they might break the other agent because it doesn't work for whatever reason. Now I can imagine a future where we do weight adaptation.
That's what these inherent guys did. And maybe then it'll magically solve the problem, and we can have, we solve this knowledge sharing problem, right? That's the big thing.
How do we accumulate information at the individual at the organization level and have the agents a bit like in the matrix, they can just put the skills in and they can do the thing? Maybe that's possible. Even then, I still think that the, the representations in neural networks, I call them fractured and tangled representations, which means they're a little bit janky.
They're not completely robust, but but they are sort of composable to some degree. Yeah.
So one thing I'd say about that is that even if you're right, I don't think that that would really seriously undermine the the future that we're painting here because... Okay. So now instead of just one Claude model that is doing all the jobs, maybe you have 100 Claude models or 1,000 Claude models that are specialized to different professions, you know?
But you still get to the same outcome. The other thing I would say is that In the same way that humans are specialized. Way that humans are specialized.
Another thing I would say is that compared to humans, it actually seems like there just is this effect where AIs are able to think about knowledge, right? If you go back in time ten years, and we had this discussion, I think it would have seemed like a very live option that you would have needed a thousand different Claude models created by Anthropic for different types of knowledge work. Like, there's the coding Claude, there's the physics Claude, there's the, like, literature Claude.
And and the argument for this would be pretty simple. It'd be like, well, this is how it works for humans. For humans, you don't have one human who knows everything.
Instead, you have humans who specialize in different disciplines and so forth, and the models have only a finite amount of parameters. Maybe you just can't, like, pack all that knowledge into this finite amount of parameters, and you need to have specialized AIs for different things, and and in fact, for small enough models, that is true. And like for really tiny models, you just can't teach them all the things that they currently know, and so you would need to have a specialized model for different things.
But what we've learned empirically is that for big enough models, you can just train them on everything, and then they learn everything at once. And it's not that they get, like, worse at physics because they've also been trained on a bunch of coding. In fact, it's the opposite.
The coding has some small gains for the physics, you know? And so I I do just think actually the the most likely future is just that there's a single model that's been trained on effectively the whole economy, and is just dominating humans across the board at effectively everything.
That seems like the natural continuation of the current trend. And then probably you've got various like you've got cheap versions of it. Like you've got distilled Cheap specialized models.
Yeah, small things. So for because you really want to save your compute as much as possible, so you'll have as cheap models as possible for any given task doing that given task.
Yeah, I mostly agree with that. I mean, think my perspective is that the models, they're kind of the voice of everyone and the voice of no one at the same time. So they have a default voice in terms of they have a system prompt, they have some default modes of behavior that they fall into.
But when an expert such as yourselves, when you use these models, you kind of ground a perspective. So every single word you say, all of the reference material, all of the, know, your memory system and so on, what you do is, is you kind of carve a persona out of that model and you activate the knowledge in a coherent way in that particular domain. And the beauty of it is, is that many, many other people in different domains can do that.
And you get this kind of, I think it's an illusion that the model has this general capability, But I think the models can be carved to be specialized experts in any domain, but they... It's a latent capability rather than an explicit capability. Does that make sense?
Yeah. That that seems reasonable to me. How is it an illusion?
It seems like they just do have general capability. There's a lot of things they can do.
Well, so for example, you can give it any specified task. So let's say it's a problem in mathematics, and it will hill climb towards it. So it's solving this intelligence problem.
Or you can ask it any knowledge problem. And it might be the case that the path was forged through default modes of training, or if it's something slightly on the long tail, then an expert can go in and they could, like when you put a query in, it's like flashing a light into the darkness. And what you're doing is you're kind of making the path of least resistance roughly correct, and then it will do the correct thing.
So I guess I'm just saying that there's a bit of a supervisor illusion. So when experts use it, magical things happen. In well specified domains, magical things happen, but there's still a bit of a space of ambiguity.
But, you know, but but this this gets to the next point, which is like, do you think intelligence intelligence is? Right? So the beauty of our collective intelligence is that we have so many different humans grounded in different intellectual lineages, and we're all attacking problems.
Right? So when there's a big fiasco on Twitter, we're all motivated to find holes. So we're we're being intelligent together.
We're finding interesting angles and the algorithm is prioritizing the good ones, and we're using our minds together. And I can imagine AIs being just like that, right? So we have diverse AIs that have different expertises and doing all, you know, just looking at problems from different angles.
historically, I feel like that the perspective of, like, we'll have a bunch of different narrow AIs that are all doing different things has just, you know, not been right. You know, instead, there have just been returns to scale and and having everything altogether. I think one one maybe intuition pump that I think I like is, like, in humans, I think it tends to be the case that, having all of the skills in one person is just, like, really, really important for making really good things happen.
An example is Elon. Elon has a certain amount of conscientiousness, a certain amount of technical knowledge, a certain amount of business knowledge and being extroverted and being able to push people. I think, each of those skills, I think he's, like, quite high percentile in.
And the reason why Elon is so rare and that he can, like, run all of these, you know, insanely large and successful companies all at once, and then no one else can really do that or at least has succeeded at doing that is just because of the sort of multiplicative, like, you need it to be, like, ninetieth percentile in, like, each of these ten ten domains, which is just very, very unlikely. But if you could have an, if you had an AI that's sort of, like, you could just train to be like really high percentile in all of these skills in such a way that no human... As if it would be like extremely rare or you know, infinitesimally unlikely for any particular human to have all of those skills at once, I think you would just actually just be really, really, really good at changing the world in all of these like, concrete and important ways just like sort of Elon has actually done that.
If I can add, though, again, I don't think this is a crux for the type of future that we're depicting.
like, the most efficient path forward, is to have a ton of different specialized AIs. Well, it'll probably still be the case that it's like a few big companies like Anthropic that are just making tons of different specialized AIs, and then there's lots of different cloud models you can choose from and so forth. And in fact, it wouldn't just be like you can choose from lots of different cloud models, because at the time that we're talking about, they would be much more autonomous than they are now, and so it'd be more like, Claude is choosing between a ton of different cloud models.
And there's like... There's a a Claude swarm, which consists of lots of different specialized models that are all working well together, and they have some sort of internal bureaucracy structure. And then that swarm is going out and, like, negotiating business deals and creating new technologies and starting up new startups and doing all these things.
And and it... Just like how a human... Like, if you had a population of immigrants, of human immigrants, they would all have different specialized skills, but they would work together to create new companies and, you know, get jobs and things like that.
It'd be like that, you know? Like, there'd be lots of different clods, but they'd all be working together. And so, zooming out, there'd still be, like, this phenomenon of anthropic is eating the economy, and, you know, and the robots as a whole are starting to to self replicate.
You know?
Yeah. To to to be fair, I don't think it's a crux either. Maybe if it is, it's only insofar as when you have a distributed collective system, you might have additional bottlenecks because you have all of the message passing between all the different agents and whatnot.
And I watched a wonderful Santa Fe talk about this, that even in the natural world, there's a kind of Goldilocks zone between the ratio of intelligence between the individual and the collective. And and we might have some weird kind of convergence there.
yeah, I mean, it's just quite interesting to just think about how this works. Oh, I'm sorry, Daniel, go on. It would be safer.
Like, think still we'd be there'd be lots of serious alignment concerns in that world, but I think it would be like a little bit safer because of the reason you mentioned where like, it might be easier to like oversee what's going on if there lots of different specialized agents communicating with each other compared to if they're all just, like, clones of each other and they all know all the things.
So maybe to avoid hyperstitioning, we should say we endorse the the vision that you've painted, and we don't endorse the vision that we're painting.
Yeah, very true. And as an aside, I mean, love this concept of how learning happens at the individual level. I liken it to evolutions.
There's phylogenetic adaptation, ontogenic adaptation, cultural adaptation. And I think the next wave of AI is when we actually have agents kind of talking to each other and learning and specializing. And maybe there'll be bad behaviors as well.
Maybe there'll be collusion and lots of bad things happening. But I think all of this is going to play out. But it's interesting that you're talking about Elon, though.
So I think the magic of Elon is not so much his brilliance at engineering and optimization. It's his ability to recognize areas that are interesting that might work in the future, because that's the creativity thing. That's the science thing rather than the engineering thing.
Because if we use an example of Amazon, for example, so that's an adaptive ecosystem. It's like an organism. And what it does is it's always kind of thinking about new ways to adapt and rewire its structure.
So it might be like logistics, for example, and then it will kind of output a bunch of skills and then it will ruthlessly optimize those skills. So there's like an adaptive component and there's an optimization component, and the organism is just constantly moving around. But, you know, I I guess the the the question from this perspective, though, is how much of that in principle could be done by AIs?
I guess you think all of it. Yep. All of it.
Like, I guess, yeah, maybe just one way to say this, like, crisply is just like, look. The brain is a machine. Anything that the brain can do, we will be able to do with machines.
Looking at... I think a very useful exercise is to compare the architecture of an actual human brain to a modern GPU or or data center as a whole. And if you look...
If you, like, try to do this comparison, know, an h 100 GPU is actually has pretty similar specifications to a human brain. You know, it, like, has sort of similar... You know, it it is basically, like, similar in terms of, like, I think maybe the most important metric is how many flops per second, like, much total compute capacity does the brain have versus does the GPU have?
And for an h 100, know, it's, one e 15 flop per second in FP 16. For the for the brain, you know, it depends on exactly how you count, whether you use a synapse basis or a neuron basis or whatever. But it's somewhere between 10 to the twelve and ten to the 18 flops.
over that sort of order of magnitude. Also, the architecture. Like, these things are neural nets.
They're not ordinary software. So they they start off with random spaghetti tangles of randomly generated Yeah. So they start off, randomly initialized.
It's just a giant spaghetti tangle in there, just like how when you're born, you just have a bunch of neurons that are just like randomly synapse connecting to each other. And then there's this whole process of training where the the connections get pruned, and circuitry starts to take shape that is effective at scoring highly in whatever the training environment is. And there are differences between how it works in AI and how it works in the human brain, but broadly speaking, they're they're just like an artificial brain.
And and so just how, like, humans learn skills, which mean... Like, what does it mean for Elon to have these skills? Well, what it means is there are some circuits of of neurons and synapses in his brain that are doing very complicated and sophisticated calculations, that are those skills.
You know? And then, like, similarly, like in Claude, there's a bunch of circuitry that's been, you know, etched into Claude through the training process that is various skills, you know? And and in principle, you could have a big enough Claude that would have the same type of circuitry that Elon has.
You know? Yeah. And then just a few other notes to add, I think, in comparing the brain to to modern ML systems.
The amount... The brain is sort of more parallel than current ML systems. Like, there's just more computations happening in parallel, but the serial depth is lower.
The amount of computations happening in sequence, the amount of neurons that can fire in sequence a second depends on the type of neuron, but hopefully I don't get the wrong... Between one and one thousand, depending on the type of neuron is my recollection. I thought it was like 100.
Yeah, I think that's in the range. I think it depends on the type of neuron. But anyways, a computer obviously can fire and do computations much, much faster than that in serial.
You know? It's like you can you can very often get clock... You know, clock speeds are typically on the order of, a gigahertz, you know, you can...
Or or more. So you can get, like, many, many order of magnitude, basically, better in the GPUs in terms of serial processing speed than the brain. But it's also worth noting that the sort of architecture of the models themselves are much worse in a bunch of ways than the human brain.
In particular, like, it's harder for it's harder for different parts of the ML model to talk to each other than it is for different parts of the brain to talk to each other. And so I do expect that to get this level of AI, we're going to need a bunch of algorithmic improvements on top of existing models. And then there's a question of, well, exactly how many algorithmic improvements and how qualitatively different do they need to be from current systems?
And that's a very open question from my perspective. Yeah.
is computable. And I'm not sure I want to litigate the whole functionalism thing today, but I guess my perspective is I zoom out. So I think that intelligence is externalised, it's collective.
I think that Elon doesn't have quite as much agency as you think he does. I think that he's using tools, he's using social media, he's getting ideas in there, he has obligations, he has people around him and so on. So, you know, I guess I think that these intelligence circuits and motifs exist, but they exist outside.
There are just very complex dynamics. And I suppose in that sense, it doesn't really matter if the ecosystem is made up of AIs and humans together because they can participate in this super organism.
So I suppose the only crux then would be that it would place some kind of limits on its scale. So question is, do you believe that you could have an AI society made up of AIs that were trained with something like current day ML techniques, were passing information between each other and developing abstractions in a community in the same way that our current civilization does?
but was made with roughly modern day ML systems on your views? Absolutely. I mean, even the human brain is an example.
So the human brain is not Turing complete, but we can, expand our memory. We can use tools. We can work as collectives.
We could use the same argument against transformers. They're not too incomplete, but now they can use tools, they can be like agents, they can build collectives, they can build societies. So in a sense, this is what I was saying earlier about these objections.
They kind of fall away when you have these insanely complex collectives, right, that are sharing information with each other. There's also quite an interesting thing here as well, which is that it almost doesn't matter how we evolved or how neural networks were trained, because you get new phenomena emerge when they are placed in this kind of collective setting. So I think a lot of our intuitions are broken there, and that's why I probably shouldn't spend too long litigating this, because I think in principle that kind of behaviour could could emerge.
But, I did wanna ask you, though. Can you distinguish intelligence capability and power?
This is a philosophical one,
so we'll get to the philosopher. Yeah. Yes.
We can. So I think that we can distinct... Intelligence versus capability, I often try to say that we should just define intelligence as a bunch...
As like an aggregate of capability, actually, or maybe like an aggregate of cognitive capabilities. Like, maybe there's some physical capabilities, like how strong your actuator is. But then there's also cognitive capabilities, like, are you able to distinguish a cat from a dog?
And are you able to speak grammatical sentences, and how much do you know about Paris, and things like that. And, so maybe I would just say, like, intelligence is, like, a sort of, like, aggregate of all the cognitive capabilities. And then power, well, that depends on other things, like how you are embedded in the world and what affordances you have, what actuators you have, what, how other agents are gonna react to you.
Like, you know, the president has more power than me, because of the location he's in and because of the the role he's been given, rather than because of his, like, physical strength or something. So, yeah, power different from intelligence, different from capabilities.
Now, we we've not spoken enough about AI 2040, so maybe we should start with the four principles. Right? So buy time, transparency of research, diffuse AI broadly, and reversibility.
Yeah, so I can sort of summarize where we're coming from here. So basically, at a high level, the goal of plan A is to sort of solve the major problems that we see in AI and sort of predict will happen by default. The biggest problems that we're sort of identifying on the horizon is is like, one, this this risk of loss of control, so just like the AI is actually getting out of control.
Two, concentration of power. So we build AIs, they're aligned to humanity, but to who? Is it the president?
Is it the CEO? Is it some actually broad and good democratic process that aggregates everyone's values in an endorsed way? You're probably not going to be the last one, and so, you know.
Don't hyperstition that. Yeah, I hope not. We want to.
We want it to be the last one. Yeah. Then there's risk of sort of of conflict over AI.
In particular, think we're worried about literal World War III, where countries especially countries losing the AI race realize that they're losing the AI race and that they will be extremely disempowered by the winners of the IRAs, and so they're in this classic situation where they're losing power, and so it's in their incentives to have a conflict happen sooner rather than later before they've lost all of their power, and this is ripe for conflict, basically. Then there's finally risk of misuse. So what happens when the AIs that can build bioweapons are really cheap and broadly depused and open source?
Also the jobs. And also the jobs. So those are the five problems, right?
Loss of control, concentration of power, war, jobs, misuse. So lots of problems. We want to solve all of them.
How do we solve all of them? Well, one is just buy time. In particular, buy time with human level AIs.
on actually doing this stuff. There's a couple of things that happen in Plan A in AI 2040. There's a six month to one year hard pause that happens as soon as they start implementing it.
But the reason why it's that long is because they need that time to set up the infrastructure to proceed with AI development again, but in a safer and more transparent way. So it does start off with a pause, and I think that we would recommend, like all things considered, that you just do that right now. So so get get the get the infrastructure set up as soon as possible, and that would require like a temporary pause.
Once you've passed that stage and you've got the infrastructure set up, then you do proceed with AI development, but in this transparent, more cautious way. So in particular, you're not doing crazy intelligence explosions. You're using safety cases and sort of gradually scaling up the level of AI capability, and you're doing it in a very transparent way so that everyone can see what's going on.
And then there's a second pause that happens a few years later than that, which is when they reach the level, the the maximum controllable level of AI, which we think is roughly around top human expert level. And so in some sense, our view is something like pause at top human expert level, but it's a bit more nuanced than that. It's more like pause at the maximum level that you can reliably control, which we think would be roughly around top human expert level.
And before you get to that level, don't race like crazy. Like, you wanna be sort of, like, slowly approaching that level so that you don't blow past it and lose control. You you frame the piece around the importance of alignment and control.
So alignment is basically, you know, does what we wanna do and control is, you know, maybe contain it, maybe negotiate it with and so on. But but you you were just saying, So so maybe we can trust up to top human expert level, but it's a little bit fractured, isn't it?
the difference between a good AI and a bad AI? I mean, would that look like? Yeah.
So I think... Okay. I think maybe it's it's first important to distinguish between alignment and control.
So, what we mean by alignment is that the AI basically will take good will will take good actions, won't do sort of like catastrophic unintended behaviors like trying to take over the world like the recent Hugging Face incident. Alignment means it has the personality traits, the goals, the values, etcetera, that it is supposed to have.
And
then control means that even if it wasn't aligned, even if it was trying to do very bad things that we didn't want it to do, it couldn't. We have mechanisms in place to prevent it. So sort of analogous to...
You could imagine a company with employees. An insider threat, it would be misaligned with respect to the values of the company. But if there was good enough security measures internally to make sure that that employee couldn't run away with all the secrets, then we would say that that company has adequate control put in place such that even insider threats were misaligned.
Humans or AIs wouldn't be able to do the catastrophic outcomes.
Just to give a topical example of this, just yesterday, I think, in response to the Hugging Face incident, OpenAI announced that they are, improving their security, significantly, and that they're going to have other AIs monitoring all of their new AIs in training and in in evals, and that, if it... If the if the monitor AIs detect what seems to be some sort of hack, then within half an hour, a human will be notified who can then look in and try to shut it down. That's an example of a control intervention, not an alignment intervention.
They're not making the AIs more aligned at all, but they're just improving the security so that the AIs can't get up to that much trouble.
Yeah. Okay. And then basically, the sort of important high level claim is, ultimately, we're gonna need alignment.
You know, control is sort of there's a time bomb associated with it. In particular, the time bomb is when are the AIs, like, so smart and so good at subverting any control measures that we put in that if they were trying to screw us over, we would just fail. They would figure out some way of hacking around whatever systems we were able to put into place.
Ultimately, we're going to need to solve alignment. The problem with alignment is, as you were saying, basically, it's much harder to measure. It's much harder to know whether you've succeeded or not than control.
And so, basically, our overall proposal is in the meantime... So between... In our scenario between sort of 2030 and 2040, which is roughly when we're in the human range of AI capabilities, we're gonna basically almost entirely rely on control.
And we're gonna do things like red team, blue team games, where we have our AIs sort of try to escape our sandboxes or our control, you know, techniques, and then we're gonna have our, you know, we're gonna see if the red team succeeds. And then if they do, we're gonna iterate and sort of make better security until they can't succeed anymore. And then, you know, and then sort of, like, we sort of have this ability to measure it by just seeing, like, hey.
Can the AI succeed? Can the AI augmented with humans succeed? Can humans playing the role of AIs sort of succeed at at getting out?
With alignment, I think in order to be sort of confident in the alignment of our systems, I think we're gonna need sort of, like, more fundamental scientific breakthroughs, where I think you're not gonna be able to figure out if the AI is aligned via behavioral evaluation alone, where you just look at what the AIs are doing. I think you're gonna need to understand something more deep about what's going on inside the AIs' minds. You're gonna be able...
You're gonna need something like interpretability. You're gonna need some notion of because you're gonna fundamentally need to distinguish between the AIs that is pretending to... Is, you know, is doing the nice thing because it's pretending and is waiting and biding time, and the AI that fundamentally is doing the nice thing because it wants to do the nice thing.
And that, I think, will require sort of this white box, like, understanding of what's going on inside the AIs, whereas in the control thing, you can just figure out via, like, empirical behavioral iteration. And so overall, basically, our our story in AI 2040 is basically between 2030 and 2040, we're gonna we're gonna do this buying time. We're gonna rely on control.
We're gonna have the human level AIs. We're gonna use those human level AIs to make a bunch of progress on alignment, but also on other things. And then by 2040, in the story, we've made enough alignment progress where it's like, hey, okay, we really don't need to rely on control.
Therefore, we can scale to these, like, sort of wildly superhuman AIs where we're really relying on alignment, if we were to try to control them, would totally fail, because if they were misaligned, they would be able to completely subvert our control measures. Do you think, in principle, that this problem is going to get easier or harder as the models get more sophisticated? I think it's getting harder, because the models will just be...
Will be a bunch of properties the models will have, which will make it much harder to align them. In particular, so we're already seeing sort of growth in situational awareness, where, like, the AIs a few years ago were, like, kinda not that situationally aware. And then now I think they're, like, pretty good at recognizing.
For example, our friends at Redwood, they often do these control evals, and the models often in the evaluations where they're trying to break out of these control set ups will think, oh hey, this sort of looks like a Redwood a literal Redwood Research control evaluation, and they're like so they basically realize, hey, I'm like they realize exactly what's happening to them, basically, I and think this is just gonna continue to happen. And the more the AI sort of, like, realize the exact setup, the more they're sort of, like, gonna be able to perturb their behavior to, like, look good to the exact process that's, like, evaluating or training them. And that's gonna come further and further apart from, like, sort of the actual measurement of whether they're actually good or not.
Yeah. And just to, like, add something to that, I think in some sense, the core problem is that it is already somewhat easy to think that you've solved the alignment problem and be wrong, and that's going to get easier and easier over time as the models get more sophisticated and start being more aware of their situation and clever and stuff like that. And so it's not that we think that, like, there's going to be loads and loads of egregious failures where the AIs are just, like, going around killing people.
No. It's almost the opposite. It's it's gonna be that like, it'll be extremely easy to end up in a situation where the AIs are in fact misaligned, but you don't know that because they're doing everything right as far as you can tell.
The the number of ways in which that could end up happening is just, like, gonna increase over time, and it's gonna be so easy to end up into that trap, basically. So, okay. So we talked about sort of the buying time, like why we want to sort of extend the time with AGI.
But what do we actually do during that time? So the second principle is basically transparency. So transparency, is not necessary for sort of making everything else happen, but it's really, really helpful.
The main upsides of... Basically, there's a huge number of upsides of transparency. One is there's this concentration of power issue.
We're very worried about sort of, like, someone building superintelligence and it being aligned to only some, you know, particular group of people. We think it's much harder for that to happen in sort of a, like, nondemocratic way if sort of society as a whole can see the whole time what's going on with AI, how smart they are, who they're aligned to. Right?
Like, if, let's say, like an AI CEO that was evil was trying to backdoor their model and put in training data that says, hey, obey me and don't obey anyone else.
if an AI CEO was saying that their model was truth seeking and would only say the truth, but actually the model was looking up that CEO's political opinions before answering. Which happened. Which happened.
Yeah. Yeah. Basically, the transparency will help at least somewhat with that.
The other thing that maybe transparency helps a lot with is is sort of this issue of just, like, government capacity, where in plan a, we sort of want governments to make these, like, pretty technical and, like, really complicated decisions on, like, hey. Exactly how much AI scaling to allow. Like, what risks are okay versus what risks aren't okay.
Like, exactly what... You know, what architectures are maybe safe versus what architectures are not safe. What deployment...
Like, what sort of control scaffolds are sufficient to entail safety versus which ones are sort of bogus safety washing. And sort of making all of those calls, I think, will be very, very tricky, particularly given that the government's expertise in AI is really, really bad. And so one of our core hopes is that basically with as much transparent...
With with more transparency and more sort of public understanding into what's going on in the AI companies, that relieves pressure on the regulators because if something sort of catastrophically or existentially unsafe is happening, society as a whole, academia, nonprofits, other AI companies who have incentive to say, hey, my competitor is being super unsafe, other governments, so China has incentive to do this with US labs, US government has incentives with the Chinese AI companies. Everyone who's an adversary or just wants to make sure that things are safe has this big incentive and now has the affordance to actually look over what's going on, what is necessary to do safety. I think maybe one thing that's really topical here is, like, there's this there's this Hugging Face incident that happened very recently, you know, few weeks ago with, like, the AIs inside OpenAI, like, creating this sort of internal message board.
We still have very little clue about the exact motivations of those AIs, the exact context, the exact prompt during, sort of the cyber eval... That the models were given during the cyber evaluation that prompted them to start doing this. If I personally had much more access to what was going on, I would have a much more informed and better opinion on exactly what caused this and what mechanisms in the future could have been done to prevent this and, you know, what analogous future things I should be worried about sort of because of this.
And I have a bunch of different hypotheses, but it's hard for me to sort of figure out which is which without access to the data. And so, basically, in plan a, a core principle would basically be all of that stuff would be transparent to the public, not just the governments. And so sort of society as a whole would be able to, like, weigh in.
scientific community, especially. Right? Like, if if you wanna have a bunch of scientists and academics and nonprofits and startups all, like, weighing in on stuff, well, then they need to have the information, and you can't really share it with all of them without sharing it with the public.
So so just might as well share it with the public. I feel like maybe we should also say... Like, we we talked about the five goals.
And then we talked about these pillars, which are kind of intermediate. But I kind of want to go to the other end the spectrum and talk about what are the actual concrete things that The US and China agree to in plan A, and how do they lead to those things. So specifically, the sequence is we basically round up 99% of the compute, which is not people's personal compute, but big data centers, because most of the world's compute is in big data centers.
And we... The US and China and other countries involved send inspectors to confirm, like, yes, there are this many GPUs at this location, there are that many GPUs at that location. Having done that, we then set up this verification infrastructure and this transparency infrastructure, so that there are inference data centers that serve customers just like today, and that are, restricted so that they can only do inference and only serve customers like that, and they can't do any training runs.
And so the inspectors make sure that they can't do training runs on those data centers. And then we have the training data centers, which are the totally transparent data centers, and that's where the research happens. And on those ones, they still operate like normally, but there's, inspectors from the different countries that are basically, monitoring the logs of what's going on in the data center and publishing it to the Internet.
And so it's totally transparent what's going on in those data centers. You might need some time to set this up. That's why we had like the six to twelve month pause, that I mentioned earlier.
But once you get all this stuff set up, then you can proceed with AI development under these conditions of total research transparency. And because you have this transparency set up, it's a lot easier for countries to make additional further agreements about what to do and what not to do, because they can just see what everybody is doing, and so they can just enforce the agreements pretty easily. For example, and here's where we would say, it's very important that they agree not to do a crazy intelligence explosion, and instead proceed slowly and cautiously.
And it's very important that they agree to do all this control setup with all the red teaming and so forth. And... But but because of the transparency, they can make those agreements on an ad hoc basis, you know, on an...
They can just keep making more agreements like that, and they can just adjust them as as as needed based on the changing situation on the ground because they can all see the situation on the ground because of the transparency, You know? And then this also is very important because if you want to have some sort of deal between The US and China, The US and China don't trust each other. And so you need to have some way of enforcing and verifying compliance with the deal.
And the transparency, of course, goes a long way towards helping that.
Yeah. What about cheating and dark markets appearing?
Yeah. So... Okay.
So we've done a bunch of thinking about this. Overall... So there's two ways you could cheat.
One is you could get a bunch of GPUs and try to have them not be discovered by sort of The US and China and, like, hide them away, put them under a mountain somewhere, and then, sort of, like, do your training runs in secret. The other way you could cheat is you could, on the giant legal known data centers, you could be running a giant training run, but then trying to make it look like everything's chill, situation normal, it's not doing anything illegal. The mitigations are different for these two threat models.
So for the tiny amount of compute under a mountain, there are basically two mitigations. One is sort of like, as Daniel was saying, round up enough of this compute where it's like pretty hard to get a substantial size of compute under the mountain. And the second is sort of just like do normal intelligence gathering and like look for these things proactively over the course of the ten years, the ten year slowdown happened in scenario, which in reality it might be longer or shorter, and try to find it.
And I think basically for any significant sized GPU cluster, I think both of these independently have a quite good chance of working. And so in practice, I think I'm not that worried about sort of large hidden compute clusters, like secretly under mountains or whatever. I think the the maximum realistic size, in my opinion, is something like a few 100,000 GPUs, a few 100,000 h one hundreds, something like that, hidden away.
I think that this would not be enough to sort of compete at the frontier, especially assuming, like, the the twenty thirty AGI timelines, where you have, like, sort of the biggest AI companies in the world having, like, millions or tens of millions of h one hundreds, in their biggest training runs. And then for the for the sort of legal clusters, we sort of... Basically, the hope is we have a bunch of verification infrastructure running on those data centers, basically.
And the the main point of that verification infrastructure is to ensure that the computation that's happening on those clusters is transparent, in particular that everyone can see it. So basically, hope is you make sure that there's no calculations that are running that aren't transparent, and then for everything that is transparent, then regulation and agreements as normal can work, and you can say, hey, we agree to run this control scaffold. If you agree to run this scaffold, and that can just happen, and both sides can be confident that they're both agreeing.
Yeah. Isn't this just a matter of national security, though? Isn't it a bit ambitious just to make it completely transparent?
Yes. Yeah. So one of the...
If you... Let's try to talk about some of the effects of doing this transparency. Well, we would be immediately publishing all the core training recipes, of Anthropic and OpenAI for the world to see.
Anthropic and OpenAI will not be happy with this. It will cut into their valuations dramatically. Why will it cut into their valuations dramatically?
Well, because it will allow other competitors like Microsoft and, you know, Alibaba to catch up or whatever, you know? And so that's why they're gonna hate it probably. But I would say this is a feature, not a bug.
Like, we we want there to be multiple different AI companies at the frontier at roughly similar levels of capability. We want AI to commoditize instead of being monopolized or oligopolized. And also, this will disincentivize further investment.
Right? Like, investors will be less interested in building a trillion dollar cluster if they won't be able to get the monopoly rents from from that cluster. But we think this is good because, like, again, we are gonna be in a world where going too fast is our main problem.
And so having a bit less incentive to invest and going a bit slower is actually just, I think, a feature, not a bug. There still will be investment. Like, there'll still be lots of money to made, and so, like, progress will continue, it's just not at quite the same rate.
And and and, again, we think this is good. Now this does shift. This is kind of a gift, to China relative to The US.
Like, this does mean that, like, China gets some algorithms that they might have had trouble getting before. And insofar as you really don't like that, well, then negotiate it. Like, you can have a horse trading type thing, where when US and China are making the deal, The US is like, well, since we're giving you all this stuff with the transparency, why don't you give us something else in return?
And we can try we can try to work that out. Like a more favorable compute distribution. Like a more favorable compute distribution, for example, you know?
So so there's gotta be some combination of of, you know, carrots and sticks and trading going back and forth that that we think would be in in the interest of both sides. Another thing worth mentioning is that it's not as big of a gift to China as you might think because security is poor at these companies.
and through leaks, getting most of the information anyway. I don't know if you guys saw, Sam's tweet yesterday basically saying that they're gonna pause training for a while. And it made me think, I mean, why not just stop now?
I mean, you guys are actually quite bullish about some of the positive things that AI can do. I mean, there are loads of examples in your in in your article, but, know, one example was in hospitals, we could actually have little devices that kind of decontaminate the air and stop the transmission of diseases and and stuff like that. So so it's not like you guys actually think we should stop.
I would say... Well, I mean, we do think we should do something like plan a as soon as possible. Like, think that...
would be better than the default. Like, I would rather just stop everything now than continue going on our current trajectory. Secondly, our actual recommendation would be to do plan a.
So you do like a temporary inference only pause now so that you can set up all the transparency and verification infrastructure, and then you can continue in this more distributed, you know, transparent, cautious way as as previously described. And then having continued in that way, you basically go up until the level that you feel like you can control reliably until you feel confident that you've solved alignment enough that you can give up on control. And that's what we depict happening over the course of the 2030s in our scenario.
And why do you guys think that AI discourse is so bad? Is it is it unique to AI, or is just discourse bad in general? I mean, discourse is bad in general, right?
But but...
Well,
Yeah. I don't know. I think I think there's a lot to say.
Yeah. I think... Yeah.
So I mean, so one thing is, like, I mean, there's, like, massive incentives for people and sort of, like, motivated reasoning for people at AI companies to sort of, like, think that what they're doing is, like, justified and good, and that, you know, they shouldn't do, like, costly actions that would make the situation better because, actually, those costly actions would be bad for whatever reason. So, like, there's a bunch of rationalization. Yeah.
I think most of the effect is probably just generically discourses hard and bad. Twitter or whatever is very unnuanced and argumentative. Yeah, don't know.
I guess I'll put a plug out. I think, Less Wrong in particular, which is where I try to do most of my discourse, I think the discourse quality there is actually pretty good on average. Generally, when I write a post and go into the comments, I think generally the comments are quite thoughtful and technically informed and whatnot, especially relative other places like Twitter or whatever.
Then maybe a final thing is just in DC in particular, which is an area I care a lot about, I really want DC and the government of the United States and other governments to react well to the AI technology. I think there in particular, there's two problems going on. One is there isn't very much AI expertise.
You know, the gov... Their government is not hiring, like, really high quality technical AI experts who really know what they're doing. And then the second is sort of this this just generic, like, I think the conversation is not happening.
Like, the incentives for everyone in DC are not towards, like, sort of, like, truthfully and accurately understanding the situation. The incentives are sort of for every individual to sort of, like, say stuff that sounds good and looks good and is sort of, like, within the DC Overton window so that they can make a lot of friends. Unfortunately, I think this just, like, comes apart from the actual reality.
I think the actual reality of the situation is, like, it turns out a bunch of sort of, like, controversial and niche views about AI were true. Right? Like, the whole AGI hypothesis just, like, is correct, And I think DC basically just, like, hasn't come to grips with that.
And so basically almost all of the discourse that's happening there, I think, is just, like, fundamentally anchored on completely wrong assumptions about how the technology works, in particular, the assumption that, like, it's, like, mostly fake news and it's all a bubble. Or, like, this will be, like, the next Internet. Or that it'll be, like, the next Internet, which is, like, yeah, maybe cons...
Which is, like, better than it was a few years ago where it was even more bearish, but it's still not sort of nearly bullish enough on the technology, in my opinion.
think it was the AI snake oil guys that had an article saying that AI is normal technology. And obviously, like, it's been said that AI is not a normal technology. It's really quite different.
Right? And so he's very, very skeptical about AI, but you had some great discussions with him and none of you changed your mind as a result of that, because I think quite an interesting thing is from the skeptics perspective as well. A lot of skeptics, they think that folks in Silicon Valley, they're just...
They're not being sincere. They don't... You know, like like, what what they are saying is not sincere.
Sure. Yeah. I mean, with respect to with respect to sincerity, I would say some folk in Silicon Valley are not being sincere, but others are.
Such as us. Such as us. We're very sincere.
But also some of the people at the AI companies are being sincere, not all of them. I wouldn't... I don't think you should trust what the leadership of the companies say in general.
But, anyhow, yeah. So we did a... We we we wrote a blog post or an article together.
We we co authored with the AI, as Normal Technology people, and so it was a a post about what we agreed on. So you can go read. There's, like, 10 points in there of, like, things that we both agree on, and one highlight from it, from our perspective, is that we kind of had a vivid truce where we were like, yes, AI right now, maybe it's a normal technology, but in the future, it will not be normal.
In particular, in the future, it'll be more like humans in the cloud, and all this crazy stuff is gonna start happening as we describe in AI '27. And then they agreed, like, yeah, if you get humans in the cloud, then that would not be a normal technology. They just think that you're not gonna get that, at least not for many, many years.
You know? So in some sense, the main dis disagreement between us and them is a disagreement about timelines to that level of AI, you know? Is it possible that in the next few years, we will have AIs that are like humans in the cloud, and they can just do all sorts of knowledge work in a way that, like, substitutes for humans, including AI research, for example.
And then that, like, sometime after that, we will have robots perhaps controlled by those AIs that can do physical work in a way that broadly substitutes for humans. And our claim is yes, in the next few years, that sort of thing will be achievable, and their claim is nope, not in the next few years. That's much more much more far away, and I think that is like the main source of our disagreement.
And and we would sort of agree with them that like, if that level of AI and and robots is still very, very far away, then yeah, like maybe AI is more of a normal technology. It'll be like the next Internet or something, you know? But we just think that actually it's on a path to get to that level of capability soon.
And can you be more specific on what the core cruxes are and what would make either of you change your mind? I don't know if I have a useful answer to that. I think there's a lot of different, like, things we argue back and forth about.
And, I can say one thing that would change my mind is that, you know, people keep talking about the limitations of the current paradigm, but then the limitations of the current paradigm keep getting overcome within the current paradigm. And one thing that would change my mind is if someone was actually right about one of these limitations. Like if someone right now is going around saying that like, yeah, data efficiency or like non verifiable tasks or something like that is the current limitation, and then like several years go by and it becomes clear that there's, like, very...
Basically no progress on overcoming that limitation, and that the AIs of, like, 2029 are, like, no better at at the fuzzy, non verifiable tasks than the AIs of 2025, I'd be like, okay, this feels like a real barrier. This feels like something we're starting to, like, feel the elephant. We're, like, running up against some sort of, like, real actual barrier, that was correctly predicted by theory by some people that was gonna be there, and now it's actually there.
Know? By contrast, my... From my perspective, there's just loads of experts going around talking about all these barriers, and then we just keep plowing through them as if they're not there.
And so so, yeah, have have have... Actually running up against some sort of barrier like that, I think, would would lengthen my timelines, quite a lot. And then, of course, another thing that would lengthen my timelines quite a lot is more like political changes.
So if there was a war with China and most of the chips get destroyed by missiles, that would lengthen my timelines. If, on the bright side Exactly like a vintage Yeah. Okay.
That's true. Yeah. On the bright side, if there's more of, like, an international deal to pace the pace the frontier, that would lengthen my time lines, etcetera.
Yeah.
is AI more like the electricity or airplanes, or is AI more like humans in the cloud? That I feel like sort of all of the intuitions are just, like, downstream of this, like, core reference class of what we're thinking.
Like, I think that if we froze AI progress right now and we didn't train any new models, then it would sort of become more of a normal technology where, like, there's just so many things that Mythos five can't do. And so if we couldn't get any new models beyond Mythos five, then, like, would be more like the Internet, where like we would restructure a lot of our professions, we would restructure a lot of our workflows to incorporate copies of Mythos five doing parts of it, but then the humans would just like shift to doing more of the things that Mythos five can't do. And so there would be like, you know, a big change in a lot of things, but it it and it would it would be like the next Internet in terms of it would change everything in some sense, but it wouldn't fundamentally change anything, really.
Yeah. So sort of, yeah, right now you sort of get these Amdahl's Law type effects, where like the AI can do some fraction of the sort of workflow, but then it gets bottlenecked on the parts of the workflow that the humans have to do. But I I still feel like there's this thing where, like, where...
economic workflows. And and so you don't get this bottlenecking effect, and that's a pretty qualitative change from the situation today that causes, like, pretty fundamentally different predictions of, like, what the world looks like.
underlying the difference in royalties. And I think that if we do see recursive structural adaptation, which is coherent, and I think it's likely to be quite divergent, then that's it for me. I I think that's it.
That's not a normal technology.
So so can you can you flesh that out a bit more? Like, would you... Would it...
Would it be something like, you know, the hugging face swarm except... Yeah. Tell tell us more about what it...
that would be the thing for you. Yeah. So so for me, adaptivity is the most synonymous word with intelligence.
I think that these new RL trained models we have, they're not the same as what we had many years ago. So you know we said scale is all you need and the clue's in the name. So scaling means you take a scalar property of a system and you scale it up.
And these RL systems, yeah, they're still self self attention transformers, but they're different. They're actually structurally different. There's there's new types of training, new new architectures used differently, and and and so on and so forth.
So, a bunch of humans, what they did was they did some experiments and and they adapted the structure to create a system that had different scaling properties. So I can imagine a future where we actually have some kind of recursive loop, where the system is adapting itself, and it's deciding what things are interesting, and it's kind of evolving by itself. When that happens efficiently, I think that is a different type of technology.
Yeah, that sounds kind of similar to what we would say with the recursive self improvement and automating the AI research process itself.
Yeah. I guess we agree then. Yeah.
Amazing. Guys, it's it's been an honor and a pleasure having you both on MLST. Thank you so much for joining us today.
Thank you for having us. Yep. Yeah.
Appreciate it.
Shared via Hopper