Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken

Welcome back. Happy to be here. Yeah.

It's fun. What's changed since last year? We talked basically this month in 2024.

Yep. Now we're 2025. What's happened?

Okay. So I think the biggest thing that's changed is RL and language models has finally worked. And this is manifested in we finally have proof of an algorithm that can give us expert human reliability and performance given the right feedback loop.

And so I think this is only really being conclusively demonstrated in competitive programming and math, basically. And so if you think of these two axes, one is, the, like, intellectual complexity of the task, and the other is the time horizon of which the task is, is being completed on. And I think we have proof that we can we can reach the peaks of intellectual complexity, along along many dimensions.

We haven't yet demonstrated, like, long running agentic Mhmm. Performance. And you're seeing, like, the first stumbling steps of that now and should see much more conclusive evidence of that basically by the end of the year Mhmm.

With, like, real software engineering agents doing real work. I think, Trenton, you're, like, experimenting with this at the moment. Right?

Yeah. Absolutely. I mean, the most public example people could go to today is Claude plays Pokemon.

Right.

Would you have said that agents would be only this powerful as of last year? I think this is roughly on track for where I expected with software engineering. I think I expected them to be a little bit better at computer use.

Yeah. But I understand all the reasons for why that is, and I think that's, like, well on track to be solved. It's just like a sort of temporary lapse.

probably not what's limiting them. I think what we're seeing now is closer to lack of context, lack of ability to, like, do complex, like, very multifile changes and, like, sort of, like maybe, like, scope or or of the change or scope of, like, the task in some respects. Like, they can they can cope with high intellectual complexity in, like, a focused context with a high with a real, like, scoped problem.

But when something's a bit more amorphous or requires a lot of discovery and iteration with the environment, this kind stuff, they're they struggle more. Yep. And and so maybe the the way I would define it now is the thing that's holding them back is if you can give it a good feedback loop for the thing that you want it to do, then it's good.

It's pretty good at it. If you can't, then they struggle a bit.

maybe, like, broadly, the domain is a little like RL from verifiable rewards or something like this where a clean reward signal. So so, you know, the initial unhappling of language models was r often human feedback Mhmm. Where, you know, typically, it was something like pairwise feedback or something like this, and and the outputs of the models became closer and closer to things that humans wanted.

Yeah. But this doesn't necessarily improve their performance at any, like like, difficulty of problem domain. Right?

Particularly, these humans are actually quite bad judges of what a what a better answer is. Humans have things like length biases and and and so forth. So you need a signal of whether the model was correct in its output Yeah.

That is that that is, like, quite true, let's say. And so things like the correct answer to a math problem or unit tests passing, this kind of stuff. These are the examples of of reward signal that's very clean, but even these can be hacked, by the way.

Like even unit tests, the models find ways around it to like hack in particular values and hard code values of unit tests if they can figure out like what the actual test is doing. Like if they can like look at the cached Python files and find what the actual test is, they'll they'll try and hack their way around it. So these aren't perfect, but they're they're much closer.

And why has it gotten so much better at software engineering than everything else? In part because software engineering is very verifiable. Like, it's a domain which just naturally lends it to this way.

I think Does the code pass a test? Does even run? Does it compile?

Yeah. Does it compile? Does it pass the test?

You know, you can go on the code and you can run, like, tests and, like, you know whether or not you got the right answer. But there isn't the same kind of thing for, like, writing a great essay. That requires like, the question of, like, taste in that regard is quite hard.

Like, we discussed the other night at dinner the Pulitzer Prize, like, you know, which would come first? Like, a Pulitzer Prize winning novel or, like, you know, a Nobel Prize or something like this. Yeah.

And I actually think a Nobel Prize is more likely than a Pulitzer Prize winning novel in some respects. There's a lot of the the tasks required in in winning a Nobel Prize or at least, like, strongly assisting in helping the win to win a Nobel Prize have more, like, layers of verifiability built up. So I I expect them to, like, accelerate the process of disc of doing Nobel Prize winning work more initially than that of, like, writing Pulitzer Prize worthy novels.

putting the blinders on them? Right. Like carving away the marbles on this.

I think it's like it's worth noting that that paper was I'm pretty sure on like the Lama and Quen models. And I'm not sure how much like IRL compute they used, but I don't think it was anywhere comparable to the amount of compute that was used in the in the base models. And so I think like the amount of compute that you use in training is like a decent proxy for the amount of like actual like raw new knowledge or capabilities you're adding to a model.

So like my prior at least, if you look at like all of DeepMind's research from RL before, RL was able to teach these like Go and chess playing agents Yeah. New knowledge that were in excess of human level performance just from RL signal provided the RL signal is efficiently clean. Yeah.

So there's like nothing structurally limiting about the algorithm here that like prevents it from imbuing the neural net with new knowledge. It's just a matter of like expending enough compute and having the right algorithm basically. Mhmm.

Why aren't you already spending more compute on this? I think Darius said in his blog post that labs or it was like a couple months ago on the export controls thing is like, deep seek, whatever. They're we're only spending 1,000,000 on RL or something.

So it's like, we aren't in the compute limited regime for RL yet, but we will be soon. Yeah. You're spending hundreds of millions on the base model.

Why only order a million on the RL? You know that the parable about, like, when you choose to launch a space mission? How, like, you should, like, sort of acquire, like, go further up the tech tree because if you launch later on, you're like, your ship will go faster and this kind of stuff.

I think it's quite similar to that. Like you wanna be sure that you're you algorithmically got the right thing. And then when you bet and you do the large compute spend on the run, then like it'll actually pay off without the right compute efficiencies and this kind of stuff.

Yeah. And I think like RL is slightly different to pre training in this regard where RL can be a more iterative thing that you're progressively adding capabilities to the base model. Pre training has, you know, in many respects, like if you're halfway through a run and you've messed it up, then like you've you've really like messed it up.

But I think that's what that's like the main reason why is people are still figuring out exactly what they wanted to do. I mean, o one to o three, right, like, opening, I put in their blog post that it was a 10x compute multiplier over o one. Yeah.

So like, clearly they, you know, bet on, you know, one level of compute and they were like, okay, this seems good. Let's actually release it. Let's get it out there.

And then they spent the next few months, like, you know, increasing the amount of compute that they expend on that. And I expect, as everyone is, everyone else is, like, scaling up RL right now. Mhmm.

So I I basically don't expect that to be true for for a lot. Yeah.

at, like, at some Passeke, like, you you've got token space. Right. Exactly.

Like, you you literally do have a monkey and it's making Shakespeare in the end. Yeah. Yeah.

Exactly. Yeah. Okay.

So the alpha the the the chess analogy is interesting. So were you about to say something? I was just gonna say, like, you do need to be able to get reward sometimes in order to learn.

And that's like the complexity in some respects. In like the alpha variants or maybe maybe you're about to say this. Yeah.

Like, one player always wins. So you always get a reward signal one way or the other. But in the kinds of things we're talking about, need to actually succeed at your task sometimes.

Mhmm. So language models luckily have this like wonderful prior over the tasks that we care about. Yeah.

And so you so if you look at all the old papers from like 2017 it's not that old, but like, you know, like the papers from 2017, the reward the learning curves always look like like flat flat flat flat flat as they're like figuring out sort of like basic mechanics of the world. And then there's this like spike up as they learn to exploit like easy Yeah. Yeah.

Rewards. And then it like sort it's like it's almost like a sigmoid in some in some respects. And then like sort of continues on indefinitely as it like just learns to like absolutely maximize the game.

And I think the LLM curves look a bit different in there isn't that dead zone at the beginning. Yeah. Interesting.

Because they already know how to solve some of the basic tasks. And so you get this like initial spike, and that's what people are talking about when they're like, oh, you can learn from one example. That one example is just like teaching you like to pull out the backtracking and like formatting your answer correctly and this kind of stuff that lets you get some reward initially at tasks, conditional on your pre training knowledge.

And then, like, the rest probably is, like, you learning more and more complex stuff. Yeah. Yeah.

And it would also be interesting.

it's an efficiency question there. Like, obviously, if you could give a dense reward for every token, right, like if you had a supervised example, then that's one of the best things you could have. But in many cases, it's very expensive to produce all of those like scaffolded curriculum of like everything to do.

Like having PhD math students grade students is something like, which you can only afford for the select category of students that you've chosen to focus in on developing. And you couldn't do that for all the language models in the world. First step is obviously, that would be better, but you're gonna like be sort of optimizing this Preo frontier of like how much am I willing to spend on like the scaffolding versus how much am I willing to spend on pure compute.

Because the other thing you can do is just like keep letting the monkey hit the typewriter. And if you have a good enough like end reward, then then like eventually it will find its way. And so like, if I'm really talking about like where sort of exactly people sit on that scaffold.

Think, like, different people, different tasks, or, like, on different Yeah. Points there. And and a lot of it depends on how strong your prior over the correct things to do is, but that's the equation you're optimizing.

It's like how much am I willing to burn compute versus how much am I willing to burn, like, dollars on people's time to give scaffolding or or give Interesting. Rewards. Yeah.

You say we're not willing to do this for LMs, we are for people. I I I would think that the economic logic would flow in the opposite direction for the reason that you can amortize the cost of training any skill on a model Yeah. Across all the copies.

Like, we we we are willing to do this for LMs, like, some degree. Yeah. But, like, there's a there's, an equation you're maximizing here of, okay.

I've, like, raised all this money. Do I spend it along this axis or do I spend it on this Yeah. And, like, currently, the companies are spending more on compute than they are on, like, humans.

Otherwise, like, Scale AI's revenue would be, like, you know, $10,000,000,000 or you'd be like like in this okay. Look at it. Like, NVIDIA's revenue is much higher than Scale AI's revenue.

Right. And so, like, currently, the equation is compute over data.

I don't know. I don't know. It's unbelievable.

Yeah. But even even in light of all these result is really cool. You should talk about the language result.

that is more so the case in larger models, where you'd think, like, actually larger models have more space so they could, like, separate things out more.

But that being said, I I do think it's, like, your point on, are these models as sample efficient as humans? Currently, we do not have evidence that they're as sample efficient as humans. We have I think we have evidence of, like, total complexity ceiling.

Like, they're currently nothing that provide you of a clean enough signal you can't teach them, but we don't have evidence of, like, we can teach them as fast as humans do. And we would prefer that we get, like, learning on the job. This is, I think, one of those things you'll see start to happen over the next, like, year or two, but it's complex more from a, like, social dynamics aspect than it is a, like, a technical aspect.

Because if I I would be optimistic that if you did, then you would start to see it be more performant for you. And if you created, like, the Dwarkesh Podcast RL, like, feedback loop Yeah. Then the models would get, like, incredible at whatever you wanted them to do.

I suspect. Yeah. Yeah.

But there currently isn't a mechanism for you to do that with the model. So you can't, like, get say, hey. Here, like, have some feedback about how I want you to do something then, like, you know, somewhere on some server, like, you know, whizzes up and and, like currently, there's a text based memory, right, where it goes that's records things about what you wanted and puts in the prompt and tries to, like, build its own scaffolding and context.

I think an interesting question over the next few years is is whether that is totally sufficient, like, whether you just like this raw base intelligence plus, like, sufficient scaffolding in text is enough to build context or whether you need, to to somehow update the weights for your use case, and, like, some or some combination thereof.

What what is the I guess, if you wanted to interact it with, a human, what's happening on the back end? Is writing practice problems for itself? Is it, like, building actual environments for itself that it can train on?

A good question. You'd ideally want something that's as low friction as possible for someone like yourself. Like, you want you know, you're having a conversation and you say, no.

Not like that. Like, you want some, like, alert to, like, you know, flip and be like, hey. Okay.

We can convert this into something we could learn from. That's complex and and like tricky and like there's a lot of subtleties in in how to do that. I I mean, like the OpenAI sick sick and sick sick But actually like thumbs up can be a pretty terrible like reward signal for a model.

And then the same way like when Claude is doing coding for me, I'll actually often like, you know, sometimes I'm there just accepting suggestions, but sometimes it actually does like pretty much the right thing and I'm just like, oh, it's like 90% of the way there, but not perfect. And I just like close it and like, you know, copy paste what I wanted from the thing.

of it.

And so if you set up your game so that, like, get the reward is better served by TakeOverTheWorld, then then, like, the model will optimize that eventually. Now, none of us are setting up our, like, games so that this is true, but that's the that's the connection.

and only pretends in the short term in order to play the long game. And we're starting with unit tests now. But over the next year and or two years, we're going to significantly expand the time horizon of those tasks.

Like and it might be like, you're gonna achieve some goal. Like, I mean, you've got like, make money on the Internet or something like this. Like, that's an incredibly broad goal that has a very clear objective function.

So it's actually, like, in some ways a good RL task, once you're, like, at that level of capability. But it's also one that has incredible scope for, for, like, misalignment, let's say. Totally.

narrow and monolithic in a way that I don't expect AI to be either. But people have done that with like the constitution of the US government. Right?

Like the US government is, I think, is a better analogy in some respects of like this body that has goals and like can act on the world in as opposed to like an amorphous force, like industrial revolution.

I mean, I think the analogy kind of breaks down here because No, maybe so. Maybe so. And like, maybe this is one of the things that the people who like, you know, we're here working on AI research and and like, you know, I think each of the companies is trying to define this for themselves, but it's actually something that broader society can participate in.

Like, if you take as premise, then in a few years, we're gonna have something that's human level intelligence, and you wanna imbue that with a certain set of values. Like, what should those values be is a question that everyone should be participating in and sort of like offering a perspective on. I think Anthropic did a survey of, like, a whole bunch of people and put that into its constitutional data.

Yeah. But, yeah, I mean, there's a lot more to be done here. Yeah.

Like, in the constitutionally paper, it's it's not just flourishing. It's like there's, you know, there's a lot of strictures and there's a lot of, like, dot points there. Yeah.

Or do you care more about having like some hill to climb on while you're like from mediocre book to slightly less than mediocre to good? Yeah. Which which one is more important?

I think at the beginning, the hill to climb. Mhmm. So like the reason why people hill climb math, Hendrix math for so long was that there's five levels of problem.

Yeah. And it starts off like reasonably easy. And so you can both get some initial like signal of are you improving and then you have this like quite continuous signal, which is important.

Something like frontier math is actually only makes sense to introduce after you've got something like Hendrix math. Yeah. That you can you can like max out Hendrix math and they go, okay, now it's time for frontier math.

Yeah. Yeah.

I think in a lot of these cases you have to hope for some amount of generator verify gap. Mhmm. You need like it to be easier to judge, did you just output a million extraneous files than it is to like generate solutions in and Like that needs to be like a very easy to verify thing.

So sloppy's hard. Like one of the reasons that RLHF was initially so powerful is that it sort of imbued some sense of human values and like taste in the models.

the best public example is actually a paper that OpenAI put out recently where they judge the answers to medical questions using these, like grading criteria feedback. So there's like doctors have posed various questions, and then there's all these like it's like a marking criteria for a long for like a short answer question in an exam where did the model mention x y z? Did it recommend to do x this kind of thing?

And they they grade the model Yeah. According to this. And in this paper, they found that one, the models are like incredible at this.

And two, that the models are sufficient to grade the answers. Because maybe like one good mental model is roughly, if you can construct a grading criteria that like an everyday off the person person off the street could do, then the models are probably capable of like interpreting that criteria. If it requires expertise and taste, that's a tougher question.

Like in viewing like, is this a wonderful piece of art? Yep. Like, that's difficult.

Right. I think one of our friends, I don't know if I can say his name or not, like at one of the companies tried to teach the models to write. And I think like had a lot of trouble hiring human writers that were like he thought had taste and like weren't weren't like like encouraging the models to write slop.

Interesting. Yeah. It So worked to some degree.

Big model smell. Yeah.

inside the circuit. Yeah. Maybe that's one thing.

I think that's changed since last year. I remember you asked, like, do these models really reason? Yeah.

And when I look at those circuits That's right. Like, I can't think of anything else for reasoning.

are normal people's jobs? When we discussed something related to this before, Dwarkash was like, yeah, like in a normal job, you don't get feedback for an entire week. Like, how is a model meant to learn?

Like, wait, it's only so much feedback. Your next podcast.

Think like the highest thought a bit is I don't think there's anything fundamentally different about computer use than there is about like software engineering than there is about as long as you can represent everything in tokens and input space, which we can. We know the models can see. They can like draw bounding boxes around things in their images.

Right? So that that is the whole problem. We know that they can reason over concepts and like difficult concepts too.

The only difference with computer use is that like it's slightly harder to pose into these like feedback loops than math and coding. And so, to me, that indicates that with sufficient effort, computer use falls too. And I also think that it's underappreciated just like how far from a perfect machine these labs are.

Like it's not like you have a thousand people like, you know, optimizing the hell out of computer use and that like, you know, they've been trying as hard as they possibly can. Like everything at these labs, every single part of the model generation pipeline is best effort pulled together under incredible time pressure, incredible constraints as these companies are rapidly growing, trying desperately to pull and like upskill enough people to do the things that they need to do. Like, I think it's like it is best understood as with incredibly difficult prioritization problems, right?

Like coding is immensely valuable right now and like somewhat more tractable. So it actually makes sense to devote more of your effort to coding initially and like get closer to solving that because there's a super exponential value as you get closer to what's solving a domain than to allocate the marginal person towards computer use. And so everyone is making these difficult trade off calls over what do they care about.

Also, there's another aspect, which is that finally, not the researchers of the labs love working on the on the bars of intelligence that they themselves resonate with. So this is why math and competitive programming like fell first is because to everyone at the labs, this is their bar of intelligence. Like this is when they think fuck, what's a really smart like what is smart?

You're totally nerds, mate. It's like, oh, if it can beat me at Amy, then that's smart. Not if it can do an Excel model better than me.

That's like, well you know, if who cares if it can do an Excel model better than me. But if it can beat me at Amy then I respect it.

just not catered to foreigners. Yeah. Yeah.

Like filling out all the countries where you've been for visa. I hate that. Yeah.

Yeah. Yeah. Yeah.

I keep thinking I'm, like, close enough for personal admin escape velocity that, like, finally in, like, a year, the models will be doing my visas and stuff for me, but we'll get there.

I don't think it'll be able to autonomously do your taxes with a high degree of trust because I

it do them well? Will it miss something? Quite possibly.

Yeah. Will it be able to like click through TurboTax? I think yes.

Yeah. And fill out boxes. Be able to like search your email?

Like Yeah. That's the kind of thing I'm talking about. Yeah.

This is like the kind of thing where literally if you gave it like like one person month of effort, like in like in like then it would be sold. I I just wanna plus one What the fuck are you doing all day? I there's just so many things to do.

My answer is still if someone cares about it. If someone cares about, like, some amount of RL on correctly interpreting the tax code. Wait.

Even by the end of 2026, the model just can't, like, do things you're not explicitly training into? It will get the I think it will get the taxes wrong. Like, it'll it's like, okay.

End of. Okay. The unreliability and non confidence stuff will be like somewhat tricky.

We like to do this like all the time. Yeah. Interesting.

I'm a bit of an end to end, Maxi. I I think in general, like when people are like talking about the separate model. So for example, like most of the robotics companies are doing this kind of like to the bi level thing where they have like a motor policy that's running at whatever, like 60 hertz or whatever, and like some high level visual language model.

I'm pretty sure like almost all like the big robot companies are doing this. And they're doing this for a number of reasons. One of them is that like they want something to act at a very high frequency, and two is they can't train the big visual language model.

And so they like relying on that for general, like, world knowledge and this kind of stuff and, like, constructing longer running plans, but then they're like, you know, you offload to the motor policy. I'm very much of the opinion that if you are able to train the big model, eventually, at some point in the future, the distinction between big models and small models should disappear because you should be able to use the amount of computation in a model that is necessary to complete the task. Like, Ultimately, there's some amount of task complexity.

You don't have to use 100% of your brain all the time. Right. Welcome to my world.

And and so you should be able run that faster and this kind of stuff, basically. Basically. So I think it's like net net, I I think typically the same model.

Because you want you wanna be able to scale the the understanding Yeah. As the complexity and difficulty dealing. Right.

You wanna be able do that dynamically.

There's a surprisingly strong bias so far towards, tokens and text. It seems to work very well. One of that, there already is some amount of neural leads.

If you think about the residual stream for each token is neural leads to some degree. And so now we're just trading off axes. How much neural ease are you doing versus how much, like, actually is, like, read out to tokens all the time.

Yeah.

it's kind of trained more in isolation Yeah. Or with a human. And there'll be, like, some selective pressure against it as so long as the the agents are working with humans, because they wanna sort of cooperate.

But then, like, as agents begin to work more and more with each other, then that's that's, like, the pressure, like, changes the other direction, basically. Although somebody would still have to make the consciousness into do, like, end to end training for multiple agents to use the system of communication. Right?

Sure. Yeah.

I don't wanna do the math on exactly how much like we can ramp up TSMC's production and this kind of stuff. Like what fraction of the supply chain at the moment? We need Dylan in here for this, but like is currently GPU it's, like, relatively small.

Right? Like, or five something like this. Yeah.

Like, Apple has a huge fraction. Yeah. Yeah.

And in like, are the 2028 estimates including, like, that ramping up over time? Yeah. Yeah.

To what? Like, 30%?

I assume like it's saturated at that point. Is that why they expect it to then just like go go at like the I do think this is underrated to some degree. Yeah.

It's like to the the extent that like you don't instantly get like a doubling of the world's population in 2028. Yeah. You maybe get, you know, tens of millions of geniuses in a data center, but you you don't get a doubling of the world's population.

And so a lot depends on exactly how smart they are, exactly how efficient the models are, thinking about this kind of stuff. Let's do some rough math, guess, factor the h 100 thing. You could probably run like a 100 model, do like, I don't know, a thousand tokens or something like that, on an h 100.

So, if, like, we're comparing that to a number of should we compare that number a second? No. Okay.

Thousand tokens a second.

Which you still get pretty substantial numbers like even with your 100,000,000 h100s and you multiply it by 100, you're starting to like get to pretty substantial numbers. This does mean that those models themselves will be like somewhat compute bottlenecked in many respects. But these are all like these are relatively short term changes in in like timelines of progress, basically.

Like, think, yes, it's highly likely we get dramatically entrance bottleneck in 2728. Yeah. The impulse like to that will then be okay, let's just like try and turn out as many possible semiconductors we can.

Right. There'll be some lag there. A big part of like how fast we can do that will depend on how much people are feeling the AGI in the next, you know, two years as they're building out fab capacity.

then we think it's just gonna take the the the probability per year just goes down a bunch. Yeah. This is like bimodal distribution.

Yeah. A conversation I had with Leopold turned into a section in a situation where it's called this decade or bust, which is on exactly this topic. Yeah.

Which is basically that, you know, for the next couple of years, we can dramatically increase our training compute. RL is gonna be so exciting this year because we can, you know, dramatically increase the amount of compute that we apply to it. Interesting.

And it's also one of the reasons why I think the gap between, like say DeepSeek and O1 was so close at the beginning of the year, because they were able to apply like the same amount of compute to the RL process.

and then were able to take advantage of all the efficiency gains that everyone else was also seeing. Mhmm. Yeah.

I like, they're exactly on the sort of cost curve that you'd expect, which I don't think can take away from, like, the, like, brilliant engineers and, like, brilliant researchers who, like I look at I look at their work and I'm like, ah, like, kindred soul there in the in the in the work they're doing. Yeah. And to go from, like, way behind the frontier to, like, oh, this is, a real player.

It's it's super incredible work. Yeah. Yeah.

Okay. So people say that they have good research taste. Yeah.

I think their research taste is good in a way that I think, like, no one's research taste is good. Nombreon? Nombreon.

Nombreon also has good research taste. Nombreon, they very clearly understand this dance between the hardware systems that you're designing the models around and the algorithmic side of it. And this is manifesting the way that the models give this sense of being like perfectly designed up to their constraints.

And you can like really very clearly see what constraints they're thinking about as they're like iteratively solving these problems. And so, I mean, let's take the base transformer and like diff that to DeepSeek v two and v three. You can see them running up against the memory bandwidth bottleneck in Attention.

Yeah. And you can see them initially they do MLA to do this, like they trade flops for memory bandwidth basically. And then they do this thing called NSA where they like more selectively load memory bandwidth.

And you can see actually like this is because the model that they trained with MLA was on h eight hundreds, so it has a lot of FLOPS. So they were were like, okay, we can freely use the FLOPS. But then the export controls so from, like, Biden came in or, like, they're they're less they knew they would have less of those chips going forward.

And so they they traded off to, a more memory bandwidth oriented algorithmic solution there. And you see a similar thing with their approach to sparsity where they're iteratively working out the best way to do this over multiple papers. And the part that I like is that it's simple.

A big failure mode that a lot of ML researchers have is like you do these like overly complicated things that don't like think hard enough about the hardware systems that you have in mind. Whereas the deep sea the first DeepSeek like sparsity MOE solution, they designed these like rack and like like like node level load balancing losses. So you can see them being like, okay, like, have to like perfectly balance it on this.

Then they actually come up with a much better solution later on where they don't have to have the auxiliary loss, where they they just have these, like, bias terms that they put in. And it's Is that less simple? Like, you're manually putting in a bias rather than having a model balancing auxiliary losses annoying.

Like, you're you're making the model, like, trade off this thing, and, like, you have to with auxiliary losses, you have to, like, control the coefficient and the weighting. The bias is, like, cleaner in some respects. Interesting.

They did have to change it during training.

But, like, I I thought it was, like, I just thought it was cute that, like, can you see them running up into like this very hardware level constraint. Yeah. They tried like go like, what do we what do we wish we could express algorithmically?

What can we express under our constraints? And like iteratively solving to like get better constraints and doing this in a really like simple and elegant way and then like backing it up with great engineering. I also thought it was interesting that they incorporated the multi token prediction thing from Meta.

So Meta had a nice paper on this multi token prediction thing. Actually, I don't know if it's good or bad, like Meta didn't include it in LAMA, but Deepsea did include it in their paper, which I think is interesting. Like yeah.

Yeah. Was that because they were, like, faster at iterating and and including an algorithm, or did Meta decide that actually, like, it wasn't a good algorithmic change at scale? I don't know.

So even he, like, wanted god of model, like, design architecture design

rates of progress almost don't change that much depending on like, so long as it's able to completely implement these ideas. Interesting. Same way?

Like if you if you have like Noam Jazeer at a 100 x speed Yeah. That's still kinda wild. Yeah.

Like there's all these like fallbacks of like of like wild worlds. Yeah.

It's very like well defined objective function. Did the loss go down? Make number go down.

Make number go down. Or make them go up depending on which number it is.

And so once you get to the stage where models are capable of like implementing one of no one's ideas. Right. And then you can just like let them loose and like let them build that intuition of scientific of like how to do scientific discovery.

Right.

more and more experiments of the form of how can I dispatch work to a software engineering agent in such a way that is async? Clawed for is GitHub integration, where you can ask it to do things on GitHub, ask it to do pull requests, this kind of stuff that's coming up. And the OpenAI's Codecs are examples of this basically, where we you can sort of obviously see this in the coding startups.

Think of this like product exponential in some respects where you need to like be designing for like a few months ahead of the model to make sure that the product you build is the right one. Yeah. And you saw like last year, you know, Cursor hit PMF with Claw 3.

5 Sonnet. Right? Like the they were they were around for a while before, but then the model was finally good enough that the vision they had of how people would program like hit.

Yep. And then, know, Windsurf bet like a little bit more aggressively even on the agenticness of the model. Like, you could like with longer running agentic workflows and this kind of stuff, think that's when they sort of like began competing with cursors when they bet on that particular vision.

And the next one is, is you're not even in the loop, so to speak. You're not in an IDE, but you're asking the model to go do work in the same way that you would ask someone on your team to go do work. Yep.

And that is not quite ready yet. Like there are still a lot of tasks where you need to be in the loop. Yeah.

But the the next six months looks like an exploration of, like, exactly what does that trend line look like. Mhmm. Yeah.

But just to be really concrete or pedantic about the bottlenecks here, a lot of it is, just tooling and are the pipes connected? Yeah.

and the ability to to to use all of the tools that are necessary. And we're almost certainly, like, under eliciting dramatically. When you look at Meta's evals of can the model solve the task, they're there solving them for like hours Yeah.

Over like multiple iterations. And eventually, one of them is like, oh, yeah, I've come back and I've solved the task. Me at the moment, least, like maybe the fault is my own, but I try the model and doing something and if it can't do it, I'm like, okay, fine.

I'll it. Yeah. I don't Which is so interesting because you we don't even treat other humans this way.

Right.

I won't really use it. Yeah. Yeah.

It's only when it's right there and it's I can send off something. If it hits, great. If not, I'm kind of working on it at the same time.

Yeah. Yeah. But this more async form factor, expect to like really quite dramatically improve the experience of these models.

Interesting. Interesting. You can just say like, let's see if it can do that.

Yeah. Just give it a while.

I think one important point is that when you look at AlphaZero, does have all of those ingredients. And in particular, think the intellectual ceiling goes quite contra what I was saying before, is we've demonstrated this incredible complexity of math and programming problems. I do think that the type of task and setting that AlphaZero, like, worked in, this two player perfect information, like, game, basically, is incredibly friendly to IRL algorithms.

And the reason it took so long to to get to like a more AGI proto AGI style models is you do need to crack that like general conceptual understanding of like the world and language and this kind of stuff, and you need to get the initial reward signal on tasks that you care about in the real world, which are like harder to specify than games.

bust timeline as of twenty to thirty or bust? I would be extremely surprised if that was the case. And I think that would be like somewhat of an update towards like, there's something like strangely difficult about Yeah.

This, like, computer use in particular. Yeah.

I'm surprised by how many deep learning critics just, like, haven't really interacted with the models or haven't in a while. And constantly move the gold first. Yeah.

Yeah. Yeah. Yeah.

Like, the Turing test used to be a thing. Right? Like Yeah.

We don't even talk about it, and it'd be, like, silly to think that it was a meaningful test. Yeah. Yeah.

Now that being said, one caveat on that is, like, if software engineering is just, like, dramatically better than computer use, I mean, computer use still sucks, then I'd be, like, still, like, oh, maybe everyone just kept focusing on software engineering. Like, it was just, like, by far the most valuable thing, like like, every marginal person in Dollar went towards software engineering. I don't think that's the case.

I do think, like, computer use is valuable enough that, like, you know, people will care about it. Yeah.

Oh,

domain? Yeah. So one intuition pump is this conversation was had a lot when models were like GPT two sized and fine tuned for various things.

And they found, you know, people would find that the models were dramatically better at things that they were fine tuned for. Mhmm. Right?

But by the time you get to GPT-four when it's trained on a wide enough variety of things, actually the, like the sort of total compute, like it generalized very well across all of like the individual subtasks actually generalized better than smaller fine tuned models in a way that was extremely useful. I think right now what we're seeing with RL is pretty much the same story playing out, where there's this jaggedness of like things that they're particularly trained at.

No. I mean, no. Yeah.

Exactly. Yeah. Like, so like all this time we're talking about like, oh, it's only good at things that's being RL ed.

Well, it's pretty good at that because that's pretty you know, that is a mixture of like science and like understanding language and like, and coding. Like, there's this sort of like mixture of domains here, all of which you need to understand. Like, you need to be both a great software engineer and be able to like think through language and then like a like state of mind and almost philosophize in some respects to be an Interp agent.

Yeah. And it is generalizing from the training Yeah. Yeah.

To do that. What's the endgame here?

You can also going back to, like, the the World War two question. You can think of it as, like, a hierarchy of abstractions of trust here. Where, like, let's say you wanna go and talk to, like, Churchill.

It helps a lot if you can verify that in that conversation, in that ten minutes, he's being honest. And this like enables you to construct better meta narratives of what's going on. And so maybe particle physics wouldn't help you there, but certainly like the neuroscience of Churchill's brain would help you verify that he was being trustworthy in that conversation and that they're like, you know, the soldiers on the front lines were being honest in their depiction of their description of what happened and this kind of stuff.

So long as you can verify like progress, like parts of the the tree up, then that that massively helps you build confidence.

there's this whole spectrum of crazy futures. But the one that I feel we're almost guaranteed to get, and this is like almost a strong statement to make, is one where, like at the very least, you get drop in like white collar worker at some point in the next five years. It's like I think it's very likely in two, but it seems almost overdetermined in like five.

And and on like the grand scheme of things, those are kind of irrelevant timeframes. Like it's the same either way. And that completely changes the world over the next decade.

And and if the sort of if we don't have the right policies in place for that, then you end up actually with almost in some respects like a fundamentally worse world. Because the thing that these models get good at by default is like software engineering and like computer using agents and this kind of stuff. And then we need to we will need to put in extra effort to put them in the loops where they help us with scientific research or they're like, we have the right robotics such that we actually, experience an increase in material quality of life.

So that's worth thinking about. Like, if you're in the perspective of, I'm a country. Like, what should I be doing or thinking about?

So I think one very important point is that let's say this this scenario turns out true. Then compute becomes the most valuable resource in the world. Yep.

Like the sort of GDP of your economy is dramatically affected by how much compute you can deploy towards these sort of organizations within your country. And so having some guaranteed amount of compute, I think will actually be quite important. So like pre getting ahead of investments in like data centers and this kind of stuff on the condition that it's like companies in your country have to be allowed to use that compute.

And yeah, not necessarily for training, like just even just for inference. Like I think the economic value here comes from inference. I think it also makes sense to invest broadly in AI, like I think these countries have the opportunity to do so.

And I think that's like a portfolio of like, foundation model companies, but also like robotic supply chain and this kind of stuff. I think that you should invest very proactively in policies that try and prove, like, prevent capital lock in. So we're in for a much worse world if, like, it just so happens that the people who had like money in the stock exchange or in land before AGI are like dramatically more wealthy than the people who don't.

Because it's a gross misallocation of resources. So having like, I know one of my favorite episodes actually on your podcast was like the Georgism one where you're trying to appropriately value or allocate land. And so I think this strikes particularly close to home coming from Australia where I think our policies with respect to land are like grossly Yeah.

Wrong. But think this is broadly true. Being very forward on regulation of integration of these models into your country is is important and proactively making sure that people have choice.

Like, so let's say you should be quite proactive about making sure that like the phones or devices or like glasses that people have, people have like free choice on like what Yeah. Things they run. And then so that's like that's the we just get white collar worker.

Right? And like you're trying to like do the best to like prepare your country for that. And then it's like, okay, well, what can you do to make all possible versions of the future go well?

That's covering some amount of economic downside. The other things I think are really important is figure out how you can either make the basically ensure dramatic upside or cover terrible downside. And so getting dramatic upside is making sure that there is investment in biology research and this kind of stuff in an automated way that these models are actually, like, able to produce novel medicines that mass like, massively improve our, like, quality of life.

And covering the downside is like AI alignment research and this kind of stuff and automated testing and, like, really thinking hard about that, AI safety institutes, this kind of stuff. But these seem like things that a rich person a random rich person could also do.

I would be doing that if I was in charge of a nation state. Think it just increases your optionality in, like, most of the future worlds. Yep.

basically and China's line is like this. And I mean, US like very clearly Yeah. We just need so many more power plants.

Yes. If intelligence becomes this like incredibly valuable input, like intelligence becomes almost a raw input into the economies and quality of life of future, the thing directly underneath that is energy.

progress stalls out and we just never figure out how to keep progress going, which I don't think is the case. That hasn't stalled out yet, it seems to be going The current suite of algorithms are sufficient to automate white collar work provided you have enough of the right kinds of data. Yes.

And in a way that, like, compared to the TAM of salaries for all of those kinds of work is so, like, trivially worthwhile. Yep. Yeah.

Exactly.

Mhmm. And and not, like, necessarily saying that, like, that's what the AIs would be, like, want to do or anything like that. But as in, like, if you were to be, like, what are the relative economic value of things?

Like, the AIs are out there doing computer programming and, the most valuable thing that humans can do is, like, be amazing robots. Yeah. Now that being said, I think Morovik's paradox is a little bit fake.

I think the main reason that robots are worse than at, like being a robot than they are at software engineering is the Internet exists for software engineering. Like GitHub exists and there is no equivalent thing. Like if you had all like, you know, mocap of everyone's actions as they were like going about their daily lives, like some reasonable fraction of the human population, robotics is also like close to solved.

Like like like on track to be solved at the same rate that software engineering is on track to be solved. So this is only like this vision is only like a sort of decade long section, but it's still a pretty terrible decade. Like imagine the world where people have lost their jobs, you haven't yet got novel biological research that means people's quality of life is dramatically better, You don't yet have material abundance because haven't actually been able to action the physical world in the necessary way.

You can't build dramatically more because building dramatically more takes robots basically. And people's like main comparative advantage is as fantastic robots is like a shocking, shocking world.

is solved and you just like to get like, you know, like radical abundance basically, provided that you have all the policies set up, like necessary to permit building. Like you you sort of you end up with that same change from, you know, the like the the before and after photos of Shanghai Yeah. Yeah.

Where like twenty years on, it's like this dramatically transformed city. Like a lot of places in the world probably end up like that Right. Over that two decade period.

But we need to make sure, like, one, do our best to estimate, is this actually what is on track to happen? Like, build SWE bench before all the other forms of white collar work and measure and track. That's a great thing that government should be doing by the way.

It's like trying to break down the sort of functions of their economy into measurable tasks and figuring out where what does the curve actually look like for that? Because they might be a bit shocked by the progress there. There's no sweep bench for taxi And then I don't like have all the answers here, like figuring out a way to like share the proceeds of this economy like broadly across people or like invest heavily in robotics and collecting data so that we get robotics faster and get material abundance faster, invest in biological research that we get but like all that faster.

It just seems more likely that we'll get the glorious transhumanist future where they're developing the things that make human life better. Yes. I I mean, I agree.

Like, the the case where you end up with, two national projects facing off against each other is dramatically worse. Right. Like, we don't wanna live in that world.

Yeah. It's much much better if it's, like, stays in a freak market, so to speak. Yeah.

Yeah. Yeah.

like, important, like, mental model, to think about RL is I think as, like, the the task gets more, there is some respect with which like longer horizon or better that task, if you can do them, if you can get that reward ever, are like easier to judge. So like, again, come back to that like, can you make money on the Internet? That's incredibly easy reward signal to judge.

But to like do that, there's like a whole hierarchy of like complex behavior. So if you could like pre train up to the easy to judge reward signals, does your website work? Does it go down?

Like do people like it? Like there's there's all these reward signals that we can respond to because we have a long we can like progress through these long enough trajectories to actually like get to interesting things. If you're stuck in this regime where like you need a reward signal every five tokens, it's a way more painful and like long process.

But if you could like pre train on every like screen in America, then probably the like RL tasks that you can design Interesting. Are very different to like, if you could only like take the existing internet as it is today. And so, like, how much of that you get access to, like, changes the the mix.

Interesting.

I don't know. Your rate of improvement of biology is gonna be, like, somewhat bound by the time it takes a cell to grow in a way that your rate of improvement on math isn't, for example. So yes, but I think for many things we'll be able to paralyze far like wisely enough and and get enough iteration loops.

Mhmm. Yeah.

whether or not you think like, there's a virtue in pretraining a new architecture. Basically, if you make some, like, architectural change, then you, like, probably need to, like, do some form of, like, at least, like, retraining a new model. Mhmm.

there is science to do, which, you know, everyone is doing of what is the optimal point at which to do URL. Mhmm. Because you need something which can both learn and discover the sparse reward itself.

So you don't want a one parameter model, useless, even though you can run it really fast. You also don't want a 100 t model because it's super slow. Yeah.

Password RL. And I'd like the sort of the marginal benefit of like its learning efficiency is like not worth it. Right.

So there's like a there's a pretty different view here. Like what's the optimal model size of like your current class of capabilities and like your current set of RL environments and this kind of stuff. Yeah.

And and even in the last year, there's been much more of a factor of the inference cost. Right? So just explicitly, like, the bigger the model, the more expensive it is to do a forward pass and generate tokens.

So I think once again, there's like it's worth considering the spectrum of possible worlds and preparing yourself for that. Yeah. And the one like, the sort of action that I think is like highest EV in that case is you are about to get dramatic at a minimum, you are about to get dramatically more leverage.

Mhmm. You already have. Like, already the startups in YC are like writing huge amounts of their code with, you know, Claude.

So what challenges, what causes do you want to change in the world with that added leverage? Like, you had 10 engineers, at your beck and call, what would you do? Or if you had a company at your beck and call, like, what would that enable you to do?

And what problems and domains suddenly become tractable? That's the world you wanna prepare for. Now that still requires a lot of technical depth.

Obviously, is the case where AI just becomes dramatically better than, like, everyone at everything. Right? But for at least a while, probably, there is like advantage I think Jensen actually talked about this in an interview in an interesting way where he's like, you know, have like a 100,000 general intelligences around me and I'm still like somewhat useful, because I'm there like, know, directing the values and like, like, you know, like, asking them to do things and, know, they're sort of like, I still have value even though I have 100,000 general intelligences.

And for many people, think that will still be true for a fair while. And then, you know, as the AIs get better and better and better and like so on, eventually no. But again, prepare for like the spectrum of possible worlds because in the event where we're just totally out competed, it doesn't matter what you do.

But in all the other worlds, it matters a lot. Get the technical depth. Study biology.

Study CS. Like, really think hard about study physics. Think about hard about what challenges you wanna solve in the world.

Yeah. That's a lot of topics. That's a lot You can now.

You can. Right? Like, it's so much easier to learn.

That's right. Yeah. Everyone now has the, like, infinite perfect tutor.

Yeah. Yeah. Yeah.

Yeah. It's definitely been helpful to me. Yeah.

I would say some combination of, like, get rid of the sunk cost of your, like, previous workflows or expertise Yeah.

I think that now that our role has, like, come back, papers building on Andy Jones' scaling board lengths, like scaling walls for board games are interesting. Showing that you can investigating these questions like the ones you asked before where you're like, oh, is the model actually learning to do more than its previous pass at k or is it just like Yeah. Yeah.

Discovering that? Like, exploring questions like that deeply, I think are interesting. Yeah.

Yeah. Like scaling laws for RL, basically.

I think this is one of the, like, like, best ways to sort of demonstrate that you have, like, the raw ability to to do it. Like, if you made a extremely efficient transformer implementation on TPU or Trainium or like in CUDA, then think there's a pretty high likelihood that you'll get a job offer. But there's a relatively small pool of people that you can trust to, like, completely own end to end the performance of a of a model.

Mhmm. And and if you have broad, deep electrical engineering skills, I think you can probably come up to speed pretty fast on accelerator stuff. Yeah.

You can come up to speed, like, reasonably fast, and it teaches you a lot of good intuitions of the actual intricacies of what's going on in the models, which means that you're then very well placed to, like, think about architecture and this kind of stuff.