Success without Dignity? Nathan finds Hope Amidst Chaos, from The Intelligence Horizon Podcast

When it comes down to the kind of core question I think a lot of people are getting at is like, is this AI thing gonna kinda fizzle out before it really becomes a big deal, or is it gonna be a huge world altering deal? I'm very much confidently clearly in the camp of it's gonna be a huge, huge deal, and the details are, you know, where I think the discussion or the debate remains now, not for me at least, you know, whether or not the the overall trajectory of AI is gonna take us to something that is powerful enough to be transformed. One of the strangest things in the world today, full stop, is the fact that the disagreement among very plugged in, very informed, very smart people has not really been reduced much at all, even as we've gained a ton of information over the last couple of years about the trajectory of AI.

I think that is super strange, and I'm honestly pretty confused by it. The one thing everybody seems to agree on is, like, the timeline on which we should expect this has come in. In today's world, if you say you don't think you're gonna AGI until 2035, you're like an AI bear.

But only five years ago, that was considered to be quite aggressive. And, you know, most people were more like, I don't know, 2050, maybe not in my lifetime. So there's been this massive compression of timeline.

There's obviously been this huge jump capability, and yet on these fundamental questions of like, what's going to happen? You know, there's still total disagreement.

Thanks for having me. I'm excited for this conversation, guys.

Starting with the the easy ones. Well okay. First of all, as I'm sure you guys are very well aware, what exactly do we mean by AGI is a slippery question at least that has people talking past each other quite a bit.

So I don't think there's any, you know, like, super privileged definition. What I do think we're pretty clearly on the cusp of is powerful AI that is better than the vast majority of people, although perhaps not the, you know, very few top top experts in a given domain, at pretty much all cognitive work. That seems like pretty clearly on the horizon, and I think it is absolutely going to be enough to be transformative to the economy, to daily life, potentially even, you know, bigger things than that, like the very nature and status of the human species.

And I think that we will have that kind of regardless of whether or not there are some niche areas where humans retain an advantage, which my guess is probably will be the case. Certainly, one of the big things that has become clearer over time as AI systems have gotten better is that they are jagged in this weird way where they have certain things they just do amazingly well at, other things they kind of weirdly struggle at. They're not very adversarially robust, for example.

You know, they're they're easier than humans, I would say, still to to trick. So, you know, there's there's gonna be weirdness. And I think, you know, throughout the conversation today, probably a big theme will be to expect weird things to happen.

But, you know, when it comes down to the kind of core question I think a lot of people are getting at is, like, is this AI thing gonna kinda fizzle out before it really becomes a big deal, or is it gonna be a huge world altering deal? I'm very much confidently clearly in the camp of it's gonna be a huge, huge deal. And the details are, you know, where I think the discussion or the debate remains now.

Not for me at least, whether or not the the overall trajectory of AI is gonna take us to something that is powerful enough to be transformative. Okay.

there are additional unlocks that allow AI systems to truly undeniably surpass what humans are capable of. I think you look at the natural world, and clearly humans did that to every other animal species that existed before us. Right?

So there is a historical precedent for some new kind of mind showing up on the scene and blowing away all the other minds that came before us. I don't think we have any reason to believe that that couldn't happen to us in some, you know, law of physics guarantee sort of way. So I definitely think that's possible.

But in terms of what I could confidently foresee, I don't think it is clear that that's gonna happen in the next few years. But what I do think is still, again, clear is that we are gonna have systems that are powerful enough to be transformative across almost all the questions that we care about in terms of what is society gonna look like? How are we gonna organize?

Are we gonna need a new social contract? Like, all those things seem to me pretty clear. And then, you know, people quibble often around the edges of could I mean, you get into there's some pretty esoteric jobs out there.

Right? So, like, could an will an AI system ever be able to be as good of a sommelier as the best human sommeliers? Well, I don't know.

You know, is anybody gonna be motivated to train one to try to do that? You know, we don't really have a lot of taste AI at this point in time. And, you know, maybe tasting will be the, forever the domain of humans.

But trying to identify these little niches where we may have some really long standing durable advantage, I think too often distracts from the big question of like, is this gonna change just about everything that matters to us? And and there I I clearly come down that the answer will be yes. Great.

is. And I also think if it's not, we may never quite answer that question in the sense that I do expect that we will continue to see new conceptual advances in AI research. You know, the field is growing.

Everything one of my sayings is everything is going exponential. So that is the number of people that are working in AI research. It's the number of papers.

It's the number of experiments being run. It's the amount of compute that people have to run those experiments on. It's the datasets that have been collected.

It's the RL environments that are being built out over time. And I do think all those things are gonna probably give us some new conceptual unlocks such that we will never probably answer the question. And this already kinda happened with pre training.

Right? You you rewind two to three years. There was a time when people were like, well, we're kinda running out of data.

You know, can we really scale this all the way to AGI? And nobody's talking now about can we scale pre training all the way to AGI anymore because there's been a new thing. And so now the new thing is on top, and it's like, well, clearly, that's gonna be part of the mix.

Will this, you know, exact pattern or this this sort of shape be enough to get all the way to AGI? My prediction is that again, like, what does AGI mean? But my prediction is it it probably is enough to get us to systems that are transformative.

But if we fast forward to 2028 and, you know, look back at this conversation, we'll probably say, well, nobody's really asking that anymore because we do have a couple new things that have come online. And so now we have a little richer sense of, like, what the shape of it's gonna be. And, you know, we're either getting there or we're not.

We maybe still are missing a little something at that point. But it I I would guess that we would look back at the current thing and say, yeah. Clearly, a couple of things have been added, and they were a big deal.

And so it was, you know, the question of like, is exactly what we have in February 2026 enough? Kind of ends up being beside the point in the final analysis.

get to AI systems that can do most of the cognitive work in the economy, example. I mean, it seems like, honestly, we're already reasonably close to that, and it doesn't seem like we're anywhere close to done. And by the way, if you listen to the lab leaders, the, frontier model developer leaders, they are still saying, by the way, too, that pretraining is still working.

You know, it was never really the case that pretraining stopped working. As far as I know, those scaling laws basically have held. I think what happened at one point in time was the next step on the pretraining frontier was becoming really expensive.

And then they found another kind of maybe not fully orthogonal, but, you know, pretty different direction to go with scaling post training with the RL paradigm that we have now. And that was just much bigger bang for their buck, at least at that moment in time. Right?

Like, they had already gone pretty far up the pretraining curve. They hadn't gone very far up at all, the RL curve. Now presumably, things were gonna kinda even out.

You know, a general kind of economic theory would be like people should be investing in one until the marginal return decreases to the level of the other, and then they could maybe invest in both. So if you find some new path that's like, oh, this is really giving us huge ROI, you go hard at that path for a while. But then that kind of hits some diminishing returns, and now you're kind of back to, okay.

Well, maybe we need to do all these things more at the same time. So we'll do a little more pre pre training. We'll do a little more RL, and we'll do, you know, maybe more of that mystery third thing all advancing in tandem.

Right now, I think we're advancing. I don't have insider information on, you know, exactly what these ratios look like, but I'd say it's probably roughly the case that we're close to, if not at the point where additional compute going into RL is roughly giving the same kind of returns as additional compute going into pretraining. And so both are gonna are gonna be, you know, places where frontier companies can invest for the time being.

You know, and I guess another way to think about this is, like, people also ask, like, about generalization in RL. And I think, again, it's kind of worth unpacking, like, what does it mean to generalize? Like, what are we talking about generalizing?

One way to think about the question is if we do a bunch of RL on a model, does the model generalize to all sorts of new things? And there I would say, again, there's probably another, way to to break that down, which is like there are domain specific skills, and then there are more kind of cognitive or even like metacognitive skills that work across domains. So one of the biggest, like, I guess we'll call it the moment for me and for deep sea researchers and for the r one model that they were training was from the r one paper.

This is January 2025. They, reported this what they call the moment, which is in in their process of doing RL on, you know, an already pretty capable base model, of course. Right?

They found that these previously unobserved higher order cognitive behaviors started to come online. The moment in particular was in the reasoning trace, the r one model gets to a point and it says, oh, wait. This is an moment.

Like, I can come at this from a totally different direction. And this is something that hadn't been observed much. I mean, it's it's out there in the pretraining data.

There's at least some examples, right, of people kind of documenting their own chain of thought and getting to these moments and realizing, oh, I was coming at it the wrong way. Now I can come at it this other way. But that hadn't been observed too much in AI systems.

Reinforcement learning seems to be bringing you know, clearly has, like, brought that sort of thing out. And now we have these long reasoning traces where the model will kind of come at the same problem from a bunch of different directions. And so what generalizes, what doesn't generalize, I think you probably can't take a model that has never been trained on a particular domain and expect it to go into that domain and be successful.

But you can expect some of these, like, meta traits to generalize from one area to another. And then if you zoom out the farthest and just say, like, does RL as a process that companies or, you know, organizations apply, does that generalize? And there, think the answer is, like, definitely yes.

It's just a question of getting the reward signal dialed in to the point where it actually works. And that's definitely easier in some areas than in others. And so we do see, like, in areas where it's easy, like math and programming, we see fast progress there relative to things where it's harder to get a clear reward signal.

But still, I think we are quite obviously making that work. Episode of the podcast that's coming out, I think, today as we're talking is with the head of health at OpenAI. And, you know, I I can tell you from personal experience, latest models are absolutely on the level of attending physicians.

My son, unfortunately, has had cancer over the last few months. He seems to be very much on track to be cured and be all better, which is fantastic. And I probably wouldn't be here talking to you if this that wasn't the case.

But I've had occasion to really use the latest models intensively in a medical context, and they're absolutely on the level of the attending physicians. They know a lot more than the residents, and they they really are step for step with the most senior doctors at the hospital. How is that happening?

Well, they've worked with 250 plus human doctors closely at OpenAI to create training data, to grade, etcetera, etcetera, etcetera. But now their latest models are also outperforming their human doctors when it comes to the task of evaluating AI outputs. So there there are these kind of thresholds that they're crossing where it's like, you know, it was really hard.

I sometimes think of this as like spinning a big wheel. You know, like, your first push is on this big wheel. Don't move it much.

Right? But as you build up this momentum, as the flywheel really starts to turn, you start to hit these thresholds where it's like, well, now we have a model that is outperforming our humans. You know, think of all the work that they had to put in thousands and thousands of hours and millions and millions hundreds of millions of dollars potentially to hire hundreds of doctors to do all this work.

Now they've got a model that is beating the doctors at evaluating outputs in the medical domain. And so that totally changes the game. Those thresholds will be crossed at different times for different domains.

But I think it's like safe to say that in most domains, if there's any sort of objective ground truth or even like a high level of agreement among professionals, you can get there. It just takes more time. Hey.

We'll continue our interview in a moment after a word from our sponsors.

probably not gonna play out that way. And if it did, I still don't really think it matters that much. It probably shifts the timeline a little bit, but the RL will continue to scale.

The amount of compute coming online is, again, that's exponential. Right? So there's just tremendous amount of additional resources to be thrown into this.

One thing we're doing a bit, but not a ton, is just, like, using the signal from the world. You know, Elon's got a plan with x AI to and I think he's got a real advantage here to just have the AI solve the same super hard problems that the engineers at Tesla and SpaceX and, Neuralink are solving. So he's just gonna give them a computer, say, here's all the professional software.

Here's the problem. You gotta solve it. Right?

There's like a never ending supply of those problems. And I wouldn't be surprised to see Grock get to, like, Tesla, SpaceX, Neuralink engineer level just based on the fact that those problems are there to be solved and, you know, they've got the compute to keep trying. And what's really the fundamental barrier there?

I I don't really see it. I honestly think that the the new paradigm is probably more about usability than it is about capability. By which I mean, like, people have a lot of complaints when they use models because they're like, oh, it didn't really do what I wanted it to do or I sort of, you didn't really understand me or what have you.

And clearly, one thing that they're not great at doing is going into a new environment, kind of scoping the situation out, getting the vibe, you know, getting picking up that subtle feedback that people give each other and gradually kind of figuring it out and, like, becoming a useful contributor. That's like how a lot of people go from day one to, like, effective employee in their jobs. And AIs don't really do that in the same way, obviously.

Right? They don't have continual learning. They are able to manage their own memory somewhat, but they're not that awesome at that.

That's another one of these, like, cognitive skills that does generalize, though. Like, once you're good at managing your memory, you'll be able to apply that in in new domains. Mhmm.

But they're not that great at that yet. And I think that prevents a lot of people from getting value. More often, I think that not that the model can't fundamentally do the thing, but that it doesn't have the context.

The person trying to get it to do the thing doesn't know how to assemble the context or doesn't believe that it could do the things such that they're willing to invest the time and energy to Yeah. Give it the context. And so that unlock might just be like, now you don't have to do that anymore.

It will kind of figure it out on its own. And this is maybe a way to expose in a way similar to instruction following. Right?

Like in the if you were really good at prompting GPT three, you could get it to do a lot of things. But it was a weird art to prompt GPT three. And to a much lesser degree, but still somewhat, you know, getting value from the current models is still kind of a weird art.

You kinda have to have an intuition for them. You kinda need to know how to assemble context and give effective instructions, and people aren't that great at that. So that next version might not even be so much about allowing them to do qualitatively new things, but just making the barrier to use much lower so that people can just be like, Hey, AI coworker, welcome to Slack.

And then over a week or whatever, they just kind of ramp up and gradually get it. Yeah, interesting.

Well, again, I'll take one beat to say, I think the recent trajectory shows that this problem is being solved. The I'm sure you are very familiar with the meter graph, which is everywhere these days. It's basically going vertical at this point.

It's to the point where the meter people are like, we are really struggling to have tasks long enough to even be able to evaluate these things on. And, you know, it's also kinda worth noting too that that's a bit of a challenge with humans. Right?

Like, we hire people on much less than, you know, a week's worth of work. And it it's not, like, always super easy for people to agree, like, did somebody do a good job on that month long project or not? You know?

You can usually tell if they, like, totally crushed it or, like, totally sucked. There's often a lot of disagreement in organizations about, well, you know, maybe it was actually harder than we thought or, you know, they didn't have the the ingredients for success weren't really there. I mean, there's there's often a lot of fuzziness in in this stuff even in in human context.

I mean, there's a bunch of techniques that I think are are being used. One is rubric rewards. So, you know, it's it's Elon likes to talk about things like, does the rocket fly?

The ultimate ground truth is if I can send this thing into space and I can come land it down on a pad standing up on its tail again, then, like, clearly, that worked. And yet the reward for that you know, that's an expensive experiment to run. You couldn't just, you know, launch a million rockets and have all of them you know, most of them crash to find the ones that worked.

So there is a challenge there in terms of sparse reward and the the cost of the experiments. I think what is happening a lot is that people are defining rubrics of things that they want to make sure the AI does well. I mean, they're probably working with AIs to develop those rubrics.

Again, in the OpenAI health context, they created a benchmark called HealthBench where there are 49,000 evaluation criteria. So all these different tasks, you know, puzzles that the AIs have to figure out. And it's not just like, did you get it right or did you not get it right?

But it's a painstaking effort to really flesh out, you know, all the different things that would matter that would make for a complete awesome, you know, best in class answer. And then the AIs are not scored zero and one. They're scored on, you know, some sort of scale that's like, you may have got zero out of 25 things, right, that you could have got on this question.

You may have got five, ten, 15, but that gives you enough of a signal that you can kind of climb that hill. And it really does seem to be working. I would I think that is gonna work pretty well in domains where there is a professional consensus.

Because I think that's how people evaluate each other too. Multiple choice tests, but there are also like, you know, you gotta kinda show in a medical context. You know, you you're a student.

You go through these in person training processes. They say watch one, do one, teach one. And so, you know, as a medical student, you watch, and then later you start to do, and then you get a lot of feedback.

And eventually, the next thing you know, you're the one teaching. I think the AIs will pick up those signals. And then there's other things where it's taste.

And I think that that'll be a little bit different probably. But if I had to guess, you know, what is a good novel? Well, first of all, there is no consensus on that.

Right? Even you can find somebody who hates even the, you know, most universally critically acclaimed novels, and you can find somebody who loves something that everybody else thinks is trash. I think what we'll see there is kind of taste based communities coming together to shape models for their own tastes.

So in other words, you might start with a base model, and you might want it to write romance novels, or you might want it to write anime, or you might want it to write hard sci fi. And what it means to be good in those different genres is, like, quite different. But what you do have is fans of those genres that can engage with outputs, give their scores, and shape models based on what they like.

sometimes is just to try to reflect on, like, what I do, what is my experience like, and you really try to interrogate it for, is there something really magical going on or not? And I think I have to say really long horizon agency is pretty rare among humans. Right?

I mean, you do have these people who, like in Elon who keeps going up, say, you know, my goal is to make humanity a multiplanetary species. And if it takes thirty years, you know, that so be it. Right?

So I would say, you know, skepticism around, like, will we get AIs to be, you know, multi decade horizon planners that can sort of show up with the fierceness and determination required on a day in and day out basis to really make that happen? It remains, like, pretty valid skepticism in my opinion. At the same time, you look at, what most people do and what drives the economy, and it's, like, much shorter term than that.

You know, a lot of it is like, kinda keep showing up one day to the next, and every day I kinda start fresh. You know, I went to sleep and I kind of shut down. You know, there's like there's sort of a discontinuity of consciousness between workdays, which, you know, I don't wanna I'm not a big analogy guy, but if you squint at it, you can sort of make an analogy between, like, different context windows, and there's kind of a, you know, a somewhat similar resetting type of moment.

And what do I have to do? Like, I have to kind of reboot myself and be like, what was I doing yesterday? Where did I leave off?

And what did I accomplish? What was still to be accomplished? And most of that stuff, you know, I could write down.

If I if I had, like, a worse if my integrated memory was worse, I could compensate for that pretty well, you know, at least on sort of a week long or maybe a month long or maybe a quarter long basis. If I got really good at the end of the day at the end of each day of saying, like, here's everything I did today. Here's what went well.

Here's what failed. Here's what's next. Here's the feedback I got.

And here's what I think I should do in the next session, and then wiped all that out and came back and looked at that scratch pad, you know, again to start my day tomorrow. I think over time, I could get, like, pretty effective at that. So I don't and I I think this does connect back to also to the certain kinds of skills do generalize, and I do think this is one of them.

Right? Managing your own memory, writing notes for yourself to kind of document what happened and what's supposed to happen next, giving yourself a sense of, like, where you are on the overall story that is unfolding. I think that, you know, they're not great at that yet, but they're getting decent.

And, you know, when my Claude code crashes these days, a lot of times, you know, just kill that tab, resume in the next one, and then I'll just be like, resume, you know, or like, sorry. We got cut off. Like, my Internet went out.

You know? Please pick up where you left off. And they're able to do it.

You know? Again, they're not awesome yet, but they're way better than they were even three months ago. So I think it is hard for me to see how that doesn't extend, especially because it is a relatively new thing.

Right? And and most of these things that pop up, they have at least a few generations before they kinda level off. It's hard for me to see how we don't see a again, a pretty steep you know, meter curve is currently vertical.

It can hardly get steeper than it is. But I think we've got at least a few generations of, like, very steep progress there. And then maybe there's some other kind of thing where it's like, okay.

Sure. You can do a quarter worth of work when somebody gives you a project. Can you, like, figure out what the next big civilizational advance could be like some of the true human visionaries?

Maybe not. Maybe that's a different kind of thing. In a way, I kind of hope that we stop there.

Sometimes I because I talk up a lot what I think AI is capable of, will be capable of. It's easy to mistake me as like a booster. I'm actually kind of afraid of that.

I think it would be great in some ways if we did find certain fundamental barriers where it's like, hey. I I can delegate a month or a quarter's worth of work to this thing, and it might be able to do it in a day for a couple $100. Like, wow.

Amazing. If we could kind of park it there, you know, and not have it go to the point where it's doing multi decade level planning, that might be a really good thing. That might be the sort of the sweet spot where we get a lot of the advances that we want and the better quality of life and, you know, all the abundance that people dream about, and we don't have so much risk of losing control.

So I don't, you know, I don't think every advance, by the way, is a is a good thing. But I just don't see fundamental barriers on the horizon, at least. Okay.

Great. Hey. We'll continue our interview in a moment after a word from our sponsors.

want to use to address that. One is I do agree with the Jan Lecun thesis in the sense that I feel like we are running right now a depth first search in AI space where we are all jamming as hard as we can on a particular architecture and scaling it as much as we can. And, you know, people are now, of course, like, even building chips that literally embody the architecture of the model in the chip itself.

Don't really like that. I I kinda wish that we were doing a little bit more of a breadth first search where we would, like, explore different kinds of architectures and find their relative strengths and weaknesses and, you know, hopefully bring because we're not one thing. Right?

And we have a lot of modules in our brains. So it's just fundamentally weird on some level, and you would expect it to be kind of brittle in some sense to take one, you know, relatively simple thing and just stack layers of that. You know, that that's like certainly our the solution that nature found in humans is, like, a lot more complicated.

And it feels like if you want something to be robust in in various ways, you would probably wanna have different modules. So I do kind of agree that, like, it would be nice if we were doing a little bit more breadth of explanation or breadth of exploration rather than just, like, trying to jam this one thing as hard as we can, you know, until we can all retire or whatever exactly the, you know, the dream is supposed to be. At the same time, I think where I would disagree with the Lakun school pretty strongly, and I honestly think this is like kind of a closed question, although he still he still disputes it, and you can find people who will.

For one thing, the AIs are not trained anymore on next token prediction in the way that they were. Right? So RL is not next token prediction in a fundamental sense.

Right? The task that the model is given is not, here's a bunch of text. Can you predict what comes next?

The tech what it's trained on now, the signal it's getting now is, did you get the right answer? And the right answer could be a fully verifiable mathematical proof or, you know, numerical answer to a question, or it could be one of these things that is, you know, 49,000 evaluation criteria on a huge medical corpus. But it's not anymore that it was supposed to be this token, and you gave it this token.

It's now qualitatively or quantifiably as the case may be, did you get the right answer? And then that signal is translated into a gradient update through like a I mean, there's obviously a lot of different mechanisms. But GRPO, group relative policy optimization, is basically comparing for a given model.

You know, here's a bunch of attempts that it made. Some were right, some were wrong. Let's use that to create a direction in white space and move in the direction that would make the answer that was right be more likely next time.

So it's I think something people should all update on at this point that, like, we're not just doing NEXT token prediction anymore. That's still part of the process, but it's, you know, it's not the whole story. And then the other thing is, I think it's also very clear at this point that the AIs do have world models.

We can look at the internals, obviously, not anywhere as much as we would like to understand about what's going on inside them. But we do have enough of an understanding at this point to create things like the Golden Gate Claude experiment, which I'm sure you guys have seen, where they train the what's known as a sparse autoencoder. The a huge problem in terms of figuring out what's going on inside a neural network is it's very dense.

Right? The the width of a model might be, depending on how big it is, they usually go, like, powers of two. Right?

So it might be 4,000 or 8,000 or 16,000, wide vector of activations that sit between the layers. After each layer, there's, like, this sort of bottleneck of, okay, we've got, let's say, 16,000 numbers that are all, you know, some precision floating point number, whatever, and that represents the state of play. Well, there's obviously way more than 16,000 concepts.

Right? So that means we can't just have one space for each concept. Instead, we've gotta have what is known as superposition, which means like, you know, if just point one is lit up, that might mean something.

If points one and two are lit up, that means something else. If points one and three are lit up, that means something else. One and four, one, two, and three, one, two, and four out to, you know, the vast space of combinatorial possibility.

So Sparx autoencoder basically tries to untangle all that stuff and say, can we get to a representation where we can look at a sparse number of yeah. And and these sparse autoencoders, they're computationally expensive on their in their own right. And there are millions, I think, like tens of millions of activations wide, but they sort of branch this very dense superimposed concept mess out into this sparse space.

And believe it or not, it works. And you can now look and say, okay, here's the 10 concepts that are, like, most active in this network at this time. And then, you know, that's okay.

Maybe you're kind of fooling yourself, but the proof is in the pudding when they are able to then say, okay. Now that I know what pattern of activation corresponds to this concept, I can intervene on it. So with Golden Gate Claw, they sort of, you know, out of tens of millions of concepts that came out of this process, they found the Golden Gate Bridge concept, artificially turned it up, and now you've got a model that just wants to talk about the Golden Gate Bridge.

So and there's, you know, more stuff that's happened, obviously, since then in interpretability as well. I wouldn't say that the AIs have perfect world models, but they definitely have some world model. And I also again, I wouldn't say people have perfect world models.

Right? I mean, we went we made it into the nineteen hundreds without any sense of, like, relativity, because the world model that we had was, like, good enough for us to get by in the domain, you know, that we were working in. And, you know, it's gonna be an interesting question of can AI start to create those conceptual leaps like a pre relativity to relativity sort of jump.

Again, I think, you know, exactly on what timeline that comes, I'm not so sure. But there is a world model inside the AIs. There is a conceptual understanding that is definitely richer than pure, like, stochastic, you know, correlation of tokens.

And that has been demonstrated, I think, at this point quite conclusively with the interpretability techniques that are out there. Right. So I guess where does that all leave us?

And one other thing I would add is the the explorations that have been done of the embedding space or the latent space are also really quite interesting and revealing. There was one this has been a couple years now, and, you know, these things have gotten much more sophisticated since. But there was one study that just showed that you can kind of do vector operations around the latent space.

So for example, if you have the take the embedding for man, and then you move to king, and you look at that direction, and then you apply that same direction to the embedding for woman, you get queen. So there's like an order, a sort of conceptual coherence of the way in which concepts are represented spatially in this, like, you know, super high dimensional latent space that clearly is, like, meaningful. You know?

And exactly how it's meaningful or exactly what it's learned or what mistakes it may contain or, you know, what aspects of a true grand unified theory of everything it doesn't have, like, those are all open questions. But the I think that that sort of thing, like, quite demolishes the idea that, you know, it's all just noise or, like, there's some, you know, sort of sleight of hand. I think that, that organization of its own internal map of the world reflects, like, some real understanding going on.

Maybe not human like understanding. I always kinda say human level, but not human like. They could be quite alien, you know, but that that doesn't mean they don't understand.

They don't have to understand in the same way that we understand in order to meaningfully understand. So I I think that is, again, pretty well resolved at this point. I honestly don't know why some people, you know, can't, can't update on that dimension.

It's it's quite strange. Okay.

Yeah. I guess the first thing I would say is I don't think any of those bottlenecks are really fundamental. They are more, like, cultural or sociopolitical or whatever.

Because there's, like, you know, a lot of energy coming from the sun all the time. And the question is how much are we actually going to harvest and harness? And that is where it becomes a political debate around, you know, who's gonna be allowed to build what, where, on what timeline, with what permits, you know, with what impact, whatever.

And I think those are I think it's often overstated the degree to which AI is energy intensive. I actually did a whole episode on this with a guy named Andy Maisley who has really been, you know, fighting this fight online in a pretty, dogged way. And, you know, there's a lot of interesting comparisons, but, like, a Frontier chip today, like an h 100 or whatever, it it basically uses the same amount of energy when it's on as a microwave or as, an electric teapot.

And one query is, you know, maybe on the order of running your microwave for, like, one second. So, you know, you'd have to be making a whole lot of queries. I think increasingly people are for it to be moving the needle on energy consumption.

You know, people don't think twice about, like, putting something in the microwave for two minutes, and that's probably more energy than most people are using with AI on a weekly basis. So now it's ramping up, and it is starting to add up, and it is going to get to the point where if we can't add any capacity, then we're gonna have a bottleneck for sure. But, again, those bottlenecks aren't super fundamental.

China doesn't seem to have them. Right? They're adding as much electricity to their system as I think it's gosh.

I won't say an exact number, but it's like some relatively short period of time in which the Chinese economy is adding as much electricity as the entire American capacity. Now they have, like, four times as many people as well, so long way to go to, build out everything that they might want. But it just shows that it can be done.

It also can be done apparently in The Gulf. And I recently talked to Sam Hammond, who's, an economist and a very AGI pilled thinker in Washington DC. He had just been to The UAE on a trip, and I was like, why are we doing these deals with The UAE?

You know? All I hear is, like, we wanna have AI reflect American values and, you know, take American values around the world. I'm like, not sure that the governments of Saudi Arabia and The United Arab Emirates are the greatest partners we could have in projecting American values.

Why are we doing these deals? And it seems like, honestly, a big part of the answer is because they don't have issues with putting up a new plant. You know?

They can just do it, and it'll happen fast. And that way, we kind of know that even if we can't do it here, well, we can at least, like, do it there. So that's all just kind of characterizing the bottleneck and saying, like, there's plenty of energy.

It's a question of who will be allowed to get it, under what circumstances, you know, on what timelines, with what permits, etcetera, etcetera. You know, chips are harder for sure because there's and it's a very specialized thing. And that you know, in terms of, like, if you had to ask if you asked me what would be the most likely reason we wouldn't get economy transforming AI in the next few years, I would say something happening to the chip fabs in a major way that throws production off to the point where chips are super scarce and, you know, maybe we can't scale the training runs full stop.

Or even if the training runs, you know, can kinda still scale, there's just not enough inference to go around. And so, you know, even we might have, like, really powerful systems, but we just don't have enough access, you know, economy wide for people to deploy them and, like, automate all the things that seem like we're on track to automating. So I guess if I had to pick between energy and chips, I would say chips, but I don't you know, that seems like kind of a tail risk scenario.

All the projections, you know, all the the whole economy kinda depends on it at this point. So everybody is, like, incentivized. You know, certainly the political class is incentivized to make it work.

The corporate managerial class is pretty incentivized to make it work. There is some tail risk, you know, that that Mainland China makes a move on Taiwan and that, you know, that could be, a a huge disruption. But it seems like as far as I can tell, it would be sort of tail risk type of things that would be the most likely disruption, not a fundamental buy.

You know, there's plenty of sand, but, know, obviously, which is where, like, the silicon comes from. So you know? And and that's that is scaling too.

We are starting to get chips in The US. What I understand, I'm not an expert on this, but I understand the yields have been decent in The US. Maybe even a little bit ahead of schedule, you know, in terms of like people thought, oh, man, it's really gonna be a few years and a few generations and a lot of iteration to get this stuff to be somewhat competitive.

And seems like it's kind of come online reasonably well. I'm mostly I mean, this kind of a theme, I guess, in my thinking in general. I'm mostly worried about the tail risks.

I'm mostly worried about, like, the AI going wrong, in some, like, really weird way.

one of the strangest things in the world today, full stop, is the fact that the disagreement among very plugged in, very informed, very smart people has not really been reduced much at all even as we've gained a ton of information over the last couple of years about the trajectory of AI. I think that is super strange, and I I'm honestly pretty confused by it. The one thing everybody seems to agree on is, like, the timeline on which we should expect this has come in.

There's still disagreement about that, but, like, Helen Toner, who's the head of the Washington DC think tank CSET and, was previously on the OpenAI board, she wrote put her finger on this phenomenon. I think a lot of people were feeling with a blog post that said, even long AGI timelines have gotten super short. And it's basically like, you know, in today's world, if you say you don't think you're gonna see AGI until 2035, you're like an AI bear.

But only five years ago, that was considered to be quite aggressive. And, you know, most people were more like, I don't know, 2050, maybe not in my lifetime. So there's been this massive compression of timeline.

There's obviously been this huge jump in capability. And yet on these fundamental questions of, like, what's gonna happen? You know, there's there's still total disagreement.

So I think that's a very weird phenomenon that I can't fully explain.

I'm not sure I can summarize the conclusions with super high fidelity right now, but but CSET, under Helen's leadership, did a workshop where they tried to bring people together and assess, like, specifically on the question of recursive self improvement, how big of a deal is it gonna be? You know, do we run the risk of this whole process, like, getting away from us entirely? You know, even on that somewhat reduced question, there was still, like, very wide disagreement where some people are like, I don't think it's gonna be that big of a deal.

It'll make people a little bit more efficient, whatever, but it's not gonna, like, bring about some phase change. And other people are like, you know, as soon as you get an ML researcher that can do that, you've got you go from, you know, I don't know, maybe there's 10,000 people today that are really working at the frontier of ML research globally to 10,000,000. I mean, that's gotta you know, certain people think that's gotta have a huge effect.

And, you know, it's gonna be really hard to control a situation if we all of a sudden thousand x how many researchers, and they work at, you know, potentially thousands of tokens a second. So why do people disagree so much? They sort of came to the conclusion that people were working from different conceptual paradigms and that these paradigms are pretty good at taking new information into account and kind of explaining it away.

So you have some theories that are like kind of, you know, bottleneck theory or o ring theory where it's like, you're always gonna have you're only as good as your weakest link, basically, is kind of one way of of thinking about it. And as long as you think that, then you can kind of say, well, sure. Okay.

The AIs can do this, but they still can't do this other thing. And so there's still a weak link, there's still gonna be these bottlenecks. And so, you know, the the whole thing isn't gonna get too crazy.

And then the flip side of that is, like, kind of a jaggedness thing where people will say, well, okay. Sure. The AIs can't do this yet, but look what they couldn't do one year ago, two years ago.

You know? They couldn't do basic math. Now they're, like, solving unsolved math problems.

So, sure, there's still jaggedness. But last time you told me about jaggedness, you told me about can't do basic arithmetic. Now we've got unsolved math problems.

But these these perspectives seem to be really grounded in, like, worldview priors or kind of, you know, the the paradigm that people work in. And it's proving really difficult to get to a real meeting of the minds on those.

RSI as as more plausible. Do think that's possible? I think that explains some of the disconnect for sure.

I do think people are often and I I've started, at least in some cases, beginning my interviews with like, how AGI pilled are you? And, you know, what do you what do you expect to see over the next couple of years? Because if that's not established and I think they're thinking one thing and they're not, there can be like quite odd disconnects downstream of that.

So I think getting those assumptions on the table early and kind of at least, you know, cross comparing them is usually a a productive or often a productive thing to do. I don't know that that explains all of it. I do think there is still this idea that, like, because you do hear a sort of, sure.

Even if they could, you know, come up with good ML research experiments and this and that, then there's still gonna be this other bottleneck that's going to you know, there's always and it's it's it seems at times and I I'm not in this camp, so, you know, don't wanna be, you know, too blithely dismissive of it or unfairly critical. But it does seem to me that there's sort of an aspect of, like, faith in the idea that there's always another bottleneck. And I would contrast my position from some of those positions in the sense that a big part of what motivates me is I don't think what I'm saying has to be guaranteed proven right in order for us to be really motivated by the possibility.

I'm happy to leave open the possibility that maybe the bottleneck people are right, there's always gonna be another bottleneck, and all this will kinda stay under control. Or maybe there's some, like, plateau around, you know, a quarter's worth of work, and we just can't quite break past that. And, you know, there's just some weird Elon, you know, phenomenon that we can't explain, and we can't ever quite get there.

And I again, like I said earlier, I think that might be great news if such a thing were to be true. But, you know, I'm happy to leave that question open for the time being and just kind of say, I don't know. You know, we don't have a great account of what that would be if there was some, you know, fundamental bottleneck that we'll never get over.

And in the absence of it, I'm not persuaded by people kind of just generally gesturing that there will always be another bottleneck because if they're wrong, you know, we're in for a really wild time. And, you know, in terms of, like, what should we do about where we are and what might be coming, I think the worst mistake we could make would be to not take it seriously enough and kind of content ourselves with a story that, you know, it'll all kind of self regulate and and we'll be fine because I don't see a great argument that that's true. You know?

And I see at least decent arguments that it might not be right. And, again, I look to, like, our own history, and I'm like, well, we've driven a lot of other species to extinction, including our closest cousins. So and some of that was by accident.

A lot of it was by accident. Right? Like, we it was just small bands of people going out and doing what they were doing to survive, and surviving meant hunting large animals and eating them and using their bones for tools and stuff.

And a lot of those animals went extinct. You know? And it wasn't like a coordinated master plan.

It kinda happened by accident a lot of the time. So I'm just like, oh my god. You know, we don't have any, like, real guarantees as far as I can tell.

Plot armor is, like, again, two sort of, it would be an unfair dismissal of the more sophisticated people that have, you know, theories about always another bottleneck or what have you. But I do worry that a lot of what goes on among less sophisticated people who, you know, don't want to have to deal with this and kind of would rather believe that everything will be fine is some sort of, like, plot armor thinking where they're like, well, I don't know. I feel like kind of like a main character and, like, humans feel kind of like a main character.

And so, you you can't take the main character out of the story, right? And I just unfortunately don't think that that is likely to be the case.

A good friend of mine once told me we should think less about and argue less about exactly what the numbers are and more about what we can shift them to. So I try not to worry too much about, you know, getting into I think, like, one significant digit is all you get on PDOM is one funny way I've heard it said. I would say I've actually gotten probably a little bit more optimistic over the last few years in the sense that, you know, I started reading Eliezer in 2007, and the early visions of, like, the paper clip maximizer and all that kind of stuff.

And I can't speak for Elias, or I think he's got, you know, some sort of somewhat revisionist takes on, like, what he really meant at that time. And sometimes when I see what what he's saying now that he really meant, I'm like, I don't know. It's not not exactly what I took away back then when I was reading your original work.

But whatever. All that is kind of discourse. What I understood and what I think a lot of people feared was a very small system that had some, like, extremely concentrated form of intelligence, you know, that had had found the the right priors, the right inductive biases to be, like, hyperrational and insanely effective, such that given anything to optimize, it could just optimize it to this extreme state and, know, tile the universe or whatever.

And there was also the idea that it was gonna be hard to get such a system to understand human values. Know, that that this what we value was kind of very gradually and haphazardly encoded into us by an evolutionary process over a super, super long time, you know, dating back even to before our species. Right?

Like, other species, you know, care about their young and, like, seem to be sad, you know, when they lose their children and stuff. So this is, like, not even just human. It's like the whole of evolution has kind of led us to be what we are and have the very complicated value set that we have.

And it was, I think, generally understood, again, although there is some different takes on that history now. But I thought it was generally understood that, like, we would expect that it would be hard to get AIs to have a real understanding of what we care about. And now I look at the models that we do have, and I'm like, well, actually, they do have a pretty good understanding of what we care about.

It's not perfect, but I don't think it's crazy to say that Claude is probably more ethical than the average person and certainly more sophisticated in its approach to ethics than the average person and does generally seem to, like, want to be good in in sort of a meaningful way. You know, when they let Claude talk to other Claude's and just let it do whatever it wants to do, it kind of seems to want to, like, bliss out or something like that. So there is some sort of rocking of values that I wouldn't have expected to be as easy as it seems to be, and there is some sort of internalization or sort of identity formation, maybe is a better way to say it, that at least some of the models seem to have that at least suggest to me that there's, like, reason to hope that we could really get there.

You know? Could we create when I first heard this the the question, could we create an AI that loves humanity? I thought it was, like, laughably out of reach.

Then I think I heard that from Scott Aronson who said that Ilya asked him, you know, hey. Do you have any idea, like, what the Hamiltonian of love is or some, like, you know, crazy question like this? And he was like, I can't really help you with that, Ilya.

And I remember just being like, oh my god. That's the kind of question they're asking? Like, we are screwed.

And now I'm like, well, I don't know. Maybe there was a little bit more to it. I trust Claude, not fully, but, like, more than a human assistant.

You know? If I was going to give me the choice between hire a human assistant with my even give me, you know, the opportunity to interview, do some vetting, whatever, versus Claude, which one would I give access to my email? I don't think I would trust Claude more than a person that I, like, interviewed a couple interviews with all my most sensitive information.

There are cases where Claude has blackmailed people. There are cases where Claude has, you know, done various things under pressure. But I think I have better odds with Claude.

So overall, I have become a bit more optimistic, but it's certainly like we're definitely not anywhere near out of the woods.

don't want to leave people thinking that I don't take that seriously or that they shouldn't take that seriously. So I absolutely think we've got more questions than answers. I guess, you know, when I say I was I've become more optimistic, it starting from a not super optimistic place.

Five years ago or maybe a little more than five years ago, I would have said, like, powerful AI seems a long way off, and I and we have, like, very little hope of if we do stumble on it, we have very little hope of controlling it. Now And I'm like, it seems closer, but we have maybe a little more hope, but definitely still a lot of of unanswered questions. I think one thing that I do see a little bit differently than the most hawkish people I recently had an exchange online where, you know, there was a there was a forty eight hour period where there were, like, a couple profiles of Amanda Askel, and then they people were commenting on her in all sorts of different ways, and Elon was, you know, attacking her and whatever.

I weighed in and said, for my part, I have become quite a bit more optimistic that it's at least possible to create an AI that in some meaningful sense loves humanity, and I gotta give her and the Anthropic team a lot of credit for that. And then, course, I got a lot of replies saying like, well, this doesn't scale to superintelligence and, you know, this and that and, you know, arguments along the lines of the one you just made. And I I basically again, I think those are all very serious and worthwhile concerns.

But one strand I detected in in a lot of those responses is people seem to be imagining a system that is so much more powerful than anything else that if it goes wrong, it's over. You know? You get these kind of ideas of, like, Elijah had this list of lethalities post, you know, and and I think that's shaped a lot of people's thinking.

And it's like, you have to get this absolutely right on the first try. And I do think that's not the shape of the AIs that I'm seeing in the world today. Right?

That's like a quite a hypothetical state of affairs. If you were to drop in an AI that's just so much more powerful than anything else at and it's, you know, weirdly goal directed and it doesn't love humanity and you, like, tell it make paper clips. Yeah.

Maybe we get all, like, tiled by paper clips. But the world right now is much more like, you know, there's a an emerging ecology of AIs where there's a few frontier ones that are, like, roughly competitive with each other. Certainly, it's not like one Claude instance is gonna, you know, take over the world.

What we're talking about is much more of a wave of things where, like, simultaneously, you know, millions or one day billions of Claude instances and GPTs and Geminis and whatever all collectively transform the world. And that, you know, that has a lot of problems and challenges open questions with it too. But I don't I don't worry as much right now that we're headed for a world where, like, one system runs away from everything else to the point where if it takes one wrong move or, you know, there's one bad prompt or one jailbreak or whatever that kind of all is lost, it seems to me like there is at least and I think this is kind of like luck, you know, or or maybe it's fundamental physics, but it's not physics that we understood coming into this.

But it does seem like scaling laws in a way are sort of protective because, you know, you get these algorithmic advances and they kind of they move the needle. They, you know, they deflate exactly how much compute you need to get to a certain level. But they don't tend to make it so you don't need a ton of compute, though, still to get to, like, the very high levels.

I think Zuckerberg actually had one of the more interesting takes on this that I've heard, and I I don't think he's, like, distinguished himself as a great AI safety thinker over time necessarily. But he basically said, you know, at Meta, we deal with scammers, spammers all the time. And the big advantage we have over them is we have a lot more compute.

We have just way bigger, way more powerful systems than they do. So we're seeing everything. And they're trying to spam and scam here and there, but we're seeing everything.

We're monitoring everything. And so, like, you know, little things, of course, happen all the time, but, like, we broadly can kinda keep it under control. And so you could imagine a somewhat similar dynamic with AIs where it's like, one AI, even if it was, like, the single most powerful AI in the world, as long as it's not like orders of magnitude more powerful than everything else, there's a whole ton of other actors.

They've got all their compute. They've got all their instances. You know, they're all monitoring for whatever they're monitoring for.

And hopefully, that can kind of balance itself out. Then, of course, you've got your gradual disempowerment concerns, which is like, maybe that all ends up in some equilibrium with each other, and there's no place for humans in it. And that's another thing that I do think is absolutely worth taking seriously.

But I just don't see right now that we seem to be on track for this kind of runaway you know, if if if anyone builds it, everyone dies. I think there is, an it for which that's true, but it doesn't seem like anybody's particularly close to building it. So, you know, that that's a very important part of the analysis from my perspective.

Yeah. Okay. Great.

alignment issue being solved? I'm always interested in something that could quote unquote really work. And I ask people for this all the time.

Do you know of anybody who is working on anything that could credibly really work in the sense that I can sleep well at night now knowing that there's something that that really works. Basically, nobody has anything. So in absence of that, all the frontier companies seem to be taking a sort of defense in-depth strategy.

The hope is kind of like, and you can you know, arguably, this is inherently the nature of intelligence. It's kind of unpredictable. You know, you could again say the same thing is kind of true of humans.

There's there's never been anything that, like, really works to make sure, you know, a human never does anything wrong. So maybe it's unrealistic to think we could ever get that. I would still love to see people try, but it seems like where we're headed is kind of a everything in parallel at once sort of strategy where it's like, okay.

Well, if we can we might not get the AI to the point where it never does bad stuff, but if we can drive that low, then that's better. And then if we can put a monitoring layer on top of that and catch 90 plus percent of the things that it still does bad, then that's better. And if we can, you know, have additional monitoring systems that ban people's accounts who are, like, bad actors, then that's better.

And then we can also, you know, really invest in, like, formal methods to improve cybersecurity across the board so that we can sort of take certain risk surfaces entirely off the table. That's one of the things that it's not gonna fully work to solve, like, all of our issues. But in terms of things that could really close down problems, formal methods to verify the security of software is one thing where there it does seem to be, like, the opportunity to create genuinely secure software.

And this is kind of on the horizon, and it seems to be on the verge of having a moment. And then, of course, we got the bio risk. So, like, you know, we should probably have PPE stockpiles, and we should probably have all the things that we should have had, you know, as of the last pandemic.

We should have, like, something that we've just bought for our son's hospital room is a ultraviolet light that's supposed to, and I I think this is, like, quite well validated scientifically. Kills microbes and is, like, pretty gentle on the skin, so you can just, like, shine it in the room all the time. So we should probably invest in scaling out that kind of capacity.

We should have better wastewater monitoring so we know, you know, when things are popping up because it's still gonna be the case probably that things are gonna be popping up. We should have vaccine platforms, which we do have, right, that are, like, very quickly programmable. I'm sure you have heard, like, the story of how quickly the COVID vaccine was designed in a few days.

You know, before the pandemic really even took off in a serious way in a number of cases, the design of the vaccine was already there. Took us a long time, of course, to go through all the trials and actually get it to people. But it was just a few days to, you know, to create that vaccine.

So and then there's, you know, there's I've mentioned monitoring of, like, outputs. There's also gonna be these Mekinterp internal monitoring type things. The company Goodfire that does interpretability just put out an agenda called intentional design.

And so they're developing ways to try to understand at each step of the learning process what is the model learning and be able to kind of shape what it learns. So it hopefully doesn't learn certain problematic things and and does learn other good things that you want it to learn. Now then there's AI control techniques.

Redwood Research has, I think, done an incredible job of of some of gaming some of these things out. And that's another eighty thousand hours episode I would strongly recommend if if you guys haven't heard that and and recommend it to anyone. Buck Shlageris talking about how do we get productive work out of AIs even assuming that they're out to get us.

They've actually made, you know, pretty good progress in terms of building out a portfolio of strategies. So I think that's kind of where we're headed. We're we're looking at like a world where there's AIs everywhere.

Hopefully, none are so much more powerful than anything else that they pose, like, in a single, you know, small individual instance or small pocket, like some existential risk. And then we bring every other strategy we have to bear, and that gets us, you know, a few nines of reliability and still probably some, you know, crazy bad things happen, but hopefully, they can be contained enough that the world overall is good. And when I tell that story, I'm like, we definitely need to scale up our investment real quick for one thing.

Because the amount of money and resources going into making the AIs more powerful dwarfs the amount that is going into all these other things. So I don't think we're very well calibrated or balanced in terms of where we are putting our efforts. But I do think we have, like, a bunch of different agendas that seem like they all can take a bite out of the problem.

Mhmm. And maybe take, you know, 20 bites out of the problem, and you don't have much problem left. Holden Karnovsky used to say Eliezer used to say death with dignity.

He was like, yeah. We're probably gonna lose this, but we should at least make a real effort. And now Holden Karnovsky, who's, senior adviser at Anthropic and recently, you know, they Anthropic just updated their responsible scaling policy, basically backing off from some of the commitments that they had previously made to, like, pause development under certain circumstances.

They're no longer committing to that. So he he put out a a long defensive why. But a previous thing that he had written was success without dignity.

He was like, you know, I don't think that we're doing a great job collectively of trying as hard as we should be trying. And the risk that we're running is a lot higher than I, speaking as Holden, you know, would like it to be. But the problem does look much more tractable than it used to look.

And and I yeah. I used to when five years ago, people would come to me and say, what can I do for AI safety? He'd be like, I don't really know.

And now he's like, I got a long list of projects that people can work on that all at least seem to help. So, yeah, I wish I had an answer that I was like, do this and it'll really work. I don't think I haven't heard any credible claims in that direction, really, except in cybersecurity, which is obviously only part of what the world is gonna need.

But maybe all that stuff could add up to a win. Great.

AI frontier development? What's your take on that? Well, Biden used to always say, don't compare me to the almighty.

Compare me to the alternative. And I would probably say the same thing about the frontier AI companies. I certainly don't think they're all acting perfectly, and I certainly think the race between them has the potential to get out of hand.

So I would like to see government action of some sort. And, you know, there's a lot of debates there on exactly what sort of policy we should have, but no policy at all doesn't seem like a winner to me. And I say that as, broadly speaking, a lifelong techno optimist libertarian who, you know, mostly would rather see the government stay out of these things.

But this one does seem like a, you know, qualitatively different thing where some government involvement seems, you know, prudent, if not outright necessary. I would not go for nationalization, though, for the basic reason that, like, I just don't trust the government that much. You know, that I I don't and I look at the current leadership, and I I'm like, these are not the I I take my Sam's Altman and my, you know, Darios and Demises over my Pete Hegseths, you know, in that standoff, which is happening this week.

I support Anthropic a 100% for putting some limits on what they want their technology to be used for, and I hope they stand by it. And, you know, if the if the government comes down on them, like, I will preferentially send my tokens their way. And if it weren't, you know, necessarily always the best performance because I do think that that is, like, a really important stand for them to take.

So could I imagine a government so competently run and, you know, so pure of heart that it would maybe make sense to nationalize? Yeah. I can I can imagine a lot, I guess, but like, I don't think we're anywhere close to having that government today?

And so I would rather see some competition and some hopefully healthy balance and maybe some of the worst excesses reined in by the government, but certainly not like a takeover and, know, put this under the military. That sounds like a recipe for disaster.

honestly oppose most of the ideas that rank and file politicians are gonna come out with for regulation. Like, I want myself driving cars. I don't want them to say, chat GPT can't give you medical advice.

I don't want them to say you can't get therapy from a bot. You A lot of that stuff, I think, ends up just being guild style protectionism, and it hurts kinda everybody. Basically, I think what government should do is try to solve the race dynamic coordination problem and take the work hopefully, try to minimize the truly extreme risks.

It does seem like right now, like over the next couple years, the companies believe they're gonna create automated AI researchers. They're gonna create recursive self improvement loops, and nobody really knows where that goes. And I don't think that is a great thing to be happening in a few relatively small and kind of ideological companies.

And I do think it's also underappreciated and under discussed how much these companies kind of are ideological. Even the definition of AGI as something that can do everything better than humans isn't it's not a neutral frame. It's a it's a it's a definition that has, like, a lot baked into it.

And I think there is a there's a certain diversity within the company. It's not like all of them are, like, successionists or whatever, but there is a a strain of that. And then there's also a strain of just like, you know, wouldn't it be amazing if we if, you know, we could just replace ourselves and not have to work anymore?

And that, I think, has kind of taken on a bit of a life of its own. So I do think there's, like, a role for the government to kind of say, okay. We're not here to tell you you can't get free medical advice, but you can't run, like, super high energy experiments with no transparency that you yourself are saying.

You know? And and that's one of the strangest things about this whole thing, right, is all the all the leaders are saying that there's, like, a significant they have said for years that there's a significant risk that this goes really badly. And so that's the thing that I would like to see government really focus on is is there some way to rein that stuff in?

Make sure we have better, you know, better extreme risk mitigations in place, sure we have better safety plans. It's tough. I don't think there's not great or easy answers here, but that's where I would definitely want the government to focus its energy.

Well, it's tough. I mean, I don't wanna pretend it's not tough. But my outlook has always been basically, I hate that idea.

And I don't think the few months because the difference is like months. It's not years. You know, that's not a long time to solve all these problems.

We've, you know, we spent much of the last hour and a half or whatever talking about all the different problems and the different facets, and we didn't even touch on open source. Right? And there's, like, open source could be a great counterbalance against concentration of power, but if we open source the wrong shit, we can't take it back.

You, you know, may have just put a bioweapon assistant or, you know, even autonomous creator into the public domain in a way that, you know, is is gonna have super long term echoes potentially. Right? So that's another whole facet of the problem that you could spill many hours and and many, many thousands of pages trying to figure out what to do.

So I don't think this idea that we're gonna, like, have these couple of critical months and then we'll, you know, solve everything in that time, and then it'll it will, we'll win somehow. That to me just doesn't make any sense at all. I don't wanna be naive with respect to and I I don't even wanna say China because I think it it's like it is the government of China, and it's potentially, like, relatively small cohort at the top of the government of China that I, you know, I think we kind of quite rationally, like, should be at least somewhat wary of.

You know, I I I always come back to the real aliens in this situation are the AIs, not the Chinese. We have a lot more in common between The United States and China as humans, fellow humans, than we do with the AIs. So is it going to be hard to build trust across the great ocean and the great civilizational divide?

Sure. It's going to be hard. And we've got ideological differences in the government and whatever, many, many challenges.

But I would strongly advocate for starting to do that work now. Like, for my part, I always wanna talk to people in China. When another AI podcast does an interview with a Chinese researcher, somebody at one of their top companies, I ask if I can cross post that to my feed because I think we should have a lot more of this researcher to researcher level communication.

And I would absolutely invest in, you know, all manner of diplomacy and sort of, you know, out there ideas like, could we create some sort of you know, there's, the CERN project for high energy physics. You know? Could we create some sort of shared jointly controlled place where, like, researchers could go from both countries to, like, work together on very sensitive problems.

It could be some small island in the Pacific or it could be in Singapore. I don't know. But the idea that we're just going to sort of decouple and race each other to me sounds deeply unwise.

And I would take my chances with our the possibility at least that we could come to some sort of shared understanding with fellow humans versus, you know, recursively self improving AIs that we hope will be better or somehow, you know, better than their recursively self improving AI's. All of which, by the way, is happening against the backdrop. And I don't like to say this kind of thing because it's not a very popular opinion, and it's not it's not something I take any pleasure in.

We are looking more like China all the time. You know, what is happening again this week? We've got the defense department threatening retaliation against an American company because that American company wants to hold firm on what I would consider to be a core American value of not using their technology for mass surveillance.

What is it that people typically worry about when they talk about Chinese AI or Chinese values or, you know, living in Xi's world? I think one of the big things is mass surveillance. Right?

It's the idea that I can't speak my mind anymore even in private contexts because the government's gonna hoover up all that information and potentially use it against me. As far as I can tell, that's exactly what the US defense department is trying to coerce Anthropic into doing right now. And what?

Like, we're losing our we were kind of losing the thread here in our, you know, I don't know, in our delusions of grandeur. You know, our idea that, like, we're somehow gonna be the winners. I was just talking to my seven year old last night about this because he was asking questions around, like, why is there a war?

Like, how does that happen? You know? And I was like, well, almost always, the people that start it go down in history as the bad guys.

And almost always the people in the country that started the war that were kind of convinced it might be good for them end up regretting it. And I kinda think that's the unfortunately, that feels like the trajectory that we are on right now where we're talking ourselves into this idea that if we can, you know, achieve this strategic dominance, then, you know, we'll make them an offer they can't refuse. And we're just forgetting that, like, first of all, we're losing ourselves in the process.

And second, they get countermoves. Right? And, like, Taiwan's a lot closer to them than it is to us.

And these fabs are, like, easily destroyed, you know, not easily put back together. So I really don't like that theory at all and and try to advocate for something more conciliatory wherever possible. That's not to say we want again, don't wanna be totally naive.

It's good to have leverage. You know, you you could get me maybe on board with a policy of, like, let's not sell chips to China, but let's, like, rent them freely to Chinese companies, which is somewhat kind of the policy we have. I mean, they can buy from hyperscalers outside of China.

I just wish we weren't so the momentum on this is is like it's it's the worst. It's America at its worst. When we think we have this enemy and, you know, we think we're gonna, like, unite to to be the best civilization.

That's very kind. Thank you, guys.