Dario Amodei — “We are near the end of the exponential”

the underlying technology, like the exponential of the technology, has gone broadly speaking, I would say about as I expected it to go. I mean, there's like plus or minus a couple there's plus or minus a year or two here. There's plus or minus a year or two there.

I don't know that I would have predicted the specific direction of code. But actually when I look at the exponential, it is roughly what I expected in terms of the march of the models from smart high school student to smart college student to beginning to do PhD and professional stuff, and in the case of code, reaching beyond that. The frontier is a little bit uneven.

It's roughly what I expected. I will tell you though what the most surprising thing has been. The most surprising thing has been the lack of public recognition of how close we are to the end of the exponential.

To me, it is absolutely wild that have peep within the bubble and outside the bubble, but you have people talking about these just the same tired old hot button political issues and, like, you know, around us. We're, like, near the end of the exponential.

What is the scaling hypothesis at this point? Yeah. So I have actually the same hypothesis that I had even all the way back in 2017.

So in 2017, I think I talked about it last time, but I wrote a doc called the big blob of compute hypothesis. It wasn't about the scaling of language models in particular. When I wrote it, GPT-one had just come out, right?

So that was one among many things, right? Back in those days, was robotics. People tried to work on reasoning as separate thing from language models.

There was scaling of the kind of RL that happened in AlphaGo and that happened at DOTA at OpenAI and people remember StarCraft at DeepMind, the AlphaStar. So it was written as a more general document. The specific thing I said was the following, and it's very Rich Sutton put out the bitter lesson a couple years later, but the hypothesis is basically the same.

So what it says is all the cleverness, all the techniques, all the kind of we need a new method to do something like that doesn't matter very much. There are only a few things that matter. I think I listed seven of them.

One is like how much raw compute you have. The other is the quantity of data that you have. Then the third is kind of the quality and distribution of data, right?

It needs to be a broad distribution of data. The fourth is I think how long you train for. The fifth is you need an objective function that can scale to the moon.

So the pre training objective function is one such objective function. Right? Another objective function is the kind of RL objective function that says like you have a goal, you're going to go out and reach the goal.

Within that, of course, there's objective rewards you know, like you see in math and coding. And there's more subjective rewards like you see in RL from human feedback or kind of higher order versions of that. And then the sixth and seventh were things around kind of like normalization or conditioning, just getting the numerical stability so that the big blob of compute flows in this laminar way instead of running into problems.

So that was the hypothesis. And it's hypothesis I still hold. I don't think I've seen very much that is not in line with that hypothesis.

And so the pre trained scaling laws were one example of kind of what we see there. And indeed, those have continued going. Like, you know, I think now it's been widely reported like, you know, we feel good about pre training.

Like pre training is continuing to give us gains. What has changed is that now we're also seeing the same thing for RL, right? So we're seeing a pre training phase and then we're seeing like an RL phase on top of that.

And with RL, it's actually just the same. Even other companies have published in some of their releases have published things that say, Look, we train the model on math contests, AIME or the kind of other things. And how well the model does is log linear and how long we've trained it.

And we see that as well. It's not just math contest. It's a wide variety of RL tasks.

And so we're seeing the same scaling in RL that we saw for pre training.

several things that should be kind of thought of thought of differently. Yeah. I think there is a genuine puzzle here, but it it may not matter.

In fact, I would guess it probably it probably doesn't matter. So let's take the RL out of it for a second because I actually think RL it's a red herring to say that RL was any different from pre training in this matter. So if we look at pre training scaling, it was very interesting.

Back 2017 when Alec Radford was doing GPT-one. If you look at the models before GPT-one, they were trained on these datasets that didn't represent a wide distribution of text. Right?

You had these very standard language modeling benchmarks. And GBT-one itself was trained on a bunch of I think it was fan fiction actually. But it was literary text, which is a very small fraction of the text that you get.

And what we found with that and in those days, was like a billion words or something. So small datasets and represented a pretty narrow distribution, right? Like a narrow distribution of kind of what you can see in the world.

And it didn't generalize well. If you did better on, you know, the the you know, I I forgot what it but it's some some kind of fan fiction corpus. It wouldn't generalize that well to kind of the other you know, we had all these measures of, you know, how well does the how well does the model do at predicting all of these other kinds of texts.

You really didn't see the generalization. It was only when you trained over all the tasks on the, you know, the Internet. When you when you kind of did a general Internet scrape, right, from something like, you know, Common Crawl or scraping links on Reddit, which is what we did for GPT-two.

It's only when you do that, that you kind of started to get generalization. And I think we're seeing the same thing on RL, that we're starting with first very simple RL tasks like training on math competitions. Then we're kind of moving to kind of broader training that involves things like code as a task.

And now we're moving to do kind of many other tasks. And then I think we're going to increasingly get generalization. So that takes out the RL versus the pre training side of it.

But I think there is a puzzle here either way, which is that on pre training, when we train the model on pre training, you know, we we use like trillions of tokens. Right? And and humans don't see trillions of words.

So there is an actual sample efficiency difference here. There there is actually something different that's that's happening here, which is that the model start from scratch and they have to get much more training. But we also see that once they're trained, if we give them a long context length the only thing blocking a long context length is like inference.

But if we give them like a context length of a million, they're very good at learning and adapting within that context length. And so I don't know the full answer to this, but I think there's something going on that pre training, it's it's not like the process of humans learning. It's somewhere between the process of humans learning and the process of human evolution.

It's like it's somewhere between like, we get many of our priors from evolution. Our brain isn't just a blank slate. Right?

Whole books have been written about. I think the language models, they're much more blank slates. They literally start as like random weights, whereas the human brain starts with all these regions, it's connected to all these inputs and outputs.

Maybe we should think of pre training and for that matter RL as well as being something that exists in the middle space between human evolution and of human on the spot learning. And as the in context learning that the models do as as something between long term human learning and short term human learning. So, you know, there there's this hierarchy of, like, there's evolution, there's long term learning, there's short term learning, and there's just human reaction.

And the LLM phases exist along this spectrum, but not necessarily exactly at the same points.

the goal is not to teach the model every possible skill within RL, just as we don't do that within pre training. Right? Within pre training, we're not trying to expose the model to every possible way that words could be put together.

Right? It's rather that the model trains on a lot of things and then it reaches generalization across pre training. Right?

That was transition from GPT-one to GPT-two that I saw up close, which is like you know, the the model reaches a point. You know? I I I I I like had these moments where I was like, oh, yeah.

You just give the model like you just give the model a list of numbers that's like, you know, you know, this is the cost of the house. This is the square feet of the house. And the model completes the pattern and does linear regression.

Not great, but it does it, but it's never seen that exact thing before. So to the extent that we are building these RL environments, the goal is very similar to what was done five or ten years ago with pre training, with we're trying to get a whole bunch of data, not because we want to cover a specific document or a specific skill, but because we wanna generalize.

two there's kind of two cases to be made here or like two two claims you could make, one of which is like stronger and the other of which is weaker. So I think starting with the weaker claim, when I first saw the scaling back in 2019, I wasn't sure. This was kind of a fiftyfifty thing.

I thought I saw something that was and my claim was this is much more likely than anyone thinks it is. This is wild. No one else would even consider this.

Maybe there's a 50% chance this happens. On the basic hypothesis of, as you put it, within ten years, we'll get to what I call country of geniuses in a data center. I'm at 90% on that.

And it's hard to go much higher than 90% because the world is so unpredictable. Maybe the irreducible uncertainty would be if we were at 95% where you get to things like, I don't know, may maybe multi you know, multiple companies have, you know, kind of internal turmoil and nothing happens. And then Taiwan gets invaded and, like, all the all the fabs get blown up by missiles and and, you know, and then now you're going to scenario.

Yeah. Yeah. Yeah.

You, you know, just you could construct a scenario where there's, like, a 5% chance that it or, you know, you you can construct a 5% world where, like, things things get delayed for 10 for for for for for for ten years. That's maybe 5%. There's another 5%, which is that I'm very confident on tasks that can be verified.

So I think I think with coding, I'm just except for that irreducible uncertainty, there's just I mean, I think we'll be there in one or two years. There's no way we will not be there in ten years in terms of being able to do it end to end coding. My one little bit, the one little bit of fundamental uncertainty even on long time scales is this thing about tasks that aren't verifiable, like planning a mission to Mars, like, you know, doing some fundamental scientific discovery like like CRISPR, like writing a novel, hard to verify those tasks.

I am almost certain that we have a reliable path to get there, but if there was a little bit uncertainty, it's there. So so so so so on the ten years, I'm like, you know, 90%, which is about as certain as you can be. Like, I think it's I think it's crazy to say that this won't happen by by by 2035.

Like, in some sane world, it would be outside the mainstream.

No. This is why I'm almost sure. We already see substantial generalization from things that verify to things that don't we're already seeing that.

But but it seems like you were emphasizing this as a spectrum which will split apart, which there means you see more progress. And I'm like, but that's it doesn't seem like how humans get away. World in which we don't make it or or or the world in which we don't get there is the world in which we do we do all the things that are that are verifiable.

And then they like you know, many of them generalize, but what we kinda don't get fully there. We don't we don't we don't fully, you know, we don't fully color in this side of the box.

which by the way, the the models are not bad. They're already pretty good at writing comments. And so with with again, I again, I'm making, like, much weaker claims here than I believe to, like, you know, to to to to to kinda set up you know, to to distinguish between two things.

Like, we're we're already almost there for software engineering. We are already almost there. By by what metric?

There's one metric, which is like how many lines of code are written by AI?

I actually agree with you on this. So, I've made this series of predictions on code and software engineering. And I think people have repeatedly misunderstood them.

So, let me lay out the spectrum, right? I think it was like eight or nine months ago or something, said, The AI model will be writing 90% of the lines of code in three to six months, which happened at least at some places. Happened at Anthropic, happened with many people downstream using our models.

But but that's actually a very weak criterion. Right? People thought I was saying like, we won't need 90% of the software engineers.

Those things are worlds apart. Right? Like, I would put the spectrum as 90% of code is written by the model.

A 100% of code is written by the model, and that's a big difference in productivity. 90% of the end to end suite tasks, right, including things like compiling, including things like setting up clusters and environments, testing features, writing memos. 90% of the SWE tasks are written by the models.

100% of today's SWE tasks are written by the models. And and even when when when that happened, doesn't mean software engineers are out of a job. Like, there's, like, new higher level things they can do where they can they can manage.

And then there's a further down the spectrum, you know, there's 90% less demand for SWEs, which I think will happen, but, like, this is this this is a spectrum. And, you know, I I wrote about it in in the adolescence of technology where I went through this kind of spectrum with farming. And so I I actually totally agree with you on that.

It's just these are very different benchmarks from each other, but we're proceeding through them super fast. It seems like in part of your vision, it's like going from 90 to a 100.

I simultaneously agree with you, agree that it's a reason why these things don't happen instantly. But at the same time, I think the the effect is going to be very fast. So like, I don't know, you could have these two poles, right?

One is like, AI is like, it's not going to make progress. It's slow. It's going to take kind of forever to diffuse within the economy.

Right? Economic diffusion has become one of these buzzwords that's like a a reason why we're not gonna make AI progress or why AI progress doesn't matter. And and, know, the other axis is like, we'll get recursive self improvement, you know, the whole thing, you know, can't you just draw an exponential line on the on the curve?

You know, it's it's we're gonna have, you know, Dyson spheres around the sun and like, you know, you know, so many nanoseconds after, you know, after after we get recursive. I mean, I'm completely caricaturing the view here, but like, there are these two extremes. But what we've seen from the beginning, at least if you look within Anthropic, there's this bizarre 10x per year growth in revenue that we've seen.

Right? So, you know, in 2023, it was like 0 to a 100,000,000. 2024, it was a 100,000,000 to a billion.

2025, it was a billion to like 9 or 10,000,000,000.

the first month of this year, like, that exponential is you would think it would slow down, but it would like we added another few billion to like we added another few billion to revenue in January. And so obviously that curve can't go on forever. Right?

The GDP is only so large. I would even guess that it bends somewhat this year, but that is like a fast curve. Right?

That's like a that's like a really fast curve. And I would bet it stays pretty fast even as the scale goes to the entire economy. So like, I I think we should be thinking about this middle world where things are like extremely fast, but not instant, where they take time because of economic diffusion, because of the need to close the loop, because it's like this fiddly, Oh man, I have to do change management within my enterprise.

I have to like I set this up, but I have to change the security permissions on this in order to make it actually work. Or I had this old piece of software that, like, you know, checks the model before it's compiled and and and, like, released, and I have to rewrite it. And, yes, the model can do that, but I have to tell the model to do that, and it has to it has to take time to do that.

And and and so I think everything we've seen so far is is compatible with the idea that there's one fast exponential that's the the capability of the model, and then there's another fast exponential that's downstream of that, which is the diffusion of the model into the economy. Not instant, not slow, much faster than any previous technology, but it has its limits. And and and and this is what we you know, when I when I look inside Anthropic, when I look at our customers, fast adoption, but not infinitely fast.

think diffusion is very real and doesn't have to you know, doesn't exclusively have to do with limitation limitation limitations on the AI models. Like, again, there are people who use diffusion to to you know, as kind of a buzzword to say this isn't a big deal. I'm not talking about that.

I'm not talking about AI will diffuse at the speed that previous I think AI will diffuse much faster than previous technologies have, not infinitely fast. So I'll just give an example of this. There's like Claude code.

Like Claude code is extremely easy to set up. If you're a developer, you can kind of just start using Claude code. There is no reason why a developer at a large enterprise should not be adopting Claude code as quickly as individual developer or developer at a startup.

And we do everything we can to promote it. We sell Claude code to enterprises and big enterprises like big financial companies, big pharmaceutical companies, all of them, they're adopting Claude code much faster than enterprises typically adopt new technology. But again, it like, it it it it it it takes time.

Like, any given feature or any given product like Claude Code or like Cowork will get adopted by the, you know, the individual developers who are on Twitter all the time, by the like, series a startups many months faster than than, you know, than they will get adopted by, like, you know, a, like, large enterprise that does food sales. There are a number of factors. Like, you have to go through legal.

You have to provision it for everyone. It has to pass security and compliance. The leaders of the company who are further away from the AI revolution are forward looking, but they have to say, Oh, it makes sense for us to spend 50,000,000.

This is what this Claude code thing is. This is why it helps our company. This is why it makes us more productive.

Then they have to explain to the people two levels below, and they have to say, okay. We have 3,000 developers. Here's how we're gonna roll it out to our developers.

And we have conversations like this every day. We are doing everything we can to make Anthropix revenue grow 20 or 30 x a year instead of 10 x a year. And again, many enterprises are just saying, this is so productive.

We're gonna take shortcuts in our usual procurement process. They're moving much faster than when we tried to sell them just the ordinary API, which many of them use, but quad code is a more compelling product. But it's not an infinitely compelling product.

And I don't think even AGI or Powerful AI or Country of Geniuses in the data center will be an infinitely compelling product. It will be a compelling product enough maybe to get three or five or 10 X a year growth, even when you're in the hundreds of billions of dollars, which is extremely hard to do and has never been done in history before, but not infinitely fast. I buy that it would be a slight slowdown.

we would know it. Right. Yeah.

We would know it if you had the country of geniuses in a data center. Like, everyone in this room would know it. Everyone in Washington would know it.

Like, you know, people in rural rural parts might not know it. But but but like, we would know it. We don't have that now.

That that's very clear.

will basically tell you where

when should we expect such an AI system? Yeah. So I guess what you're talking about is like, we're doing this interview for three hours and then you know, someone's gonna come in, someone's gonna edit it.

They're gonna be like, oh, you know, you know, I don't know, Dario like, you know, scratched his head and, you know, we could we could edit that out and, you know Magnify that. There was this like long was this long discussion that is less interesting to people, then there's other thing that's more interesting to people. So let's make this edit.

So I think the country of geniuses in a data center will be able to do The way it will be able to do that is it will have general control of a computer screen. Right? You'll be able to feed this in, and it'll be able to also use the computer screen to go on the web, look at all your previous interviews, look at what people are saying on Twitter in response to your interviews, talk to you, ask you questions, talk to your staff, look at the history of edits that you did, and from that, do the job.

Yeah. So I think that's dependent on several things. One, that's dependent, and I think this is one of the things that's actually blocking deployment, getting to the point on computer use, where the models are really masters at using the computer.

Right? And we've seen this climb in benchmarks, and benchmarks are always imperfect measures, but like OS world is went from 5%, I think when we first released computer use a year and a quarter ago, was like maybe 15%. I don't remember exactly.

But we've climbed from that to like 65 or 70%. And and, you know, there may be harder measures as well, but but I think computer use has to pass a point of reliability.

gets back to what we were talking about before with learning on the job where it's very interesting. I think with the coding agents, I don't think people would say that learning on the job is what is preventing the coding agents from doing everything end to end. They keep getting better.

We have engineers at Anthropic who don't write any code. And when I look at the productivity, to your previous question, we have folks who say, this this GPU kernel, this chip, I used to write it myself. I just have Claude do it.

And so there's this there's this enormous improvement in productivity. And I don't know. Like, when I see Claude code, like, familiarity with the code base or, like, you know, or or a feeling that the model hasn't worked at the company for for a year, that's not high up on the list of complaints I see.

advantage that other economic activity doesn't. But when you say that, what you're implying is that by reading the code base into the context, I have everything that the human needed to learn on the job. So that would be an example of whether it's written or not, whether it's available or not, a case where everything you needed to know, you got from the context window.

Right? And that and that what we think of as learning, oh, man, I started this job. It's gonna take me six months to understand the code base.

The model just did it in the context. Yeah.

amount of commercial pressure and make it even harder for ourselves because we have all the safety stuff we do that I think we do more than other companies. So the pressure to survive economically while also keeping our values is just incredible. We're trying to keep this 10x revenue curve going.

There is zero time for bullshit. There is zero time for feeling like we're productive when we're not. These tools make us a lot more productive.

Like, why why do you think we're concerned about competitors using the tools? Because we think we're ahead of the competitors and, like, we don't we don't wanna excel. We we wouldn't be going through all this trouble if this was secretly reducing our productivity.

We see the end productivity every few months in the form of model launches. There's no kidding yourself about this. The models make you more productive.

think it's all like my model of the situation is there's an advantage that's gradually growing. Like, I would say right now, the coding models give maybe, I don't know, a like 15, maybe 20% total factor speed up. That's my view.

And six months ago, it was maybe 5%. And so it didn't matter. 5% doesn't register.

It's now just getting to the point where it's one of several factors that kind of matters. That's going to keep speeding up. And so, I think six months ago, there were several companies that were at roughly the same point because this a notable factor.

But I think it's starting to speed up more and more. I would also say there are multiple companies that write models that are used for code and we're not perfectly good at preventing some of these other companies from using from kind of using our models internally. I think everything we're seeing is consistent with this kind of snowball model where there's no hard again, my theme in all of this is like, all of this is soft takeoff, like soft, smooth exponentials, although the exponentials are relatively steep.

And so and so we're seeing this snowball gather momentum where it's like 10%, 20%, 25%, you know, 4440%. And as you go, yeah, Amdahl's Law, you have to get all the like things that are preventing you from closing the loop out of the way. But this is one of the biggest priorities within Anthropic.

think two things here. There's the state of the technology right now, which is, again, we have these two stages. We have the pre training and RL stage where you throw a bunch of data and tasks into the models and then they generalize.

So it's like learning, but it's like learning from more data and not learning over kind of one human or one model's lifetime. So again, this is situated between evolution and human learning. But once you learn all those skills, you have them.

Just like with pre training, just how the models know more if I look at a pre trained model, it knows more about the history of samurai in Japan than I do. It knows more about baseball than I do. It knows more about low pass filters and electronics.

All of these things, its knowledge is way broader than mine. So I think even just that may get us to the point where the models are better at everything. Then we also have, again, just with scaling the existing setup, we have the in context learning, which I would describe as human on the job learning, but a little weaker and a little short term.

You look at in context learning, you give the model a bunch of examples, it does get it. There's real learning that happens in context, and a million tokens is a lot. That can be days of human learning.

Right? If you think about the model, you know, kind of reading a million words, you know, it takes me how long would it take me to read a million? I mean, you know, like days or weeks at least.

So you have these two things, and I think these two things within the existing paradigm may just be enough to get you the country of geniuses in the data center. I don't know for sure, but I think they're going to get you a large fraction of it. There may be gaps, but I certainly think just as things are, this I believe is enough to generate trillions of dollars of revenue.

That's one. That's all one. Two is this idea of continual learning, this idea of a single model learning on the job.

I think we're working on that too. I think there's a good chance that in the next year or two, we also solve that. Again, I think you get most of the way there without it.

I think trillions of dollars you know, the the I think the the trillions of dollars a year market, maybe all of the national security implications and the safety implications that I wrote about in adolescence of technology can happen without it. But I I I also think we, and I imagine others, are working on it. And I think there's a good chance that that, you know, that we get there within the next year or two.

There are a bunch of ideas.

your entire KV cache. It's difficult to store all the memory in the GPUs, to juggle the memory around. I don't even know the detail.

You know, at this point, this is at a level of detail that that that I'm no longer able to follow. Although, you know, I I knew it at the GPD three era of, know, these are the weights, these are the activations you have to store. But, you know, these days, the whole thing is flipped because we have MOE models and kind of all of that.

This degradation you're talking about, again, without getting too specific, like a question I would ask is like, there's two things. There's the context length you train at, and there's a context length that you serve at. If you train at a small context length and then try to serve at a long context length, like maybe you get these degradations.

It's better than nothing. You might still offer it, but you get these degradations. And maybe it's harder to train at a long context length.

So there's a lot.

mean, my guess for that is, there's a lot of problems that are basically like, we can do this when we have the country of geniuses in a data center. And so my picture for that is, you know, again, if you if you if you if you you know if you made me guess, it's like one to two years, maybe one to three years. It's really hard to tell.

I have a I have a strong view, 99, 95% that like all this will happen in ten years. Like, that's I think that's just a super safe bet. Yeah.

And then I have a hunch. This is more like a fifty fifty thing that it's gonna be more like one to two, maybe more like one to three. So one to three years.

It seems pretty economically valuable, let me tell you. It's just there are a lot of use cases like Exactly. There are a of similar Exactly.

So you're predicting that within one to three years.

with your stated views about AI progress. Yeah. So it actually all fits together.

And we go back to this fast, but not infinitely fast diffusion. So let's say that we're making progress at this rate. Technology is making progress this fast.

Again, I have very high conviction that it's going We're gonna get there within a few years. I have a hunch that we're gonna get there within a year or two. So a little uncertainty on the technical side, but pretty strong confidence that it won't be off by much.

What I'm less certain about is, again, the economic diffusion side. I really do believe that we could have models that are a country of in the data center in one to two years. One question is, how many years after that do the trillions in revenue start rolling in?

I don't think it's guaranteed that it's going to be immediate. I think it could be one year, it could be two years, I could even stretch it to five years, although I'm skeptical of that. And so we have this uncertainty, which is even if the technology goes as fast as I suspect that it will, don't know exactly how fast it's going to drive revenue.

We know it's coming, but with the way you buy these data centers, if you're off by a couple years, that can be ruinous. It is just like how I wrote in Machines of Loving Grace, I said, look, I think we might get this powerful AI, this country of genius in the data center. That description you gave comes from the Machines of Loving Grace.

I said, we'll get that twenty twenty six, maybe twenty twenty seven again. That is my hunch. Wouldn't be surprised if I'm off by a year or two, but like, that is my hunch.

Let's say that happens. That's the starting gun. How long does it take to cure all the diseases?

Right? That's one of the ways that drives a huge amount of economic value. Right?

You cure every disease. There's a question of how much of that goes to the pharmaceutical company, to the AI company, but there's an enormous consumer surplus because everyone assuming we can get access for everyone, which I care about greatly, cure all of these diseases. How long does it take?

You have to do the biological discovery. You manufacture have the new drug. You have to go through the regulatory process.

We saw this with vaccines and COVID. There's just this, we got the vaccine out to everyone, but it took a year and a half. So my question is, how long does it take to get the cure for everything, which AI is the genius that can, in theory, invent out to everyone.

How long from when that AI first exists in the lab to when diseases have actually been cured for everyone? Right? We've had a polio vaccine for fifty years.

We're still trying to eradicate it in the most remote corners of Africa. And, you know, the Gates Foundation is trying as hard as they can. Others are trying as hard as they can, but, you know, that's difficult.

Again, I, you know, I don't expect most of the economic diffusion to be as difficult as that. Right? That's like the most difficult case.

But there's a real dilemma here. And where I've settled on it is it will be faster than anything we've seen in the world, but it still has its limits. And so then, when we go to buying data centers, again, the curve I'm looking at is, okay, we've had a 10x a year increase every year.

So beginning of this year, we're looking at 10,000,000,000 in rate of annualized revenue at the beginning of the year. We have to decide how much compute to buy. And it takes a year or two to actually build out the data centers, to reserve the data centers.

So basically, I'm saying in 2027, how much compute do I get? Well, I could assume that the revenue will continue growing 10x a year, so it'll be 100,000,000,000 at the end of 2026 and 1,000,000,000,000 at the end of 2027. And so I could buy a trillion dollars.

Actually, it would be like $5,000,000,000,000 of compute because it would be a trillion dollar a year for five years. Right? I could buy a trillion dollars of compute that starts at the 2027.

If my revenue is not a trillion dollars, if it's even 800,000,000,000, there's no force on earth. There's no hedge on earth that could stop me from going bankrupt if I buy that much compute. So even though a part of my brain wonders if it's going to keep growing 10x, I can't buy a trillion dollars a year of compute in 2027.

If I'm just off by a year in that rate of growth or if the growth rate is 5x a year instead of 10x a year, then you go bankrupt. And so you end up in a world where you're supporting hundreds of billions, not trillions, and you accept some risk that there's so much demand that you can't support the revenue, and you accept still some risk that you got it wrong and it's still slow. And so when I talked about behaving responsibly, what I meant actually was not the absolute amount.

That that actually was not you know, I think it is true we're spending somewhat less than some of the other players. It's actually the other things like, have we been thoughtful about it? Or are we YOLO ing and saying, oh, we're gonna do a $100,000,000,000 here, a $100,000,000,000 there.

I kinda get the impression that, you know, some of the other companies have not written down the spreadsheet, that they don't really understand the risks they're taking. They're just kind of doing stuff because it sounds cool. We've thought carefully about it.

Right? We're an enterprise business. Therefore, we can rely more on revenue.

It's less fickle than consumer. We have better margins, which is the buffer between buying too much and buying too little. And so I think we bought an amount that allows us to capture pretty strong upside worlds.

It won't capture the full 10x a year, and things would have to go pretty badly for us to be in financial trouble. So I think we've thought carefully and we've made that balance. And that's what I mean when I say that we're being responsible.

like, substantially more gains from buying a trillion dollars a year of compute versus $300,000,000,000 a year of compute. If your competitor is buying a trillion, yes, there is. Well, no.

There's some gain, but then but again, there's this chance that they go bankrupt before, you know, be again, if you're off by only a year, you destroy yourselves. That's the that's the balance. We're buying a lot.

We're buying a hell of a lot. Like, we're not we're we're you know, we're buying an amount that's comparable to that that, you know, the the the the biggest players in the game are buying. But but if you're asking me, why why haven't we signed, you know, $1,010,000,000,000,000 of compute starting in starting in mid twenty twenty seven?

First of all, it can't be produced. There isn't that much in the world. But but second, what if the country of geniuses comes, but it comes in mid twenty twenty eight instead of mid twenty twenty seven?

You go bankrupt.

What what accordance. What what makes you think that?

on the order of 50,000,000,000,000 a year. If if you look at so so I won't I won't talk about Anthropic in particular, but if you talk about the industry, like the amount of compute the industry the amount of compute the industry is building this year is probably in the, I don't know, very low tens of call it ten, fifteen gigawatts Next year, it goes up by roughly three X a year. So like next year's 30 or 40 gigawatts and 2028 might be 100, 2029 might be like 300 gigawatts.

And each gigawatt costs like maybe 10 I mean, I'm doing the math in my head, but each gigawatt costs maybe $10,000,000,000 border 10 to $15,000,000,000 a year. Put that all together and you're getting about what you described. You're getting multiple trillions a year by 2028 or 2029.

So you're getting exactly that. You're getting exactly what you predict.

So then it's like a 100,000,000,000 a year. But then you're saying the TAM by 2028, I 20 I don't wanna give exact numbers for Anthropic, but but these numbers are too small. These numbers are too small.

is this kind of weird thing in this field. I think in this field profitability is actually a measure of spending down versus investing in the business. Let's just take a model of this.

I actually think profitability happens when you underestimated the amount of demand you were going to get and loss happens when you overestimated the amount of demand you were going to get because you're buying the data centers ahead of time. So think about it this way. Ideally, would like And again, these are stylized facts.

These numbers are not exact for I'm just trying to make a toy model here. Let's say half of your compute is for training and half of your compute is for inference. And the inference has some gross margin that's like more than 50%.

And so what that means is that if you were in steady state, you build a data center, if you knew exactly the demand you were getting, you would get a certain amount of revenue, say, I don't know, let's say you pay $100,000,000,000 a year for compute, and on $50,000,000,000 a year, you support $150,000,000,000 of revenue, and the other 50,000,000,000 are used for training. So basically, you're profitable, you make $50,000,000,000 of profit. Those are the economics of the industry today.

Or sorry, not today, but that's where projecting forward in a year or two. The only thing that makes that not the case is if you get less demand than 50,000,000,000, then you have more than 50% of your data center for research and you're not profitable. So you train stronger models, but you're not profitable.

If you get more demand than you thought, then your research gets squeezed, but you're able to support more inference and you're more profitable.

in compute. Right? Because if you actually like compute I'm it's hard to predict.

So these things about 2028 and when it will happen, that's our attempt to do the best we can with investors. All of this stuff is really uncertain because of the cone of uncertainty. Like, we could be profitable in 2026 if the if the revenue grows fast enough.

And then and then, you know, if we overestimate or underestimate the next year, that could swing wildly. Like, I I I what I'm trying to get is you have a model in your head of, like, the the business invest, invest, invest, invest, get scale, and kind of then becomes profitable. There's a single point at which things turn around.

I don't think the economics of this industry work that way.

revenue is high, but losses are also high. If we predict, if every year we predict exactly what the demand is going to be, we'll be profitable every year. Because spending 50% of your compute on research, roughly, plus a gross margin that's higher than 50% and correct demand prediction leads to profit.

know, just like a given constant. Whereas you in fact, if AI progresses fast and you can increase the progress by scaling up more, you should just have more than 50% and not make progress. Here's what I'll say.

You might wanna scale up it more. You might wanna scale it up more. But but but, you know, remember the log returns to scale.

Right? If if 70% would get you a very little bit of a smaller model through a factor of of 1.4 x, right, like, that extra $20,000,000,000 is, you know, that each dollar there is worth much less to you because of the log linear setup.

And so you might find that it's better to invest that $20,000,000,000 in in serving inference or in hiring engineers who are who are who are kinda better who are who are kinda better who are kinda better at what they're doing. So the the reason I said 50%, that's not that's not exactly our target. It's not exactly gonna be 50%.

It'll probably vary vary over time. What what I'm saying is the the the the the like log linear return, what it leads to is you spend of order one fraction of the business. Right?

Like, not 5%, not 95%.

after you're spending 50,000,000,000 a year, right?

the other companies in the market can like do more things with this money that I can't then put aside anthropic. I'm just trying to like because I I, you know, I don't wanna give information about anthropic is why I'm giving these stylized numbers, but let's just derive the equilibrium of the industry. Right?

I think the so why doesn't everyone spend 100% of their compute on training and not serve any customers? Right? It's because if they didn't get any revenue, they couldn't raise money, they couldn't do compute deals, they couldn't buy more compute the next year.

So there's gonna be an equilibrium where every company spends less than 100 percent on training and certainly less than 100% on inference. It should be clear why you don't just serve the current models and never train another model because then you don't have any demand because you'll fall behind. So there's some equilibrium.

It's not gonna be 10%. It's not gonna be 90%. Let's just say as a stylized fact, it's 50%.

That's what I'm getting at. I think we're gonna be in a position where that equilibrium of how much you spend on training is less than the gross margins that you're able to get on compute. And so the underlying economics are profitable.

The problem is you have this hellish demand prediction problem when you're buying the next year of compute, and you might guess under and be very profitable, but have no compute for research, or you might guess over and you are not profitable and you have all the compute for research in the world. Does that make sense?

is not my view. So I'll make another prediction. It is hard for me to see that there won't be trillions of dollars in revenue before 2030.

I can construct a plausible world. It takes maybe three years, so that would be the end of what I think it's plausible. Like in 2028, we get the real country of geniuses in the data center.

The revenue has been going into the maybe is is in the low hundreds of billions by by by by 2028. And and and then the country of geniuses accelerates it to trillions, you know, and and we're basically we're basically on the slow end of diffusion. It takes two years to get to the trillions.

That that that would that that that would be the world where it takes until that would be the world where it takes until 2030. I I suspect even composing the technical exponential and diffusion exponential will get there before 2030.

let's just abstract the whole industry industry here. Let's just imagine we're in an economics textbook. We have a small number of firms, each can invest a limited amount or each can invest some fraction in R and D.

They have some marginal cost to serve. The gross profit margins on that marginal cost are very high because inference is efficient, there's some competition, but the models are also differentiated. There's some companies will compete to push their research budgets up, but because there's a small number of players, we have the what is it called?

Cornot equilibrium, I think is what the small number of firm equilibrium is. It the point is it it doesn't equilibrate to perfect competition with with with with with with zero margins. If there's, like, three firms if there's three firms in the economy, all are kind of independently behaving rationally, it doesn't equilibrate to zero.

Yeah. So again, the gross margins right now are very positive. What's happening is a combination of two things.

One is we're still in the exponential scale up phase of compute. Yeah. So basically what that means is we're training like a model gets trained.

Yep. It costs let's say a model got trained that costs a billion dollars last year. And then this year, it produced $4,000,000,000 of revenue and cost $1,000,000,000 to inference from.

So, know, again, I'm using stylized number here, but, you know, that would be 75%, you know, gross gross margins and, you know, this this 25% tax. So that model as a whole makes $2,000,000,000. But at the same time, we're spending $10,000,000,000 to train the next model because there's an exponential scale up.

And so the company loses money. Each model makes money, but the company loses money. The equilibrium I'm talking about is an equilibrium where we have the country of geniuses.

We have the country of geniuses in a data center, but that that model training scale up has equilibrated more. Maybe maybe it's still it's still going up. We're still trying to predict the demand, but it's more it's more leveled out.

have algorithmic improvements, but we're spending roughly the same amount to train the next model as we spend to train the current model. So this equilibrium relies- I mean, at some point, at some at some point, run out of money in the economy.

much faster with AI than I think it ever has before. But it's not like right now, the computer is growing three x a year. Yeah.

I don't believe the economy is gonna grow 300% a year. Like, I said this in Machines of Love and Grace. Like, I think we may get 10 or 20% per year growth in the economy, but we're not gonna get 300% growth in the economy.

more progress. I don't think that's true. I mean, I I feel feel like we're we're, like, we're talk we're we're, know, we're they feel like this is economics.

Like, you know, this is this is like an economics class. You know that Tyler Cowen code? We we never stop talking about economics.

We never we never stop talking about economics. So no. But but there there are there are worlds in which, you know, there the so I I don't think this field's gonna be a I don't think this field's gonna be a monopoly.

All my lawyers never want me to say the word monopoly. But I don't think this field's gonna be a monopoly. But but you do get you get industries in which there are small number of players, not one, but a small number of players.

And ordinarily, like, the the way you get monopolies like Facebook or or Meta, I always call them Facebook, but is these kind of network effects. The way you get industries in which there are small number of players are very high costs of entry. Right?

So, cloud is like this. I think cloud is a good example of this. You have three, maybe four players within cloud.

I think I think that's the same for AI. Three, maybe four. And the reason is that it's it's so expensive.

It requires so much expertise and so much capital to run a cloud company. Right? And so you have to put up all this capital, and then in addition to putting up all this capital, you have to get all of this other stuff that requires a lot of skill to, you know, to make it happen.

And so it's like if you go to someone and you're like, I wanna disrupt this industry. Here's a $100,000,000,000. You're like, okay.

I'm putting a $100,000,000,000 and also betting that you can do all these other things that these people have been doing. Only decrease the profit in the industry. And and then and then the effect of your entering is the is the profit margins go down.

So, you know, we have equilibria like this all the time in the economy where we have a few we have a few players. Profits are not astronomical, margins are not astronomical, but they're they're not zero. Right?

And and, you know, I think I think that's what we see on cloud. Cloud is very undifferentiated. Models are more differentiated than cloud.

Right? Like, Everyone knows, is good at different things than GPT is good at than Gemini is good at. And it's not just Claude's good at coding, GPT is good at math and reasoning.

It's more subtle than that. Like, models are good at different types of coding. Models have different styles.

Like, I think I think these things are actually, you know, quite different from each other, and so I would expect more differentiation than you see in in cloud. Now, actually is a counter there is one counterargument. And that counterargument is that if all of that, the process of producing models becomes, if AI models can do that themselves, then that could spread throughout the economy.

But that is not an argument for commoditizing AI models in general. That's kind of an argument for commoditizing the whole economy at once. I don't know what what quite happens in that world where basically anyone can do anything, anyone can build anything, and there's like no moat around anything at all.

I mean, I don't know. Maybe we want that world. Like like, maybe that's the maybe that's the end state here.

Like, maybe maybe, you know, when what maybe when when when kind of AI models can do you know, when when when when AI models can do everything, if we've solved all the safety and security problems, like, you know, that's one of the one of the one of the mechanisms for for, you know, you know, just just kind of the economy flattening itself again. But but that's kinda like post, like, far post country geniuses in a data center.

So I think coding is going fast, but I think AI research is a super set of coding and there are aspects of it that are not going fast. But I do think, again, once we get coding, once we get AI models going fast, then that will speed up the ability of AI models to do everything else. So I think while coding is going fast now, I think once the AI models are building the next AI models and building everything else, the kind of whole the whole economy will start to kind of go at the same pace.

I am I am worried geographically, though. I'm a little worried that, like, just proximity to AI, having heard about AI, that may be one differentiator. And so when I said the 10% or 20% growth rate, a worry I have is that the growth rate could be 50% in Silicon Valley and parts of the world that are kind of socially connected to Silicon Valley and not that much faster than its current pace elsewhere.

And I think that'd be a pretty messed up world. So one of the things I think about a lot is how to prevent that. Yeah.

but current AI models can't, at least not in a way that's super productive. And so if we have this ability to learn like a human, should it solve robotics immediately I as don't think it's dependent on learning like a human. It could happen in different ways.

Again, we could have trained the model on many different video games, which are like robotic controls or many different simulated robotics environments, or just train them to control computer screens and they learn to generalize. So it will happen. It's not necessarily dependent on human like learning.

Human like learning is one way it could happen if the model's like, oh, I pick up a robot. I don't know how to use it. I learn.

That could happen because we discovered discovering continual learning. That could also happen because we train the model on a bunch of environments and then generalized, or it could happen because the model learns that in the context length. It doesn't actually matter which way.

If we go back to the discussion we had an hour ago, that type of thing can happen in several different ways. But I do think when for whatever reason the models have those skills, then robotics will be revolutionized, both the design of robots because the models will be much better than humans at that, and also the ability to kind of control robots. So we'll get better at building the physical hardware, building the physical robots, and we'll also get better at controlling it.

Now, does that mean the robotics industry will also be generating trillions of dollars of revenue? My answer there is yes. But there'll be the same extremely fast, but not infinitely fast diffusion.

So will robotics be be revolutionized? Yeah. Maybe tack on another year or two.

That's the way I think about these things. Makes sense.

found the pieces of human intelligence. Well, to be clear, I mean, I think continual learning, as I've said before, might not be a barrier at all. Yeah.

Right? Like, I think we maybe just get there by pretraining generalization and RL generalization. Like, I think there just might not be there basically might not be such a thing at all.

In fact, I would point to the history in ML of people coming up with things that are barriers that end up kind of dissolving within the big blob of compute, right? That people talked about how do you have how do your models keep track of nouns and verbs and how do they they can understand syntactically, but they can't understand semantically. It's only statistical correlations.

You can understand a paragraph. You can understand a word. There's reasoning.

You can't do reasoning, but then suddenly it turns out you can do code and math very well at all. So I think there's actually a stronger history of some of these things seeming like a big deal and then and then kind of and then kind of dissolving. Some of them are real.

I mean, the need for data is real. May maybe continual continual learn continual learning is a real thing. But again, I would ground us in something like code.

Like, I think we may get to the point in like a year or two where the models can just do SWE end to end. Like, that's a whole task. That's a whole sphere of human activity that that we're just saying models can do it now.

90% of code, 100% of code, 90% of end to end SWE, 100% of end to end SWE, new tasks are created for SWE, eventually those get done as well. The long spectrum but we're traversing the spectrum very quickly. Yeah.

know, the the truth of the the truth of the matter is that we're all trying to figure this out together. Yeah. Right?

There there are some ways in which I'm able to see things that others aren't. These days, that probably has more to do with like, I can see a bunch of stuff within Anthropic and have to make a bunch of decisions than I have any great research insight that that that others don't. Right?

I've you know, I'm running a 2,500 person company like it's it's actually pretty hard for me to have have concrete research insight, you know, much harder than, you know, than than it would have been, you know, ten years ago or or, you know, or even two or three years ago.

are going to be experimented with. I actually do think that the API model is more durable than many people think. One way I think about it is if the technology is advancing quickly, if it's advancing exponentially, what that means is there's always a surface area of new use cases that have been developed in the last three months.

And any kind of product surface you put in place is always at risk of sort of becoming irrelevant. Right? Any given product surface probably makes sense for our range of capabilities of the model.

Right? The chatbot is already running into limitations of, you know, making it smarter doesn't really help the average consumer that much. But I don't think that's a limitation of AI models.

I don't think that's evidence that, you know, the models are are the models are good enough and they're them getting better doesn't matter to the economy. It doesn't matter to that particular product. And so I think the value of the API is the API always offers an opportunity very close to the bare metal to build on what the latest thing is.

And so there's always going to be this front of new startups and new ideas that weren't possible a few months ago and are possible because the model is advancing. And and so I I actually I I I kind of actually predict that we are it's gonna exist alongside other models, but we're always gonna have the API business model because there's there's always gonna be a need for a thousand different people to try experimenting with the model in different way, and a 100 of them become startups, and 10 of them become big successful startups, and two or three really end up being the way that people use the model of a given generation. So I I basically think it's always gonna exist.

At the same time, I'm sure there's gonna be other models as well. Like, not every token that's output by the model is worth the same amount. Think about, you know, how how how what is the value of the tokens that are like, you know, that the model outputs when someone, you know, call you know, someone, you know, calls them up and says, my Mac isn't working or something, you know, the models like restart it.

Right? Yeah. And like, you know, someone hasn't heard that before, but like, you know, the model said that like 10,000,000 times.

Right? You know, that's that maybe that's worth like a dollar or a few cents or something. Whereas if the model, you know, the model goes to, you know, one of the one of the pharmaceutical companies and it says, oh, you know, this molecule you're developing, you should take the aromatic ring from that end of the molecule and put it on that end of the molecule.

And and, you know, if you do that, wonderful things will happen. Like those tokens could be worth tens of millions of dollars. Right?

So I think we're definitely gonna see business models that recognize that at some point we're gonna see pay for results or, you know, some form, or we may see forms of compensation that are like labor, you know, that kind of work by the hour. I I I, you know, I don't know. I think I think that I think because it's a new industry, a lot of things are gonna be tried.

And I, you know, I don't know what will turn out to be the right thing.

we had our coding models, which were good at coding. Around the beginning of 2025, I said, I think the time has come where you can have nontrivial acceleration of your own research if you're an AI company by using these models. And of course, you know, we you need an interface.

You need a harness to use them. And so I encourage people internally, you know, I didn't say this is one thing that, you know, that you have to use. I just said people should experiment with this.

And then this thing, I think it might have been originally called Claude CLI, then the name eventually got changed to Claude Code internally, was the thing that kind of everyone was using, and it was seeing fast internal adoption. And I looked at it and I said, probably we should launch this externally. Right?

It's seen such fast adoption within Anthropic, coding is a lot of what we do. So we audience have of many hundreds of people that's in some ways at least representative of the external audience. So it looks like we already have product market fit.

Let's launch this thing. Then we launched it. I think just the fact that we ourselves are kind of developing the model and we ourselves know what we most need to use the model.

I think it's kind of creating this feedback loop. I see.

one version of it, but then there's just the ordinary product iteration of like, you know, we have a bunch of coders within Anthropic. Like, you know, they like use quad code every day, and so we get fast feedback. That was more important in the early days.

Now, of course, there are millions of people using it. And so we get a bunch of external feedback as well, but it's, just great to be able to get kind fast internal feedback. I think this is the reason why we launched a coding model and didn't launch a pharmaceutical company.

My background's in biology, but we don't have any of the resources that are needed to launch a pharmaceutical company.

skeptical of the balance of power. But I think I was particularly skeptical of, or the thing I was specifically skeptical of is you have, like, three or four of these companies, like, kind of all building models that are kind of dry you know, sort of sort of, like, derived from the like, derived from the same thing and, you know, that that these would check each other or or even that kinda, you know, any number of them would would would check each other. Like, we might live in a offense dominant world where, you know, like, one person or one AI model is, like, smart enough to do something that, like, causes damage for everything else.

I think in the I mean, in the short run, we have a limited number of players now. So we can start by within the limited number of players. We, you know, we kind of, you know, we we need to put in place the, you know, the safeguards.

We need to make sure everyone does the right alignment work. We need to make sure everyone has bio classifiers. Like, you know, those are those are kind of the immediate things we need to do.

I agree that that doesn't solve the problem in the long run, particularly if the ability of AI models to make other AI models proliferates, then the whole thing can become harder to solve. You know, I think I think in the long run, we need some architecture of governance. Right?

Some some architecture of governance that preserves human freedom, but but kind of also allows us to, like, you know, govern the the very large number of kind of, human systems, AI systems, hybrid human AI, like companies or or like or like or like economic units. So, you know, we're we're gonna need to think about, like, you know, how do we how do we protect the world against, you know, bioterrorism? How do we protect the world against, like, you know, against, like, against, like, mirror life?

Like, you know, probably probably we're gonna need to, you know, need some kind of, like, AI monitoring system that, like, mono you know, kinda monitors for for all of these things, but then we need to build this in a way that, like, preserves civil liberties and our constitutional rights. So I think just as is anything else, like a new security landscape with a new set of tools and a new set of vulnerabilities. And I I think my worry is if we had a hundred years for this to happen all very slowly, we'd get used to it.

You know? Like, we've gotten used to, like, you know, the presence of, you know, the presence of explosives in society or, like, the, you know, the presence of various, you know, like, new weapons or the, you know, the the presence of video cameras. We would get used to it over over over over a 100 and we develop governance mechanisms.

We'd make our mistakes. My my worry is just that this is happening all so fast. And so I think maybe we need to do our thinking faster about how to make these governance mechanisms work.

Yeah.

make this happen. It just the governments of the world may have to work together to make it happen. We may have to you may have to talk to AIs about building societal structures in such a way that these defenses are possible.

I don't know. I mean, is you know, I I don't wanna say so far ahead in time, but, like, so far ahead in technological ability that may happen over a short period of time that it's hard for us to anticipate it in advance.

threats that you're concerned about. So I'm curious about to understand in the context of things like this, your anthropics position against the federal moratorium on state AI laws. Yes.

So I don't know. There's there's many different things going on at once. Right?

I think I think that that I think that particular law is is dumb. Like, you know, I think it was it was clearly made by legislators who just probably had little idea what AI models could do and not do. They're like, AI models serving as that that just sounds scary.

Like, I don't want I don't want that to happen. So, you know, we're we're we're not we're not in favor of that. Right?

But but but that, you know, that that wasn't the thing that was being voted on. The thing that was being voted on is we're going to ban all state regulation of AI for ten years with no apparent plan to to do any federal regulation of AI, which would take congress to pass, which is a very high bar. So, you know, the idea that we'd ban states from doing anything for ten years, and people said they had a plan for federal government, but, you know, there was no actual there was no proposal on the table.

There was no actual attempt. Given the serious dangers that I lay out in adolescence of technology around things like the, you know, kind of biological weapons and bioterrorism, autonomy risk, and the timelines we've been talking about, like, ten years is an eternity. Like, that's that's a that's a I I think that's a crazy thing to do.

So if if that's the choice, if that's what you force us to choose, then then we're gonna we're gonna choose not to have that moratorium. And, you know, I I think the the benefits of that position exceed the costs, but it's it's not a perfect position if that's the choice. Now I think the thing that we should do, the thing that I would support, is the federal government should step in, not saying states you can't regulate, but here's what we're gonna do, and and states you can't differ from this.

Right? Like, I think preemption is fine in the sense of saying that federal government says, here's our standard. This applies to everyone.

States can't do something different. That would be something I would support if it would be done in the right way. What but but this idea of states, can't do anything and we're not doing anything either, that that struck that struck us as, you know, very much not making sense.

And I think we'll not age well. It's already starting to not age well with with all the backlash that that you've seen. Now in terms of in terms of what we would want, I mean, the things we've talked about are starting with transparency standards in order to monitor some of these autonomy risks and bioterrorism risks.

As the risks become more serious, as we get more evidence for them, then I think we could be more aggressive in some targeted ways and say, hey, AI bioterrorism is really a threat. Let's pass a law that kind of forces people to have classifiers. And I could even imagine, it depends.

It depends how serious the threat it ends up being. We don't know for sure. Then we need to pursue this in an intellectually honest way where we say ahead of time, the risk has not emerged yet.

But I could certainly imagine with the pace that things are going that, you know, I could imagine a world where later this year we say, hey. This AI bioterrorism stuff is really serious. We should do something about it.

We should put it in a federal we should, you know, put it in a federal standard. And if the federal government won't act, we should put it in a state standard.

as compared to the dangers of AI. And so that's maybe where the cost benefit makes less sense to me. So there's a few things here.

People talk about there being thousands of these state laws. First of all, the vast, vast majority of them do not pass. And, you know, the the the the the you know, the world works a certain way in theory, but, like, just because a law has been passed doesn't mean it's really enforced.

Right? The people the people, you know, implementing it may be like, oh my god, this is stupid. It would mean shutting off, like, you know, everything that's ever been built in everything that's ever been built in Tennessee.

So, you know, very often laws are interpreted in, you know, a way that makes them that that that makes them not as dangerous or not as harmful. On on the same side, of course, you have to worry if you're passing a law to stop a bad thing, you had this you had this problem as well. Yeah.

Look. My my look. I mean, my basic view is, you know, if if we could decide what laws were passed and how things were done, which we're only one small input into that, I would deregulate a lot of the stuff around the health benefits of AI.

I think I don't worry as much about the chatbot laws. I actually worry more about the drug approval process, where I think AI models are going to greatly accelerate the rate at which we discover drugs and just the pipeline will get jammed up. The pipeline will not be prepared to process all of the stuff that's going through it.

So I think reform of the regulatory process to buy us more towards we have a lot of things coming where the safety and the efficacy is actually gonna be really crisp and clear. Like, mean, beautiful thing. Really, really crisp and clear and, like, really, really effective.

But and maybe we don't need all this superstructure around it that was designed around an era of drugs that barely work and often have serious side effects. But at the same time, I think we should be ramping up quite significantly this kind of safety and security legislation. And like I've said, you know, starting with transparency is is my view of trying not to hamper the industry.

Right? Trying to find the right balance. I'm worried about it.

Some people criticize my essay for saying that's too slow. The dangers of AI will come too soon if we do that. Well, basically, I kind of think like the last six months and maybe the next few months are gonna be about transparency.

And then if these if these risks emerge when we're more certain of them, which I think we might be as soon as as later this year, then I think we need to act very fast in the areas that we've actually seen the risk. Like, I think the only way to do this is to be nimble. Now, the legislative process is normally not nimble, but we need to emphasize to everyone involved the urgency of this.

That's why I'm sending this message of urgency. Right? That's why I wrote adolescents of technology.

I wanted policymakers to read it. I wanted economists to read it. I want national security professionals to read it.

You know, I want decision makers to read it so that they have some hope of acting faster than they would have otherwise.

different kinds of moral panics or political economy problems. Yeah. Don't actually So I don't actually agree that much in the developed world.

I feel like in the developed world, markets function pretty well. And when there's a lot of money to be made on something and it's clearly the best available alternative, it's actually hard for the regulatory system to stop it. You know, we're we're seeing that in AI itself.

Right? I you know, like, a thing I've been trying to fight for is export controls on chips to China. Right?

And, like, that's in the national security interests of The US. Like, you know, that's like square within the, you know, the the policy beliefs of, you know, every almost everyone in congress of both parties. But and, you know, I think the case is very clear.

The counterarguments against it are I'll politely call them fishy. And yet, it doesn't happen, and we sell the chips because there's so much money. There's so much money riding on it.

And that money wants to be made, and in that case, in my opinion, that's a bad thing. But it also applies when it's a good thing. And so I don't think that if we're talking about drugs and benefits of the technology, I am not as worried about those benefits being hampered in the developed world.

I am a little worried about them going too slow. As I said, I do think we should work to speed the approval process in the FDA. I do think we should fight against these chatbot bills that you're describing, described individually.

I'm against them. I think they're stupid. But I actually think the bigger worry is a developing world, where we don't have functioning markets, where we often can't build on the technology that we've had.

I worry more that those folks will get left behind. And I worry that even if the cures are developed, maybe there's someone in rural Mississippi who doesn't get it as well. That's a kind of smaller version of the thing, the concern we have in the developing world.

And so the things we've been doing are, we work with philanthropists, right?

on a data center? Why can't you know, why won't it happen or why should they say? Why shouldn't it happen?

Why shouldn't it happen? You know, I think I think if this does happen, you know, then then we kind of have a well, we could have a few situate if we have, like, an offense dominant situation, we could have a situation like nuclear weapons, but, like, more dangerous. Right?

Where it's, like, you know, kind of kind of either side could could easily destroy everything. We could also have a world where it's kind of it's unstable. Like, nuclear equilibrium is stable.

Right? Because it's, you know, it's like deterrence. But let's say there were uncertainty about, like, if the two AIs fought, which AI would win?

That could create instability. Right? You often have conflict when the two sides have a different assessment of their likelihood of winning.

Right? If one side is like, oh, yeah. There's a 90% chance I'll win, the other side's like, there's a 90% chance I'll win, then then then a fight is much more likely.

They can't both be right, but they can both think that. But this is like a fully general argument against the diffusion of AI technology, which it may which is that's the implication of this world. Let me let me just go on because I think we will get diffusion eventually.

The other concern I have is that people the governments will oppress their own people with AI. And so I'm worried about some world where you have a country that's already kind of there's a government that kind of already is kind of building a high-tech authoritarian state. And to be clear, this is about the government.

This is not about the We need to find a way for people everywhere to benefit. My worry here is about governments. So, yeah, my worry is if the world gets carved up into two pieces, one of those two pieces could be authoritarian or totalitarian in a way that's very difficult to displace.

Now, will governments eventually get powerful AI? And there's risk of authoritarianism? Yes.

Will governments eventually get powerful AI and there's risk of kind of bad equilibrium? Yes, I think both things. But the initial conditions matter.

Right? At some point, we're going to need to set up the rules of the road. I'm not saying that one country, either The United States or a coalition of democracies, which I think would be a better setup, although it requires more international cooperation than we currently seem to wanna make.

But, you know, I don't I don't think a coalition of democracies or or certainly one country should just say these are the rules of the road. There's gonna be some negotiation. Right?

The world is gonna have to grapple with this. And what I would like is that the the the, you know, the democratic nations of the world, those with you know, who are whose governments have represent closer to prohuman values are holding a stronger hand then, have more leverage when the rules of the road are set. So I'm very concerned about that initial condition.

wanna better understand what you're imagining here. Yeah. So just to be precise about it, I think the exponential of the underlying technology will continue as it has before.

The models get smarter and smarter even when they get to country of geniuses in a data center. Think you can continue to make the model smarter. There's a question of getting diminishing returns on their value in the world.

Right? How much does it matter after you've already solved human biology or at some point you can do harder math. You can do more abstruse math problems, but nothing after that matters.

But putting that aside, I do think the exponential will continue, but there will be certain distinguished points on the exponential, and companies, individuals, countries will reach those points at different times. So, could there be some I talk about, is nuclear deterrent still in adolescence of technology? Is nuclear deterrent still stable in the world of AI?

I don't know, but that's an example of one thing we've taken for granted that the technology could reach such a level that it's no longer we can no longer be certain of it at least. Think of others. There are kind of points where if you reach a certain point, maybe you have offensive cyber dominance.

And every computer system is transparent to you after that, unless the other side has a kind of equivalent defense. So I don't know what the critical moment is or if there's a single critical moment, but I think there will be either a critical moment, a small number of critical moments, or some critical window where it's like AI is AI confers some large advantage from the perspective of national security, and one country or coalition has reached it before others. That that, you know, that that that you know, I'm not advocating that they're just like, okay.

We're in charge now. That's not that's not how that's not how I think about it. You know, that there's always the the other side is catching up.

There's extreme actions you're not willing to take, and and and it's not right to take, you know, to take complete to take complete control anyway. But but at at the point that that happens, I think people are gonna understand that the world has changed. And there there's gonna be some negotiation implicit or implicit about what what is the what is the post AI world order look like?

that paragraph was I think I said something like, you could take it even further and say x. So I wasn't I wasn't necessarily endorsing that that I wasn't necessarily endorsing that view. I, know, I was saying like, here's first, you know, here here's a weaker thing that I believe.

But, you know, I think I, you know, I think I said, you know, we have to worry a lot about authoritarians and, know, we should try and, you know, kind of kinda check them and limit their power. Like, you could take this kind of further, much more interventionist view that says, like, authoritarian countries with AI are these, you know, the the the you know, these kind of self fulfilling cycles that you can't that are very hard to displace, and so you just need to get rid of them from from the beginning. That that has exactly all the problems you say, which is, you know, if you were to make a commitment to overthrowing every authoritarian country, mean, they then they would take a bunch of actions now that like you know, that that that could could lead to instability.

So that that may or that just may not be possible. But the point I was making that I do endorse is that it is quite possible that today, the view or at least my view or the view in most of the Western world is is democracy is a better form of government than authoritarianism. But it's not like if a country's authoritarian, we don't react the way we reacted if they committed a genocide or something.

Right? And and I guess what I'm saying is I'm a little worried that in the age of AGI, authoritarianism will have a different meaning. It will be a graver thing.

And we have to decide one way or another how to deal with that. And the interventionist view is one possible view. I was exploring such views.

It may end up being the right view. It may end up being too extreme to be the right view. But I do have hope.

One piece of hope I have is there is we have seen that as new technologies are invented, forms of government become obsolete. I mentioned this in adolescence of technology where I said, you know, like feudalism was basically, you know, like a form of government. Right?

And and then when when we invented industrialization, feudalism was no longer sustainable. It no longer made sense. Why is that hope?

Why couldn't that imply that democracy is no longer gonna be a competitive system? It it could right. It it could go it could go either way.

Right? But but I actually so I these problems with authoritarianism. Right?

That the problems of authoritarianism get deeper. I just I wonder if that's an indicator of other problems that authoritarianism will have. Right?

In other words, people become because authoritarianism becomes worse, people are more afraid of authoritarianism. They work harder to stop it. It's it's more of a like, you have to think in terms of total equilibrium.

Right? I just wonder if it will motivate new ways of thinking about, with with with the new technology, how to preserve and protect freedom. And and even more optimistically, will it lead to a collective reckoning and, you know, a a a of a more emphatic realization of how important some of the things we take as individual rights are.

Right? A more emphatic realization that we just we really can't give these away. There's there we've seen there's no other way to live that actually works.

I I I am actually I am actually hopeful that I I guess one way to say it, it sounds too idealistic, but I actually believe it could be the case, is that is that dictatorships become morally obsolete. They become morally unworkable forms of government. And that and that and that the the the the crisis that that creates is is is sufficient to force us to find another way.

a kind of government to government decision and in national security terms, that's one lens, but there are a lot of other lenses. You could imagine a world where we produce all these cures to diseases, and the cures to diseases are fine to sell to authoritarian countries, the data centers just aren't. The chips and the data centers just aren't, the AI industry itself.

Another possibility is and I think folks should think about this, could there be developments we can make either that naturally happen as a result of AI or that we could make happen by building technology on AI, could we create an equilibrium where it becomes infeasible for authoritarian countries to deny their people private use of the benefit to the technology? You know? Are there are there are there are there equilibria where we can kind of give everyone in an authoritarian country their own AI model that kind of, you know, defends themselves from surveillance?

And there isn't a way for the authoritarian country to, like, crack crack down on this while while retaining power. I don't know. That that sounds to me like if that went far enough, it would be it would be a reason why authoritarian countries would disintegrate from the inside.

But but maybe there's a middle world where, like, there's an equilibrium where if they wanna hold on to power, the authoritarians can't deny kind of individualized access access to the technology. But I actually do have a hope for the for the for the for the more radical version, which is, you know, is it possible that the technology might inherently have properties or that by building on it in certain ways, could create properties that have this kind of dissolving effect on authoritarian structures? Now, we hoped originally, right, if we think about back to the beginning of the Obama administration, we thought originally that that social media and and the Internet would have that property and turns out not to.

But but I I don't know. What what if we could what if we could try again with with the knowledge of how many things could go wrong and that this is a different technology? I don't know that it would work, but it's worth a try.

Yeah. I think it's just it's very unpredictable. Like, there's first principles reasons why authoritarianism might be prevalent.

But very unpredictable. I I don't think I mean, we gotta we we just gotta we kind of we gotta recognize the problem, and then we gotta come up with 10 things we can try, and we gotta try those and then assess whether they're working or which ones are working, if any, and then try new ones if the old ones aren't working.

empower this country's act. What I would say is that, you know, we are we are about to be in a world where growth and economic value will come very easily. If right?

If we're able to build these powerful AI models, growth and economic value will come very easily. What will not come easily is distribution of benefits, distribution of wealth, political freedom. These are the things that are gonna be hard to achieve.

And so when I think about policy, I think that the technology in the market will deliver all the fundamental benefits almost faster than we can take them. That these questions about distribution and political freedom and rights are the ones will actually matter and that policy should focus on. Okay.

mean, philanthropy should obviously play some role as it has in the past. But I think better growth is and stronger if we can make it endogenous. Yeah.

So what are the relevant industries in an AI driven world? Look, there's lots of stuff. I said we shouldn't build data centers in China, but there's no reason we shouldn't build data centers in Africa.

Right? In fact, I think it'd be great to build data centers in Africa. As long as they're not owned by China, we build data centers in Africa.

I think that's a great thing to do. We should also build, you know, there's no reason we can't build, know, a pharmaceutical industry that's like AI driven. Like, you know, the the if if AI is accelerating accelerating drug discovery, then, you know, there will be a bunch of biotech startups.

Like, let's make sure some of those happen in the developing world. And certainly, during the transition, I mean, we can talk about the point where humans have no role, but humans will have still have some role in starting up these companies and supervising supervising the AI models. So let's make sure some of those humans are humans in the developing world so that fast growth can happen there as well.

Yeah. So I'm not sure I'd quite draw the distinction in that way. There may be two relevant distinctions here, which are I think you're talking about a mix of the two.

One is, should we give the model a set of instructions about do this versus don't do this? No. And the other, should we give the model a set of principles for how to act?

There, it's purely a practical and empirical thing that we've observed that by teaching the model principles, getting it to learn from principles, its behavior is more consistent, it's easier to cover edge cases, and the model is more likely to do what people want it to do. In other words, if you're like, don't tell people how to hotwire a car, don't speak in Korean, don't if you give it a list of rules, it doesn't really understand the rules, and it's kind of hard to generalize from them, you know, if if it's just kind of a, like, you know, list of do do's and don'ts. Whereas if you give it principles and then, you know, it has some hard guardrails, like don't make biological weapons.

But overall, you're trying to understand what it should be aiming to do, how it should be aiming to operate. So just from a practical perspective, that turns out to be just a more effective way to train the model. That's one piece of it.

That's the kind of rules versus principles trade off. Then there's another thing you're talking about, which is kind of like the corrigibility versus, like, you know, I would say, kind of intrinsic motivation trade off, which is like, how much should the model be a kind of I don't know, like a skin suit or something where you just kind of it just kind of directly follows the instructions that are given to it by whoever is giving it those instructions versus how much should the model have an inherent set of values and go off and do things on its own. And there, I would actually say everything about the model is actually closer to the direction of you know, it should mostly do what people want.

It should mostly follow the we're not trying to build something that, like, you know, goes off and runs the world on its own. We're actually pretty far on the corrigible side. Now now what we do say is there are certain things that the model won't do.

That it's like I think we say it in various ways in the constitution, that under normal circumstances, if someone asks the model to do a task, it should do that task. Should be the default. But if you've asked it to do something dangerous or if asked it to kind of harm someone else, then the model is unwilling to do that.

So I think there's two There's maybe three kind of sizes of loop here. Three ways to iterate. One is you can iterate we iterate within Anthropic.

We train the model. We're not happy with it, and we kinda change the constitution. And I think that's good to do.

Putting out publicly, making updates to the constitution every once in a while saying, here's a new constitution. Right. I think that's good to do because people can comment on it.

The second level of loop is different companies will have different constitutions. And I think it's useful for like Anthropic puts out a constitution and the Gemini model puts out a constitution and other companies put out a constitution and then they can kind of look at them, compare. Outside observers can critique and say, I like this one, this thing from this constitution, and this thing from that constitution.

And then that creates some soft incentive and feedback for all the companies to take the best of each elements and improve. Then I think there's a third loop, which is society beyond the AI companies and beyond just those who comment on the constitutions without hard power. And there, we've done some experiments.

A couple years ago, we did an experiment with, I think it was called the collective intelligence project to basically poll people and ask them what should be in our AI constitution. And I think at the time we incorporated some of those changes. And so you could imagine with the new approach we've taken to the constitution, doing something like that, it's a little harder because it's like that was actually an easier approach to take when the constitution was like a list of dos and don'ts.

At the level of principles, it has to have a certain amount of coherence. But you still imagine getting views from a wide variety of people. And I think you could also imagine, and this is like a crazy idea, but hey, this whole interview is about crazy ideas.

Right? So you could even imagine systems of representative government having input. Right?

I wouldn't do this today because the legislative process is so slow. This is exactly why I think we should be careful about the legislative process and AI regulation. But there's no reason you couldn't in principle say, like, you know, all AI you know, all AI models have to have a constitution that starts with, like, these things.

And then, like, you can append you can append other things after it, but, like, there has to be this special section that, like, takes precedent. I wouldn't do that. That's too rigid.

That that sounds, you know, that that that that sounds kind of overly prescriptive in a way that I think overly aggressive legislation is. But, like, that is a thing you could you know, like like, that is that is a thing you could try to do. Is is there some much less heavy handed version of that?

Maybe.

I think that vision has things to recommend it and things that will kind of go wrong with it. I think it's an interesting, in some ways, compelling vision, but also things will go wrong with it that you hadn't imagined. So I like loop two as well, but feel like the whole thing has got to be some mix of loops one, two, and three, and it's a it's a matter of the proportions.

Right? I I think that's gotta be the the answer.

I think a few things. One is at every moment of this exponential, the extent to which the world outside it didn't understand it. This a bias that's often present in history where anything that actually happened looks inevitable in retrospect.

So I think when people look back, it will be hard for them to put themselves in the place of people who are actually making a bet on this thing to happen that wasn't inevitable, that we had these arguments, like the arguments that, you know, that I make for scaling or that continual learning will be solved, that some of us internally in our heads put a high probability on this happening. But it's like there's a world outside us that's not acting on that's not kind of not acting on that at all. And and and I think I think the the weirdness of it I I think, unfortunately, like, the insularity of it, like, you know, if if we're one year or two years away from it happening, like, average person on the street has no idea.

That's one of the things I'm trying to change, like, with the memos, with talking to policymakers, but, like, I don't know. I think I I I think that's just a that's just like a crazy that's just like a crazy thing. Yeah.

Finally, I would say, and this probably applies to almost all historical moments of crisis, how absolutely fast it was happening, how everything was happening all at once. And so decisions that you might think, you know, were kinda carefully calculated, well, actually, have to make that decision, and then you have to make 30 other decisions on the same day because it's all happening so fast. You don't even know which decisions are gonna turn out to be consequential.

So, one of my, I guess, worries, although it's also an insight into kind of what's happening is that some very critical decision will be some decision that someone just comes into my office and is like, Dario, you have two minutes. Should we do thing A or thing B on this someone gives me this random half page memo and is like, should we do A or B? And I'm like, I don't know, I have to eat lunch, let's do B.

And that ends up being the most consequential thing ever.

So that was the tie and I was having a hard time finding the time to actually find it, to actually write it. But I actually think about this in a broader way. I actually think it relates to the culture of the company.

So I probably spend a third, maybe 40% of my time making sure the culture of Anthropic is good. As Anthropic has gotten larger, it's gotten harder to just get involved in directly involved in the training of the models, the launch of the models, the building of the products. It's 2,500 people.

It's like, there's just I have certain instincts, but there's only it's very difficult to get involved in every single detail. I try as much as possible. But one thing that's very leveraged is making sure Anthropic is a good place to work.

People like working there. Everyone thinks of themselves as team members. Everyone works together instead of against each other.

We've seen as some of the other AI companies have grown without naming any names, we're starting to see decoherence and people fighting each other. And I would argue there was even a lot of that from the beginning, that it's gotten worse. But I think we've done an extraordinarily good job, even if not perfect, of holding the company together, making everyone feel the mission, that we're sincere about the mission, and that everyone has faith that everyone else there is working for the right reason, that we're a team, that people aren't trying to get ahead of each other's expense or backstab each other, which again, I think happens a lot at some of the other places.

And how do you make that the case? I mean, it's a lot of things. It's me, it's Daniella who runs the company day to day.

It's the co founders. It's the other people we hire. It's the environment we try to create.

But I think an important thing in the culture is I, and just the other leaders as well, but especially me, have to articulate what the company is about, why it's doing what it's doing, what its strategy is, what its values are, what its mission is, and what it stands for. And, you know, when you get to 2,500 people, you can't do that person by person. You have to write or you have to speak to the whole company.

This is why I get up in front of the whole company every two weeks and speak for an hour. It's actually I mean, I wouldn't say I write essays internally. I do two things.

One, I write this thing called the DVQ, Dario Vision Quest. I wasn't the one who named it that. That's the name it it received, and it's one of these names that I kind of I tried to fight it because it made it sound like I was, like, going off and smoking peyote or something, but but the name just stuck.

So I get up in front of the company. Every two weeks, I have, like, a three or four page document, and I just talk through three or four different topics about what's going on internally, the models we're producing, the products, the outside industry, the world as a whole as it relates to AI and geopolitically in general, just some mix of that. And I just go through very, very honestly, I just go through and I just say, this is what I'm thinking, this is what anthropic leadership is thinking.

And then I answer questions. That direct connection, I think, has a lot of value that is hard to achieve when you're passing things down the chain, six levels deep. Large fraction of the company comes to attend, either in person or virtually.

And it really means that you can communicate a lot. And then the other thing I do is I just I have a channel in Slack where I just write a bunch of things and comment a lot. And often that's in response to just things I'm seeing at the company or questions people ask or like we do internal surveys and there are things people are concerned about and so I'll write them up.

And I'm like, I'm very honest about these things. I just say them very directly. And the point is to get a reputation of telling the company the truth about what's happening, to call things what they are, to acknowledge problems, to avoid the sort of corpo speak, the kind of defensive communication that often is necessary in public because the world is very large and full of people who are interpreting things in bad faith.

But if you have a company of people who you trust and we try to hire people that we trust, then then, you know, you can you can you know, you can you can really just be entirely unfiltered. And, you know, I think I think that's an enormous strength of the company. It makes it a better place to work.

It makes people more, you know, more of the sum of their parts and increases likelihood that we accomplish the mission because everyone is on the same page about the mission, and everyone is debating and discussing how best to accomplish the mission.