Scaling Test Time Compute to Multi-Agent Civilizations — Noam Brown, OpenAI

since working on Cicero and then now personally playing it? When you work on these games, you kinda have to understand the game well enough to, like, be able to debug your bot. Because if the bot does something that's, like, really radical and, like, that Tim humans typically wouldn't do, you're not sure if that's, like, a mistake or if that's just like, if it's a bug in the system or it's actually just, the bot being brilliant.

When we were working on Diplomacy, I kind of did this deep dive, trying to understand the game better. I played in tournaments, I watched a lot of tutorial videos and commentary videos on games. And over that process, I got better.

And then also seeing the bot, like, the way it would behave in these games, like, sometimes it would do things that humans typically wouldn't do. And that taught me about the game as well. We released Cicero, we announced it in, like, late twenty twenty two, I still found the game, like, really fascinating, and so I, like, kept up with it.

I, like, continue to play, and that led to me winning the championship in the world championship in 2025, so just a couple months ago. There's always a question of, like, Centaur systems where humans and machines work together. Like, was there an equivalent of what happened in Go where you updated your play style?

Because If you're asking if I used Cicero when I played in the tournament, the answer is the answer is no. Seeing the way the bot played and, like, taking inspiration from that, I think, did help me in in the tournament. Yeah.

Yeah.

Ask to try to to tell if the person they're playing with is a bot or a human. Yeah. Like, that's the one thing you're worried about when you started.

It was really interesting when we were working on Cicero because, like, you know, we didn't have the best language models. We were really bottlenecked on the quality of the language models, and sometimes the bot would do would say, like, bizarre things. Like, you know, 9099% of time it was fine, but then like every once in a while it would say this like really bizarre thing, like it would just hallucinate about something.

Somebody would reference something that they said earlier in a conversation with the bot, and the bot would be like, I have no idea what you're talking about. I never said that. And then the person would be like, look, you could scroll up in the chat and it's literally right there, and the bot would be like, no, you're lying.

Talk to Windows. And when it does these kinds of things, people just kind of shrugged it off as like, oh, that's just the person's tired or they're drunk or whatever, or they're just trolling me. But I think like that's because people weren't looking for a bot, they weren't expecting a bot to be in the games.

We were actually really scared because we were afraid that people would would figure out at one point that that there's a bot in these games, and then they would just like always be on the lookout for it and they would always be and if you're if you're looking for it, you're able to spot it, that's the thing. So I think now that it's announced and that people know to look for it, I think they would have an easier time spotting it. Now that said, the language models have also gotten a lot better since 2022.

It's adversarial. Yeah. So at this point, like, know, truth is, you know, GPT four o and, like, o three, these models are, like, passing the Turing test.

a lot of the AI safety community was really happy with the research and the way it worked, because it was a very controllable system. We conditioned Cicero on certain concrete actions, and that gave it a lot of steerability to say, like, okay, well, it's going to pursue a behavior that we can, like, very clearly interpret and and very clearly define. It's not just like, oh, it's a language model, like, running loose and doing whatever it feels like.

No. It's actually, like, pretty steerable, and there's this whole reasoning system that steers the way the language model interacts with the human. Actually, a lot of researchers reached out to, like, reached out to me and said, like, we think this is, like, potentially a really good way to achieve safety with these systems.

and would you would you expect a lot more difference? I have not. I think I said this on on Twitter at one point that I think this would be a great benchmark.

I would love to see all the leading bots play a game of diplomacy with each other and see, like, who does best. And I think a couple of people have, like, taken inspiration from that and are actually, like, building out these benchmarks and, like, eval ing the models. My understanding is that they don't do very well right now.

And I think that everything that's unfolded since then has been pretty on track for what I expected. So I wouldn't say that my perception of where where things are going or has honestly changed that much. I think that we're gonna continue to see, as I said before, that we're gonna see this paradigm continue to progress rapidly, and I think that that's true even today that we we saw that with, like, going from o one preview to o one to o three, consistent progress, and we're gonna continue to see that going forward.

And I think that we're going to see a broadening of what these models can do as well. You know? Like, we're going to start seeing agentic behavior.

We're already starting to see agentic behavior. Like, honestly, for me, o three, I've been using it a ton in my day to day life. I just find it so useful, especially the fact that I can now browse the web and, do meaningful research on my behalf, it's kind of like a mini deep research that you can just get a response in three minutes.

okay, we're seeing these reasoning models exceed in math and coding and these easily verifiable domains, but are they ever going to succeed in domains where success is less well defined? I'm surprised that this is such a common perception because we've released deep research and people can try it out. People do use it.

It's very popular. And that is very clearly a domain where you don't have an easily verifiable metric for success. It's very like, what what is the best research report that you could generate?

And yet these models are doing extremely well at this at this domain. So I think that's, like, an existence proof that these models can succeed in tasks that don't have as easily verifiable rewards. Is it because there's also not necessarily like a wrong answer?

Like, there's a spectrum of deep research quality. Right? You can have like a report that like looks good, but the information is kinda so and so, and then you have a great report.

Do you think people have a hard time understanding the difference when they get the result? My impression is that people do understand the difference when they get a result, and I think that they're surprised at how good the deep research results are. They're certainly it's not not a 100%, it could be better, and we're gonna make it better.

But I think people can tell the difference between a good report and a bad report, and and certainly and and a good report and and a mediocre report. And that's enough to kind of feed the the loop later to build the product and improve the model performance. I mean, think if you're in a situation where people can't tell the difference between the outputs, then it doesn't really matter if you're, like, you know, hill climbing on on progress.

These models are gonna get better at domains where there is a measure of success. Now I think this idea that it has to be, like, easily verifiable or something like that, I don't think that's true. I think that you can have you can have these models do well even in domains where success is a very difficult to define thing, could sometimes even be subjective.

People lean on a lot.

One thing that I think is underappreciated is that the models the pretrained models need a certain level of capability in order to really benefit from this, like, extra thinking. This is kind of why you you've seen the reasoning paradigm emerge around the time that it did. I think it could have happened earlier, but if you try to do the reasoning paradigm on top of g b d two, I don't think it would have gotten you almost anything.

Is this emergence? Hard to say if it's emergence necessarily, but, like, I haven't done the, you know, the measurements to really define that clearly. But I think it's pretty clear.

You know, people try chain of thought with GPD, like really small models, and they saw that it just didn't really do anything. Then you go to bigger models and it starts to to give a lift. I think there's lot of debate about, like, the extent to which this kind of behavior is emergent, but clearly there is a difference.

So it's not like there are these two independent paradigms, I think that they are related in the sense that you need a certain level of system one capability in your models in order to have system two, to be able to benefit from system two. Yeah. I have played tried to play amateur neuroscientists before and try to compare it to the evolution of the brain Mhmm.

And how you have to evolve the cortex first before you evolve the other parts of the brain, and perhaps that is what we're doing here. Yeah. And you could argue that actually this is not that different from, like, I guess, the the system one, system two paradigm because, you know, if you ask, like, a pigeon to think really hard about playing chess, you know, it's not gonna get that far.

It's, know, it doesn't matter if it, like, thinks for a thousand years, it's, like, not gonna be able to be better at playing chess. So maybe you do still also with animals and humans that you need a certain level of intellectual ability just in terms of system one in order to benefit from system two as well. Yeah.

Just as side tangent, does this also apply to visual reasoning?

Does that apply to other modalities too? I I think the evidence is yes. It depends on exactly the kinds of questions that you're asking.

Like, there are some questions that I think don't really benefit from system two. I think geoguests are certainly one where where you do benefit. I think image recognition, if if I had to guess, it's like one of those things that you probably benefit less from system two thinking.

Because you know it or you don't. Yeah. Exactly.

There's no way. Yeah. And the the thing the thing I typically point to is just like information, like retrieval.

If somebody asks you like, when was this person born? And you don't have access to the web. Then you either know it or you don't, and you can sit there and you can think about it for a long time.

Maybe you can make an educated guess, so you can say, like, well, this person was, like probably lived around this time, and so this is, like, a rough date. But you're not gonna be able to, like, get the date unless you've actually just just know it. But, like, spatial reasoning like tic tac toe might be better because you have all the information there.

Yeah. And I think it's true that, like, with tic tac toe, we see that, like, g p d 4.5 falls over.

You know, it plays decently well. I shouldn't say it falls over. It it does reasonably well.

You control the board. It can make legal moves, but it will make mistakes sometimes. And if you really need that system two to enable it to play perfectly.

Now it's possible that if you got to g b d six and you just did system one, it would also play perfectly.

that you have in your in in the system one like, this is the same thing with humans, you know, like humans are when they're playing for the first time a game like chess, and they can apply a lot of system two thinking to it.

in my view is that the ideal harness is no harness. You know? I think harnesses are are, like, a crutch that eventually we're gonna be able to move beyond.

So only two calls. And you could ask. Yeah.

You could just ask o three. And, actually, you know, it's interesting because, like, when this playing Pokemon thing kind of, like, emerged as as this, like, you know, benchmark, I was actually, like, pretty opposed to eval ing this with our with our, like, OpenAI models because my feeling is, okay, if we're gonna do this eval, let's just do it with o three. You know?

How far does o three get without any harness? How far does it get playing Pokemon? And the answer is like not very far, you know?

Mhmm. And that's fine. I think it's fine to have an eval where the models do terribly.

And I don't think the answer to that should be like, well, let's build a really good harness so that now it can do well on this eval. I think the answer is like, okay. Well, let's just, like, improve the capabilities of our models so they can do well at everything, and then they also happen to make progress on this eval.

Would you consider things like checking for a valid move, a harness, or is this in the in the model? You know, like chess. It's like you can either have the model learn in system one what moves are valid and what it can and cannot do versus in system two figuring out I think I think there's like a lot of this is design questions.

Like for me, I think you should give the model the ability to check if a move is legal if you want. Like that that could be an option in the environment of like, okay, here's a, you know, an action that you can like a tool call that you can make to see if an action is legal. If it wants to use that, it it can.

And then there's, like, design question of, like, well, what do you do if the model makes an illegal move? And I think it's totally reasonable to say, like, well, if they make an illegal move, then they lose the game. Like, I don't know.

What what happens when a human makes an illegal move in a game of chess? Mhmm. I actually don't know.

I have a pretty chest just not allowed to? Yeah. Like, do you just lose the game?

I don't know. So if that's if that's the case, then I think it's totally reasonable to say, like, yeah, we're gonna have an eval where that's also the criteria for for the AI models. Yeah.

But I think, like, maybe one way to interpret that in sort of researcher terms is, are you allowed to do search?

like, a tool call to check whether a move is legal or illegal is different from actually making that move, and then seeing whether it ended up being legal or illegal. Right? Mhmm.

So if that tool call is available, think it's totally fine to make that tool call and check whether a move is legal or illegal. I think it's different to have the model say, oh, I'm making this move Yeah. And then, you know, it gets feedback that like, oh, you made an illegal move.

And so then it's like, just kidding. Like, I'm gonna do something else now. So so that's that's the distinction I'm I'm drawing.

Some people have tried to classify that second type of playing things out as test time compute. You would not classify that as test time compute. There's a lot of reasons why you would not want to rely on that paradigm when you're going to the imagine you have a robot, know, and your robot like takes some action in the and it like breaks something, and you just like, oh you can't say like, just kidding, didn't mean to do that, I'm gonna undo that action, like the thing is broken.

So if you want to simulate what would happen if I move the robot in this way, and then in your simulation you saw that this thing broke and then you decide not to do that action, that's totally fine. But you can't just like undo actions that you've taken in the world. There's a couple more things I wanted to cover in this rough area.

I think it's possible for a dumb model to recognize that a problem is really hard and that it won't be able to solve it and then route it to a more capable model. But it's also possible for a dumb model to be fooled or to be overconfident. I don't know.

I think there's a real trade off there. But I I will say, like, I think I think there are a lot of things that people are building right now that will eventually be washed away by scale. So I think harnesses are a good example where I think eventually the models are going to be and I think this actually happened with the reasoning models.

Like, before the reasoning models emerged, there was, like, all of this work that went into engineering these, like, agentic systems that, like, made a lot of calls to g p d four o or, like, these these non reasoning models to get reasoning behavior. And then it turns out, like, oh, we just, like, created reasoning models, and they you don't need this, like, complex behavior. In fact, in in many ways, makes it worse.

Like, you just give the reasoning model the the same question without any sort of scaffolding, and it just does it. Now that you can still and so people are building scaffolding on top of reasoning models right now, but I think in many ways, like, those scaffolds will also just be replaced by the reasoning models and models in general becoming more capable. And similarly, I think things like model, like these routers, you know, we've said pretty openly that we want to move to a world where there is a single unified model.

And in that world, you shouldn't need a router on top of the model. So I think that the router issue will eventually be solved also. Like, you're building the router into the model kind of weights itself.

I don't think there'll be a a benefit for like, I don't I shouldn't say it because it's I could be wrong about this. Like, know, it's certainly maybe there's, you know, reasons to route to different model providers or whatever, but I think that routers are going to eventually go away. And I can understand why it's worth doing it in the short term, because like the fact is it is beneficial right now, and if you're building a product and you're getting a lift from it, then it's it's worth doing right now.

Of the tricky things I'd I'd imagine that a lot of developers are facing is that like you kind of have to plan for where these models are going to be in six months and twelve months, and that's like very hard to do because things are progressing very quickly. You know, you don't want to spend six months building something and then just have it be totally washed away by scale. But I I think I would encourage developers, like, when when they're, you know, building these kinds of things, like scaffolds and and routers, keep in mind that the field is evolving very rapidly.

You know, things are gonna change in three months, let alone six months, and that might require radically changing these things around or or tossing them out completely. So don't spend six months building something that might get tossed down in six months. It's so hard, though.

Everyone says this, and then, like, no one has concrete suggestions on how.

it's really about specializing the models for the data that you have. And I think that something that's like worth worth looking into for for developers.

reinforcement fine tuning, you're collecting data that's going to be useful as the models improve as well. So if we come out with future models that are even more capable, you could still fine tune them on your data. That's I think actually a good example where you're building something that's going to complement the model scaling and becoming more capable rather than necessarily getting washed away by the scale.

Yep.

So Ilya, for me, I saw that in all of these domains that I'd worked on in poker and Hanabi and diplomacy, having the models think before acting made a huge difference in performance, like orders of magnitude difference. Like 10,000 times is the Yeah. Like, you know, a thousand to a 100,000 times, like it's the the equivalent of a model that's like a thousand to a 100,000 times bigger.

And in language models, you weren't really seeing that, that the model the models would just respond instantly. Some people in the field in the LLM field were, like, convinced that, like, okay. We just keep scaling pre training.

We're gonna get to superintelligence. And I was kinda skeptical of that perspective. In late twenty twenty one, I was having a meal with Ilya.

He asked me what my HII timelines are, a very standard SF question, and I told him, like, look, I think it's actually quite far away because we're gonna need to figure out this reasoning paradigm in a very general way. And with things like LMs, LMs are very general, but they don't have a reasoning paradigm that's very general. And until they do, they're gonna be limited in what they can do.

You know, like, we're gonna scale it each other. We're gonna scale these things up by a few more orders of magnitude, they're gonna become more capable, but we're not gonna see superintelligence from just that. And like yes, if we had a quadrillion dollars to train these models, then maybe we would, but like you're gonna hit the limits of what's economically feasible before you get to superintelligence, unless you have a reasoning paradigm.

And I was convinced incorrectly that the reasoning paradigm would take a long time to figure out because it's like this big unanswered research question. And, you know, Ilya agreed with me, and he said, like, yeah. You know, I think we need this, like, additional paradigm.

But his take was that, like, maybe it's not that hard. I I didn't know it at the time, but, like, he and others at OpenAI had also been thinking about this. They'd also been thinking about RL.

They've been working on it, and I think they had some success. But, like, you know, with most research, like, it does you have to iterate on things. You have to try out different ideas.

You have to, yeah, try different things. And then also, as the models become more capable, as they become faster, it becomes easier to iterate on experiments.

we did end up getting a big success with with reasoning. And I but I think it was, in many ways, a gradual thing. To some extent, was gradual.

You know? Like, there were signs there were signs of life, and then we, like, you know, iterated and tried out some more things. We got, like, better signs of life.

I think it was around, like, no November 2023 or October 2023 when I think I was convinced that we had, like, very conclusive signs of life that, like, oh, this this was going to be this is the paradigm, and it's gonna be a big deal. That was in many ways a a gradual thing. I think what OpenAI did well is, like, when we got those signs of life, they recognized it for what it was and invested heavily in in scaling it up.

And I think that's that's ultimately what what led to reasoning models arriving when they did.

There's always different opinions about this stuff. I think there were some people that felt that pretraining was all we need, and we scaled it up to infinity and we're there. I think a lot of the leadership actually at OpenAI recognized that there was another paradigm that was needed, and that was why they were investing all this, like, research effort into this, like, RL stuff.

And I think that's also to the credit of OpenAI that, like, okay, yes, they figured out the pre training paradigm and they were very focused on scaling it up. In fact, the vast majority of resources were focused on scaling it up, but they also recognized the value that that something else was gonna be needed and it was worth researching putting researcher effort into into other directions to figure out what that extra paradigm was going to be. There was a lot of debate about, first of all, like, what is that extra paradigm?

So I think a lot of the researchers looked at reasoning and and RL was not really about scaling test on compute, it was more about data efficiency. Because, you know, the feeling was that, well, we have tons and tons of compute, but we actually are more limited by data. So there's there's the data wall, and we're gonna hit that before we hit limits on on the compute.

So how do we make these algorithms more data efficient? They are more data efficient, but I think that also, like, they are also just, like, the equivalent of scaling up compute also by a ton. That was interesting.

There was, like, a of debate around, like, okay. What what exactly are we doing here? And then I think also, even we got the signs of life, I think there was a lot of debate about the significance of it.

That, like, okay. How much should we invest in scaling up this paradigm? I think especially when you're when you're in a small company, like, you know, OpenAI, like, in 2023 was not as big as it is today, and compute was more constrained than it is today.

And if you're investing resources in in a direction, that's coming at the expense of something else. And so if you look at these signs of life on reasoning and you're saying like, okay. Well, this looks promising.

We're gonna scale this up by a ton and invest a lot more resources into it, where are those resources coming from? You have to make that tough call about where to draw the resources from, and that is a very controversial, very difficult call to make that makes some people unhappy. And I think there was debate about whether we're focusing too much on this paradigm, whether it's really a big deal, whether we would see it generalize and do various things.

And I remember it was interesting that I I talked to somebody who left OpenAI after we had discovered the reasoning paradigm, but before we announced o one Mhmm. And they ended up going to a competing lab. I saw them afterwards, after we announced o one, and they told me that, like, at the time, they really didn't think this, like, reasoning thing, like, this these o series, the strawberry models were, like, that that big of a deal.

It was, like, they thought we were making a bigger deal of it than it really deserved to be. And then when we announced o one and they saw the reaction of their coworkers at this competing lab about how everybody was like, oh crap, this is a big deal, and they pivoted the whole research agenda Oh my god. To focus on this, that then they realized like, oh, actually this maybe is a big deal.

the best research is obvious in retrospect,

inefficiency that machine learning has that will just always be there compared to humans? I think it's a good point that if you look at the amount of data these models are trained on and you compare it to, like, the amount of data that a human observes to get the same performance, I guess pre training, it's a little hard to make an apples to apples comparison because, like, I don't know, how how many tokens does a baby actually absorb when they're developing. But I think it's a fair statement to say that these models are are less data efficient than humans, and I think that that's an unsolved research question and probably one of the most important unsolved research questions.

because you can just you can you we we can increase the supply of data out of the existing set of the worlds and humans. I guess that's okay. So a couple of thoughts on that.

Like, one is that the answer might be an algorithmic improvement. Like, maybe maybe algorithmic improvements do lead to greater data efficiency. And the second thing is that, like, it's not like humans learn from just reading the Internet.

So I think it's certainly easiest to learn from just, like, data that's on the Internet, but I don't think that's, like, the limit of what data you could collect. The last follow-up before we change topics to coding, any other just anecdotes or insights from Ilya just in general? Because, like, you've worked with him, so there's not that many people that we can talk to that have worked with him.

I think I've just been very very impressed with his vision. That I think, like especially when I joined and I saw, you know, the internal documents at at OpenAI of, like, what he had been thinking about back in, like, 2021, 2022, even earlier. I I was very impressed that he had a clear vision of, like, where this was all going and what was needed.

I mean, I think it is true that I mean, one of OpenAI's big success was betting on the scaling paradigm. It is just kind of odd because, you know, they were not the biggest lab, you know. It was, like, difficult for them to scale.

Back then, it was much more common to do, like, a lot of small experiments, more academic style.

universities or or, you know, research labs as they traditionally existed.

Lately, I've been using Windsurf and Codex, like actually a lot of Codex. I've been having a lot of fun. Like, you just give it a task and it just goes off and does it and comes back five minutes later with like a, you know, pull request.

And is it core research task or like side stuff that you don't super care about? I wouldn't say it's like side stuff. I would say basically anything that I would normally try to code up, I try to do it with Codecs first.

For you it's free, but yeah. For everybody, it's free right now. And I think that's partly because it's the it's the most effective way for me to do it, and also it's good for me to get experience working with this technology and then also, like, seeing the shortcomings of it.

It just helps me, like, better understand, like, okay, this is the the limits of these models and, like, what we need to push on next. Have you felt the AGI? I've felt the AGI multiple times.

Yes.

feel the AGI. It's kind of funny how, like, you feel the AGI and then you get used to it very quickly, you know, so so it's really like Dissatisfied with like where it's lacking. Yeah.

I know, know, it's it's magical one day. I was actually looking back at the old Sora videos when they were announced. Yeah.

Because like, remember when Sora came out, it was just like The biggest it was ever. It was just magical. You look at that and you're like, oh, it's like it's really here, like this is AGI.

But if you look at it now and it's kind of like, oh, know, the people like don't move like very organically and it's like there's, like, a lack of consistency in some ways. And you see all these flaws in it now that you just didn't really notice when it was first came out. And, yeah, you get used to this technology very quickly.

And but I think what's cool about it is that because it's developing so quickly, you get those feel the AGI moments, like, every few months. So something else is gonna come out and just like it's magical to you. And and then you get used to it very quickly.

Yeah. What are your Windsurf pro tips now that you've immersed in it? I think one thing I was surprised by is how few people I mean, maybe your audience is gonna be more comfortable with reasoning models and, like, use reasoning models more, but I'm surprised at how many people don't even know that o three exists.

Like, I've been using it day to day. It's basically replaced Google Search for me. Like, I just use it all the time.

Like and also for things like coding, like, I I tend to just use the reasoning models. My suggestion is, like, if people are not have not tried the reasoning models late yet, because, like, honestly, like, we do like, people love them. People that use it love them.

Obviously, a lot more people use gbt four o and just, like, the default on what on chat gbt and that kind of stuff. I think it's worth trying the reasoning models. Like, I think people would be surprised at what they can do.

I use Windsurf daily, and they still haven't actually enabled it as, like, a default in Windsurf. Like, I always have to dig up, like type in o three and then if and then it's like, oh, yeah. That that exists.

Mhmm. It's it's it's weird. I would say, like, my struggle with it has been that it's takes so long to reason and actually break out of flow.

I think that is true, yes. And I think this is one of the advantages of Codex that like, okay, you can give it a task that's kind of self contained and like it can go off and do its thing and come back ten minutes later. And I can see that if you're doing if you're using this thing as like more like a like a pair programmer kind of thing, then, yeah, you wanna use GP 4.

1 or something like that. What do you think are the most broken part of the development cycle with AI?

I think it's really on us to build a lot more stuff. These models are very limited in in in some ways. I think I find it frustrating that, you know, you ask them to do something and then they spend ten minutes Mhmm.

Doing it, and then you ask them to do something pretty similar, and then they go spend ten minutes doing it, and like, you know, it's I I think I describe them as like, they're geniuses, but it's their first day on the job, you know, and that's like kind of annoying. Like even the the smartest person on Earth when they're when it's their first day on the job, you know, they're not gonna be like as useful as you would like them to be.

you're just limited by how many hours there are in the day. Right. You know?

So progress can only proceed so quickly.

looking at this time next year in terms of how things are how what models we're able to do that they're not able to today? I don't think it's gonna be limited to ASUI. You know?

I think I don't think it's gonna be limited to software engineering. I think it's gonna be able to do a lot of remote work kind of tasks. Yeah.

Like, freelancer type Upwork. Yeah. Or just, like, even things that are not necessarily software engineering.

Okay. So the way I think about it is, like, anybody that's doing a remote work kind of job, I think it's valuable to become familiar with their technology and, like, kinda get a sense of, like, what it can do, what it can't do, what it's good at, what it's not good at. Because I think the the breadth of things that it's gonna be able to do is gonna expand over time as well.

for things like virtual assistants, the the models, like, if they're aligned well, they could end up being, like, really preferable for that kind of work, you know? If there's always this like principal agent problem, where if you delegate a task to somebody, then like, are they really aligned with like doing it as you would want it to be done and Just as cheaply, as quickly as they can. Yeah.

Yeah. And so if you have an AI model that's, like, actually really aligned to you and your preferences, then that can end up doing a way better job than a human could. Well, not not that it's doing a better job than a human could, but, like, it's doing a better job than a human would.

I wonder where they diverge. Okay. So I think where it diverges is, like, what do you want to align the models to?

Like that that's I think a difficult question, you know? Like you could say like you wanted to align it to the user. Okay.

Well, what happens if the user wants to build a novel virus that's gonna wipe out half of humanity? That's safety alignment. Yeah.

So there's a question of like I think alignment I think they're related, you know? And I think the the the big question is, like, what are you aligning towards?

anything from there? Yeah. There's hasn't really been announcements on this.

I think we're working on cool stuff, and I think we'll get to announce some cool stuff at some point. I think the team in many ways it's actually a misnomer because we're working on more more than just multi agent. Multi agent is one of the things we're working on.

Some other things we're working on is just like being able to scale up test time compute by by a ton. So how you know, we get these models thinking for fifteen minutes now. How do we get them to think for hours?

How do we get them to think for days, even longer, and be able to solve incredibly difficult problems? So that's one direction that we're pursuing. Multi agent is another direction.

And here, I think there's a few different motivations. We're interested in, like, both the collaborative and the competitive aspect of multi agent. I think the way that I describe it is people often say in AI circles that humans occupy this very narrow band of intelligence, and AIs are just gonna, like, quickly catch up and then surpass, like, this band of intelligence.

And I actually don't think that the band of of human intelligence is that narrow. I think it's actually quite broad. Because if you compare anatomically identical humans from, you know, caveman times, they didn't get that far in terms of, like, you know, what we would consider intelligence today.

Right? Like, they're not putting a man on the moon. You know?

They're not, like, building semiconductors or nuclear reactors or anything like that. And and we have those today even though we as humans are not anatomically different. And so what's the difference?

Well, I think the difference is that you have thousands of years, a lot of humans, billions of humans cooperating and competing with each other, building up civilization over time. The technology that we're seeing is the product of this civilization. And I think, similarly, the AIs that we have today are kinda like the cavemen of AI.

And and I think that if you're able to have them cooperate and compete with billions of AIs over a long period of time and build up a civilization, essentially. The things that they would be able to produce and answer would be far beyond what is possible today with with the AIs that we have today.

we're until we have something to announce, which I think that we Yeah. Yeah. Think that we will in the not too distant future, I think I'm going to be a bit vague about like exactly what we're doing.

But I will say that the way that we are approaching multi agent in the details and the the way we're actually going about it is I think very different from how it's been done historically and how it's being done today by by other places. I've been in the multi agent field for a long time. I've kind of felt like the multi agent field has a bit misguided in some ways in the the things that the the approaches that the field has taken and, like, the way that's been approached.

Think that a lot of the approaches that have been taken have been very heuristic k. And haven't really been following, like, the bitter lesson approach to scaling and research.

like how to exploit somebody? I'm guessing your audience is probably not super familiar with poker terminology, so I'll just like explain this a bit. A lot of people think that poker is just like a luck game, that's not true.

It's actually like, there's a lot of strategy in poker. So you can win consistently in poker if you're playing the right strategy. So there's different approaches to poker.

One is game theory optimal. This is like you're playing an unbeatable strategy and expectation. Like, you're just unexploitable.

It's kinda like in rock, paper, scissors. You can be unbeatable on rock, paper, scissors if you just randomly choose between rock, paper, and scissors with equal probability. Because no matter what the other guy does, know, they're not gonna be able to exploit you or you're gonna win you're gonna, like, not lose an expectation.

Now a lot of people hear that, and they think, like, well, that also means that you're not going to win an expectation because you're just playing totally randomly. But in poker, if you play the equilibrium strategy, it's actually really difficult for the opponents to figure out how to tie you, and they're gonna end up making mistakes that will lead you to win over the long run. It might not be a massive win, but it is going to be a win.

If you play enough hands for a long enough period of time Mhmm. You're you're going to win in expectation. Now there's also exploitative poker, and the idea here is that you're trying to spot weaknesses in how the opponent plays.

You know? Maybe they're maybe they're not bluffing enough, or maybe they fold too easily to a bluff. And so you start adapting from the game theory optimal balance strategy of, like, you bluff sometimes, you you don't bluff sometimes, to then playing a very unbalanced strategy that's like, oh, I'm just gonna, like, bluff a ton against this person because they always fold whenever I bluff.

Now the key is that there's a trade off here because if you're taking this exploitative approach, then you're opening yourself up to exploitation as well. And so you have to choose this balance between playing a defensive game theory optimal policy that guarantees you're not going to lose but might not make you as much money as you potentially could versus playing an exploitative strategy that could be much more profitable, but also it creates weaknesses that the opponents could take advantage of and and trick you. And there's no way to to perfectly balance the two.

It's kinda like in rock paper scissors, if you notice somebody is playing paper for five times in a row, you might think like, oh, they're they they have a weakness in their strategy. I should just be throwing scissors, and I'm gonna take advantage of them. And so on the sixth time you throw scissors, but actually that's the time when they throw a rock.

You know? So you and you never really know. So you always have this trade off.

The poker AIs that have been extremely successful and, like, my background is, like, I worked on AI for poker for several years during grad school and made the first superhuman no limit poker AIs. The approach that we took was this game theory optimal approach, where the AIs would play this unbeatable strategy, and they would play against the world's best and beat them. Now that also means they they beat the world's worst.

Like, they would just beat anybody. But if they were up against a a weak opponent, they might not beat them as severely as a human expert might because the human expert would know how to adapt from the Game three optimal policy to be able to exploit these weak players. And so there's this kind of unanswered question of like, how do you make an exploitative poker AI?

And a lot of people had pursued this research direction. I had, like, dabbled in it a little bit during grad school, and I think fundamentally it just comes down to AI is not being as sample efficient as humans, you know, we discussed earlier. If a human's playing poker, they're able to get a really good sense of of the strengths and weaknesses of a player within a dozen hands.

It's, like, honestly really impressive. And back when we were working on AI for poker in, like, the, you know, mid twenty tens, you'd have to these AIs would have to play, like, 10,000 hands of poker to, like, get a good profile of, like, who this player is, like, how they're playing, where their weaknesses are. Now I think with more recent technology, that has come down, but still the sample efficiency has been a big challenge.

Now what's interesting is that after working on poker, I worked on diplomacy. I think we talked about this earlier. And diplomacy is this you know, it's a seven player negotiation game.

And when we started working on it, I took a very game theory approach to the problem. I I felt like, okay. We're it's kinda like poker.

You have to compute this game theory optimal policy, you just play this. You're gonna not lose an expectation. You're gonna win in practice.

But that actually doesn't work in diplomacy, and it doesn't work again, for it's a question of, like, how how much of a of a rabbit hole do you wanna go down on this, but, like, basically, when you're playing, like, the zero sum games like like poker, game theory optimal works really well. When you're playing a game like diplomacy where there's, like, you need to collaborate and compete and you need there's there's room for collaboration, then game theory optimal actually doesn't work that well. And you have to understand the players and adapt to them much better.

So this ends up being very similar to the problem in poker of, like, how do you adapt to your opponents? In poker, it's about adapting to their weaknesses and take advantage of that. In diplomacy, it's about adapting to their play styles.

It's kinda like if you're at a table and everybody's speaking French, you don't wanna just keep talking in English. You wanna adapt to them and speak in French as well. That's the realization that I have with diplomacy that we need to shift away from this game theory optimal paradigm towards modeling the other players, understanding who they are, and then responding accordingly.

And so in many ways, the techniques that we developed in diplomacy are exploitative. Like, they're not exploitative. They're they're really, you know, just adapting to the to the opponents to the other players at the table.

But I think the same techniques could be used in AI for poker to make exploitative poker AIs. If I didn't get, you know, AGI pilled by the incredible progress that we were seeing with language models and, like, shifting my whole research agenda to focusing on, like, general reasoning, probably what I would have worked on next was making these, like, exploitative poker AIs. It would be a really fun research direction to go down.

I think it's still there for anybody that wants to do it. And I think the key would be taking the techniques that we use in in diplomacy and applying them to things like poker. I think to me, the core piece is when you play online, you have a HUD, which tells you, you know, all these stats about the other player and, like, know, how much they participate, preflop, blah blah blah.

GTO. GTO Yep. Strategy, and they're not adapting to the other players at the table.

And, like, you can do various, like, kinda hacky things to get them to adapt, but, you know, they're not they're not very principled. They're not they don't work super well. Yep.

Okay. Any grad students listening?

a world model. I think it's pretty clear that as these models get bigger, they have a world model, and that world model becomes better with scale. So they are implicitly developing a world model, and I don't think it's something that you need to explicitly model.

of what which are the many types of entities you could be dealing with. You know, there was this, like, long debate in the multi agent AI community for a long time about and it's still going on, about whether you need to explicitly model other agents, other people, or if they can be implicitly modeled as part of the environment. For a long time, was like on the on the took the perspective of like, of course, you have to, like, explicitly model these other agents because they're they're behaving differently from the environment.

Like, they they take actions, they're unpredictable, you know, they they have agency. But I think I've actually shifted over time to thinking that, like, actually, these models become smart enough, they develop things like theory of mind. They develop an understanding that there are other agents that, like, can take actions and and have motives and all this stuff.

And these models just develop that implicitly with scale and and workable behavior broadly. Cool. So that's the perspective I take these days.

So, like, what I just said was an example of a heuristic that is not bitter lesson filled, and you just it just goes away. Yeah. You're it's really all come back to the bitter lesson.

r one zero, whatever that was, do you think this will hold for multi agents, like self play to improve better than humans? Yeah. So okay.

So this this is a great question, and I I think this is, like, worth expanding on. So I think a lot of people today see self play as, like, the next step and maybe the last step that we need for superintelligence, and I think if you're following you look at something like AlphaGo and AlphaZero, we seem to be following a very similar trend. Right?

Like the first step in AlphaGo was you do large scale pre training. In that case it was on human Go games. With LMs it's pre training on, you know, tons of like Internet data.

And that gets you a strong model, but it doesn't get you, you know, an extremely strong model. You know, it doesn't get you superhuman model. And then the next step in the AlphaGo paradigm is you do large scale test time compute or, like, large scale inference compute, and in that case with MCTS, and now we have, like, reasoning models that also do, like, this large scale inference compute.

And, again, that, like, boosts the capabilities a ton. Finally, with AlphaGo and AlphaZero, you have self play, where the model plays against itself, learns from those games, gets better and better and better, and just, like, goes from something that's, like, around human level performance to, like, way beyond human capability. It's like these Go policies now are so strong that it's just, like, incomprehensible.

Like, what they're doing is incomprehensible to humans. Same thing with chess. And we don't have that right now with language models.

And so it's, like, it's really tempting to look at that and say, oh, well, we just need these, AI models to now interact with each other and learn from each other, and then they're just gonna, like, get to super intelligence. The challenge and I kind of mentioned this, like, a little bit when I was talking about diplomacy. The challenge is that Go is this two player zero sum game, and two player zero sum games have this very nice property where when you do self play, you are converging to a minimax equilibrium.

And I I guess I should take a step back and say, like, in two player zero sum games, two player zero sum games are are chess, go even two player poker, all two player zero sum. What you typically want is what's called a minimax equilibrium. This is that that GTO policy, this policy that you play where you're guaranteeing that you're not going to lose to any opponent in expectation.

And I think in chess and Go, that's, like, pretty clearly what you want. Interestingly, in when you look at poker, it's not as obvious. In a two player zero sum version of poker, you could play the GTO minimax policy, and that guarantees that you won't lose to any opponent on earth.

But, again, I mentioned there's you're not gonna need to beat a weak player. You're not gonna make as much money off of them as you could if you instead played an exploitative policy. So there's this question of, like, what do you want?

Do you want to make as much money as possible, or do you want to guarantee that you're not gonna lose to any human alive? What all the bots have decided is, well, we're what what all the, like, AI developers in these games have decided is, like, well, we're gonna choose the minimax policy. And conveniently, that's exactly what self play converges to.

If you have these AIs play against each other, learn from their mistakes, they converge over time to this minimax policy guaranteed. But once you go outside of two players or some games, like in the case of diplomacy, that's actually not a useful policy anymore. You don't want to just, like, have this very defensive policy, and you're gonna end up in with really weird behavior if you start doing the same kind of self play in things like math.

So for example, what does it mean to do self play in math? You could fall into this trap of like, well, I just want one model to pose really difficult questions and the other model to solve those questions. You know, that's like a two player zero sum game.

The problem is that, like, well, you could just, like, pose really difficult questions that are not interesting. You know? You could just, like, get ask it to do, like, 30 digit multiplication.

It's a very difficult problem for the AI models. Is that really making progress in the dimension that we want? Like, not really.

So self play outside of these two player zero sum games becomes, like, a much more difficult nuanced question. So I think and and Tim Tim kind of, like, basically said something similar in his talk that there's a lot of challenges in really deciding what you're optimizing for when you start to talk about self play outside of two players or some games. My point is that, like, this is where the AlphaGo analogy breaks down.

And not necessarily breaks down, but, like, it's not gonna be as easy as self play was in AlphaGo.

What is the new objective function? Yeah. It's a good it's a good question.

Yeah. And I think that that's something that, you know, a lot of people are thinking about. Yeah.

new updates or in that sort of generative media space is autoregressive image gen. Is that interesting or surprising in any way that you wanna comment about? I don't work on image gen, so I I my ability to comment on this is kinda limited, but I will say, like, I I love it.

Like, I think it's super impressive. It's like one of those things where you work on these reasoning models and you think like, wow, we're going to be able to do all sorts of crazy stuff, like advanced science and solve agentic tasks and software engineering, and then there's this whole other dimension of progress where you're like, oh you're able to make images and videos now and it's so much fun, and that's getting a lot more of the attention to be honest, especially in the general public, and it's probably driving a lot more of the subscription plans for Chesh Bhutti, which is is great, but I think it's just kinda funny that like, yeah, we're also I promise, we're also working on superintelligence.

diffusion. Mhmm. Do we have both?

Does one win? The beauty of research is, like, you know, you gotta pursue different different directions, and it's it's not always gonna be clear, like, what is, you know, the processing path. Like and I think it's great that people are looking into different directions and trying different things.

I I think that there's a lot of value in that exploration, and I think we all benefit from seeing what works. Any potential in diffusion reasoning?

I did a master's in robotics years ago, and my takeaway from that experience first of I didn't actually work with robots that much. I was, like, technically in a robotics program. I play around some Lego robots my my first week at the program, but then honestly, I just, like, pretty quickly shifted just working on AI for poker and was kinda nominally in the robotics masters.

But my takeaway from, like, interacting with all these roboticists and seeing their research was that I did not wanna work on robots because the research cycle is so much slower and so much more painful when you're dealing with, like, physical hardware. Like, software goes so much more quickly, and I think that's why we're seeing so much progress with language models and, like, all these, like, virtual coworker kind of tasks, but haven't seen as much progress in robotics that, like, physical hardware just is much more painful to iterate on. On the question of humanoids, I don't have very strong opinions here because this isn't what I'm working on, but I think there's a lot of value in nonhumanoid robotics as well.

Think I drones are a perfect example where, like, there's clearly a lot of value in that. Is that a humanoid? No.

But in many ways, that's great. You know? Like, you don't want a humanoid for for that kind of technology.

in house would I'm terrible at product design. So I I am not the person to ask on this. I think there is a question of like, is it better to make humanoids because they're more familiar to us, or is it worse to make humanoids because they're more similar to us but not quite identical?

Like, I I don't know which one I would actually find creepier.

Again, I don't have very strong opinions in this field because, like, I don't work in I was, like, weekly in favor of humanoids, and I think what really persuaded me to be weekly in favor of, like, nonhumanoids was listening to the physical intelligence CEO and, like, some of his pitches about, like, why they're not pursuing why they're pursuing, like, nonhumanoid robotics. Okay. And conveniently, their office is actually like very close to here.

So if you wanted to They're speaking at the the conference I'm running. Okay. Perfect.

Yeah. Looking forward to that. I'd say like listen to his pitch and maybe he can convince you that nonhumanoid is the way to go.

It's it's really hard. I think that a lot of people have this perception that, like, academic research is irrelevant, and that's actually not the case. I think that we do we look at academic research.

I I think one of the challenges is, like, a lot of academic research shows promise in their papers, but then actually doesn't work at scale or even doesn't replicate. I think if we find interesting papers, like, we're we're gonna try to reproduce that in house and see if it, like, still holds up and then also does it scale well. But that is, like, a big source of inspiration for us.

Whatever hits archive, literally, you do the same as the rest of us, or do you have, like, a special process? Especially if I get recommendations. Like, we have an internal It's word of us.

Channel Yeah. Where people will post interesting papers. And like, I think that's a good source of like, okay, well, this person that is more familiar with this area thinks that this paper is interesting, therefore I should read it.

Yeah. And similarly, like, I'll keep track of things that are happening in, like, in my space that I think are interesting, and like, if I think it's really interesting, maybe I'll share it. For me, it's like WhatsApp and Signal group chats with researchers, and that's it.

Yeah. I think it is like I mean, a lot of people look at things like Twitter, and I think it's really unfortunate that we've reached this point where things need to get a lot of attention on social media for it to be paid attention to. That's what the grad students are trained.

They're taking classes to do this. I I do recommend to, like know, I've worked with grad students. I work with fewer now because we don't publish as much.

But when I was at fair publishing papers, like, I would tell the grad students I was working with that, like, you need to post it on Twitter. Yeah. And you need to and we go over, like, the Twitter thread about, like, how to present the work and everything, and there's a real art to it, and it does matter.

And it's kind of the sad truth.

Absolutely. And I I think one example is for benchmarks that you look at things like humanity's last exam, like you have these incredibly difficult problems, but then are still very easily gradable. And I think that actually limits the scope of what you can evaluate these models on if you if you stick to that paradigm.

It's very convenient because it's very easy to then score the models, but actually a lot of the things that we want to evaluate these models on are kind of like more fuzzy tasks that are not multiple choice questions. And making benchmarks for those kinds of things is so much harder, and probably also like a lot more expensive to evaluate. But I think that those are really valuable things to work on.

And that would fit the same moment, g p d 4.5 as like a high taste model in a way. There's kinda like all these like non measurable things about a model that are really good that maybe people are not.

Well, I think there are things that are measurable, but they're just much more difficult to measure.

It's very similar to pretraining where, like, you can push pretraining a lot further, it just becomes more expensive with each iteration. I think we're gonna see something similar with test time compute. We're like, okay we're going to get them thinking instead of three minutes they're for three hours and then three days and then three weeks.

two concerns. One is that it becomes much more expensive to get the models to think for that long or scale up test time compute. Like as you scale up test time compute, you're spending more on test time compute, which means that there's a limit to how much you could spend.

That's one potential ceiling. Now obviously well not obviously, but I should say that also becoming more efficient. These models are becoming more efficient in the way they're thinking is they're able to do more with the same amount of test time computing.

I think that's a very underappreciated point, that it's not just that we're getting these models to think for longer. In fact, if you look at o three, it's thinking for longer than o one preview for some questions, but it's not, like, a radical difference, but it's way better. Why?

Because it's just like becoming better at thinking. Anyway, yeah, these models, you're gonna scale up tests on compute, you can only scale it up so much. Like, that becomes a soft barrier in the same way that pretraining, it's actually more and more expensive to train better and better pretrained models or bigger pretrained models.

The second point is that as you have these models think for longer, you kinda get bottlenecked by walk clock time. Like if you wanna iterate on experiments, it is really easy to iterate on experiments when these models would respond instantly. It's actually much harder when they take three hours to respond.

And what happens when they have three weeks? It takes you at least three weeks to do those evaluations and to then iterate on that. And and a lot of this, you can paralyze experiments to some extent, but a lot of it you have to run the experiment, complete it, and then see the results in order to decide on the next set of experiments.

I think this is actually the strongest case for for long timelines that the models, because they just have to, like, do so much in serial time, we can only iterate so quickly. How would you overcome that wall? It's it's a challenge, and I think it I think it depends on the domain.

So drug discovery, think, is one domain where this could be a real bottleneck. I mean, you want to see if something, like, extends human life, it's gonna take you a long time to figure out if, like, this new drug that you developed, like, actually extends human life and doesn't have, like, terrible side effects along the way. Side note, do we not have perfect models of human chemistry and biology by now?

Well, so this this is, I think, the thing. And, again, I I wanna be cautious here because I'm not actually a biologist or a chemist. Like, I don't I know very little about about these fields.

Last time I took a biology class was tenth grade in high school. I don't think that there's a perfect simulator of human biology right now, and I think that that's something that could potentially help address this problem. Right?

That's like the number one thing that we should all work on. Well, that's one of the things that we're hoping that these racing models will help us with. Yeah.

It's it's such the all these definitions are so fuzzy.

I think mid training is between pre training and post training. It's like it's like it's it's not it's not post training. It's not pre training.

It's like adding more to the models, but like after pre training, like In adjusting ways. Yeah. Okay.

All right. Well, I was trying to get some clarity.

just an artifact that then spawns other models? And it's almost like the core pretraining model is never really exposed anymore. And it's the mid training, the new pretraining, and then there's the post training once you have the models branched out.

You never interact with an actual just like raw pretrained model. Like, if you're gonna interact with the model, it's gonna go through mid training and post training. So so you're seeing the final product.

Well you don't let us do it, but you know, used to. Well yeah, I mean I guess you you know, there's open source models where can just interact with the raw pre trained model. But for for OpenAI models, like they go through a mid training step, then they go through a post training step, and then and then they're released.

And they're a lot more useful. Like, frankly, if you interacted with the only pre trained model, it would be super difficult to work with and it would Yeah. It would seem kinda dumb.

Yeah. But it'd be it'd be in weird ways, you know, because there's a mode collapse when you when you post trade for it for, like, chat. Yeah.

And in some ways, you want that mode collapse. Like, you want you that collapse of, like Yes. To be useful.

Yeah. I I get it. Yeah.

We're interviewing Greg Brockman next.

talked to him a lot. Would you ask him? What would I ask Greg?

I mean I mean, I get to ask Greg all the time. What what should you ask Greg?

it's worth asking where this goes. You know? Like, what does the world actually look like in five years?

What does the world look like in ten years? What does that distribution of outcomes look like? And what could the world or individuals do to help steer things towards, like, the good outcomes instead of the negative outcomes?

Okay. Like an alignment question. I think people get very focused on what's going to happen in, like, one or two years.

And I think it's also worth spending some time thinking about, like, well, what happens in five or ten years? And what what does that world look like?

But he certainly has he certainly has thoughts. Yeah.

especially socially? What are games that I recommend to people? I've been playing a lot of this game called Blood on the Clock Tower lately.

What is it? It's kinda like Mafia or Werewolf. It's become very popular in San Francisco.

Oh that's the one who played in your house? Yeah. Okay, got it.

It's kind of funny because like I was talking to a couple people now that had told me that it used to be that poker was the way that VCs and tech founders and stuff would socialize with each other, and actually now it's shifting more towards blood on the clock tower, like that's the thing that people use to connect in the Bay Area. And I was actually told that a a startup held a recruiting event that was a blood on the clock tower game. Wow.

Yeah. So I guess it's like it's really catching on, but it's a fun game. And I guess you lose less money playing it than you do Right.

Playing poker. So it's like better for people that are not very good at these I I think it's kind of like a weird recruiting event, but it's certainly a fun game. What qualities make a winner here that is interesting to hire for?

That's the thing. It's like, okay, I guess you get Ability to lie. Picking up on deception, is that the best employee?

I don't know.

that is? Like, how does the difficulty of this problem scale? I love that you asked that because I have this, like, huge store of knowledge on AI for imperfect information games.

Like, this is my my area of research for so long, and I know all these things, but I don't get to talk about it very often. We've made superhuman Poker AIs for No Limit Texas Hold'em. One of the interesting things about that is that, like, the amount of hidden information is actually pretty limited because you have two hidden cards when you're playing Texas Hold'em.

And so the number of possible states that you could be in is 1,326 when you're playing heads up at least. And, you know, that's multiplied by the number of other players that there are at the table, but it's still, like, not a a massive number. And so the way these AI models work is they enumerate all the different states that you could be in.

So if you're playing, like, six handed poker, there's five other players, five times 1,326, that's the number of states that you'd in, and then you assign a probability to each one. And then you feed those probabilities into your neural net, and you get actions back for each of those states. The problem is that as you scale the number of hidden possibilities, like the number of state of of possible states you could be in, that approach breaks down.

And there's still this very interesting unanswered question of what do you do when the number of hidden states becomes extremely large. Mhmm. You know?

So if you go to Omaha poker where you have four hidden cards, there are things you could do that's kind of like that are kind of heuristic that you could do to reduce the number of states, but actually, it's still a very difficult question. And then if you go to a game like Stratego where you have 40 pieces, so there's, like, close to 40 factorial different states you could be in, then all these, like, existing approaches that we used for poker kinda break down, and you do need different approaches. And there's lot of active research going on about, like, how do you how do you cope with that?

So for something like Magic the Gathering, the techniques that we used in poker would not out of the box work. And it's still an interesting research question of, like, what do you do? Now I should say this becomes a problem when you're doing the kinds of search techniques that we used in poker.

If you're just doing model free URL, it's not a problem. And my guess is that if somebody put in the effort, they could probably make a superhuman bot for Badger the Gathering now. Yeah.

There's still some unanswered research questions in that space. Now are they the most important unanswered research questions? Like Right.

I'm inclined to say no. I think there's like problem is that, like, the techniques that we used in poker to do this kind of search stuff were pretty limited. And, like, if you expand if you expand those techniques, maybe you get them to work on things like Stratego and Magic the Gathering, but they're still gonna be limited.

They're not gonna get you, superhuman encode forces with language models. So I think it's more valuable to just focus on the very general reasoning techniques. And one day, as we improve those, I think we'll have a model that just out of the box, one day plays Magic: The Gathering at a superhuman level.

And I think that's the more important and more impressive research direction.