Andrej Karpathy — AGI is still a decade away

Andre, why do you say that this will be the decade of agents and not the year of agents? Mhmm. Well, first of all, thank you for having me here.

I'm excited to be here. So the quote that you've just mentioned, it's the decade of agents. That's actually a reaction to an existing preexisting quote, I should say, where I think a lot of some of the labs I'm not actually sure who said this, but they were alluding to this being the year of agents Mhmm.

With respect to LLMs and how they were gonna evolve. And I think I was triggered by that because I feel like there's some over predictions going on in the industry. And in my mind, this is really a lot more accurately described as the Decade of Agents.

Yep. And we have some very early agents that are actually, like, extremely impressive and that I use daily, you know, Claude and Codex and so on. But I still feel like there's so much work to be done.

And so I think my, like, my reaction is, like, we'll be working with these things for a decade. They're gonna get better, and it's gonna be wonderful. But I think I was just reacting to the timelines, I suppose, of the of the implication.

Well, actually make it work. So in my mind I mean, when you're talking about an agent, I guess or what the labs have in mind and what maybe I have in mind as well, is it's you should think of it almost like an employee or like an intern that you would hire to work with you. So for example, you work with some employees here.

Yeah. When would you prefer to have an agent like Cloud or Codex do that work? Like, currently, of course, can't.

What would it take for them to be able to do that? Why don't you do it today? Yeah.

And the reason you don't do it today is because they just don't work. So Right. Like, they don't have enough intelligence.

They're not multimodal enough. They can't do computer use and all this kind of stuff. And they don't do a lot of the things that you've alluded to earlier.

You know, they don't have continual learning. You can't just tell them something, and they'll remember it. Yeah.

And they're just cognitively lacking, and it's just not working. And I just think that it will take about a decade to work through all of those issues. Interesting.

Yeah. I guess this is where you get into, like, a bit of a I guess, my own intuition a little bit, and also just kind of doing a bit of an extrapolation of with respect to my own experience in the field. Right?

So I guess I've been in AI for almost two decades. I mean, it's gonna be maybe fifteen years or so. Not that long.

You had Richard Sutton here who was all around, of course, for much longer. But I do have about fifteen years of experience of people making predictions of seeing how they actually turned out. And also, was in the industry for a while, and I was in research, and I worked in the industry for a while.

So I guess I kind of have just a general intuition that I have left from that. And I feel like the problems are tractable. They're surmountable.

Yeah. But they're still difficult.

breakthrough moments. Right. What were the ways in which their feelings were either overly pessimistic or overly optimistic?

Yeah. Maybe should we just go through each of them one by one? Yeah.

Mean, that's a giant question because, of course, you're talking about fifteen years of stuff that happened. I mean, AI is actually, like, so wonderful because there have been a number of, I would say, seismic shifts Yeah. That were, like, the entire field has sort of, like, suddenly looked a different way.

Yeah. Right? And I guess I've maybe lived through two or three of those.

Yeah. And I still think there will continue to be some, because they come with some kind of like almost surprising irregularity. Well, when my career began, of course, like, when I started to work on deep learning, when I became interested in deep learning, this was just kind of like by chance of being right next to Jeff Hinton at University of Toronto.

And Jeff Hinton, of course, is kind of like the godfather figure of AI. And he was training all these neural networks, I thought it was incredible and interesting. But this was not, like, the main thing that everyone in AI was doing by far.

No. This was a niche little subject on the side. That's kind of maybe like the first, like, dramatic sort of seismic shift that came with the AlexNet and so on.

I would say, like, AlexNet sort of reoriented everyone, and everyone started to train neural networks. But it was still, like, very, like, per task, per specific task. Maybe I have an image classifier, or I have a neural machine translator, or something like that.

And people became very slowly actually interested in basically kind of agents, I would say. And people started to think, okay. Well, maybe we have a check mark next to the visual cortex or something like that.

But what about the other parts of the brain? How can we get an actual, like, full agent or a full entity that can actually interact in the world? And I would say the Atari sort of deep reinforcement learning shift in 2013 or so Yeah.

Was part of that early effort of agents, in my mind. Because it was an attempt to try to get agents that not just perceive the world, but also take actions and interact and get rewards from environments. And at the time, this was Atari Games.

Right? And I kinda feel like that was a misstep, actually. And it was a misstep that actually even the early OpenAI that I was a part of, of course, kind of adopted.

Because at that time, the zeitgeist was reinforcement learning environments, games, game playing, beat games, get lots of different types of games, and OpenAI was doing a lot of that. So that was maybe like another, prominent part of, I would say, AI, where maybe for two or three or four years, everyone was doing reinforcement learning on games. And basically, that was a little bit of a misstep.

And what I was trying to do at OpenAI actually is like, I was always a little bit suspicious of games as being like this thing that would actually lead to AGI, because in my mind, you want something like an accountant, or like something that's actually interacting with the real world. And I just didn't see how games kind of like add up to it. And so my project at OpenAI, for example, was within the scope of the universe project, on an on an agent that was using keyboard and mouse to operate web pages.

Mhmm. And I really wanted to have something that, like, interact with, you know, the actual digital world that can do knowledge work. And it just so turns out that this was extremely early, way too early.

Mhmm. So early that we shouldn't have been working on that. You know?

Because if you're just stumbling your way around and keyboard mashing and mouse clicking and trying to get rewards in these environments, your reward is too sparse, and you just won't learn, and you're gonna burn a forest computing. Yeah. And you're never actually gonna get something off the ground.

And so what you're missing is this power of representation in the neural network. Yeah. And so, for example, today, people are training those computer using agents, but they're doing it on top of a large language model.

And so you actually have to get the language model first. You have to get the representations first, and you have to do that by all the pre training and all the LLM stuff. So I kinda feel like maybe loosely speaking, it was like people keep maybe trying to get the full thing too early a few times, where people, like, really try to go after agents too early, I would say.

And that was Atari and Universe, and even my own experience. And you actually have to do some things first before you sort of get to those agents. And maybe now the agents are a lot more competent, but maybe we're still missing sort of some parts of that stack.

But I would say maybe those are like the three, like, major buckets of what people were doing.

I mean, so so Sutton was on your podcast, I saw the podcast. And I had a write up about that podcast almost that gets into a little bit of how I see things. And I I kinda feel like I'm very careful to make analogies to animals because they came about by a very different optimization process.

Animals are evolved, and they actually come with a huge amount of hardware that's built in. And when, for example, my example in the post was the zebra. The zebra gets born, and a few minutes later, it's running around and following its mother.

That's an extremely complicated thing to do. Yeah. That's not reinforcement learning.

That's something that's baked in. And evolution obviously has some way of encoding the weights of our neural nets in ATCGs. And I have no idea how that works, but it apparently works.

So I kinda feel like brains just were came from a very different process. And I I'm very hesitant to take inspiration from it because we're not actually running that process. So in my post, I kind of said, we're not actually building animals.

We're building ghosts Yeah. Or spirits, or whatever people wanna call it. Because we're not we're not doing training by evolution.

We're doing training by basically imitation of humans Yeah. And the data that they've put on the Internet. And so you end up with these, like, sort of ethereal spirit entities because they're fully digital, and they're kinda like mimicking humans.

And it's a different kind of intelligence. Like, you imagine a space of intelligences, we're starting off at a different point almost. We're not we're not really building animals.

But I think it's also possible to make them a bit more animal like over time. Mhmm. And I think we should be doing that.

And so I kinda feel like sorry. Just, I guess one more point is, I do feel like Sutton basically has a very like, his framework is, like, we wanna build animals. Yeah.

And I actually think that would be wonderful. If we can get that to work, that would be amazing. If there was a single algorithm that you can just run on the Internet and it learns everything, that would be incredible.

I almost suspect that I'm not actually sure that it exists. And that's certainly actually not what animals do. Because animals have this outer loop of evolution.

Right. And a lot of what looks like learning is actually a lot more maturation of the brain. And I think there's actually actually very little reinforcement learning for animals.

And I think a lot of the reinforcement learning is actually more like motor tasks. It's not intelligence tasks. So I actually kind of think humans don't actually really use RL, roughly speaking is what I would say.

Can you repeat the last sentence? A lot of that intelligence is not motor tasks. That's why?

A lot of the reinforcement learning in my perspective would be things that are a lot more like motor like, like simple kind of like tasks, throwing a hoop, something But like I don't think that humans use reinforcement learning for a lot of intelligence tasks like problem solving and so on. Interesting. That doesn't mean we don't have we we shouldn't do that for research, but I just feel like that's what animals do or don't.

I'm gonna take a second to digest that because there's a lot of different ideas.

to RL, to your point. Is that compatible with the thing you were saying, would you disagree with that? I think so.

I would agree with you that there's some miraculous compression going on. Yeah. Because obviously, the weights of the neural net are not stored in ATCGs.

Yeah. There's some kind of a dramatic compression, and there's some kind of, like, learning algorithms encoded that that take over and do some of the learning online. So I can definitely agree with you on that.

Basically, I would say, I'm a lot more kind of, like, practically minded. I don't come at it from the perspective of, like, let's build animals. I come from it from perspective of, like, let's build useful things.

So I have a hard hat on. And I'm just observing that, look, we're not gonna do evolution because I don't know how to do that. But it does turn out we can build these ghost spirit like entities by imitating Internet documents.

This works. And it's actually kind of like it's a way to bring you up to something that has a lot of sort of built in knowledge and intelligence in some way, similar to maybe what evolution has done. So that's why I kind of call pretraining this kind of, like, crappy evolution.

It's like the practically possible version with our technology and what we have available to us to get to a starting point where we can actually do things like reinforcement learning and so on.

the thing that pre training is doing so you're basically getting the next token predictor on over the Internet, and you're training that into a neural net. It's doing two things actually that are kind of like unrelated. Number one, it's picking up all this knowledge, as I call it.

Number two, it's actually becoming intelligent. By observing the algorithmic patterns in the Internet, it actually kind of like boots up all these like little circuits and algorithms inside the neural net to do things like in context learning Yeah. And all this kind of stuff.

And actually, you don't actually need or want the knowledge. I actually think that's probably holding back to neural networks overall, because it's actually like getting them to rely on the knowledge a little too much sometimes. For example, I I kinda feel like agents, one thing they're not very good at is going off the data manifold of what exists on the Internet.

because I mean, it's not doing explicit gradient descent. But I I still think that so in context learning, basically, it's it's pattern completion within a token window. Right?

And it just turns out that there's huge amount of patterns on the Internet. And so you're right. The model kind of like learns to complete the pattern.

Yeah. And that's inside the weights. The weights of the neural network are trying to discover patterns and complete the pattern.

And there's some kind of an adaptation that happens inside the neural network. Right? Which is kind of magical and just falls out from Yeah.

Internet, just because there's a lot of patterns. I will say that there have been some papers that I thought were interesting that actually look at the mechanisms behind in context learning. And I do think it's possible that in context learning actually runs a small gradient descent loop internally in the layers of the neural network.

And so I recall one paper in particular where they were doing linear regression actually using in context learning. So basically, your inputs into the neural network are x y pairs Mhmm. X y x y x y that happen to be on the line.

Yeah. And then you do x, and you expect the y. Yeah.

And the neural network, when you train it in this way, actually does do does do linear regression. And normally, when you would run linear regression, you have a small gradient descent optimizer that basically looks at x y, looks at an error, calculates the gradient of the weights, and does the update a few times. It just turns out that when they looked at the weights of that in context learning algorithm, they actually found some analogies to to gradient descent mechanics.

In fact, I think even the paper went was stronger because they actually hard coded the weights of a neural network to do gradient descent through attention and all the all the internals of of the neural network. So I guess that's just my only pushback is that who knows how in context learning works, but I actually think that it's probably doing a little bit of some kind of funky gradient descent internally, and that I think that that's that's possible. So I guess I I was only pushing back on, you're saying it's not doing in context learning.

Who knows what it's doing? But it's probably maybe doing something similar to it, but we don't know.

assimilated by the model. Yeah. I wonder if that's relevant at all.

I think I kind of agree. I mean, the way I usually put this is that anything that happens during the training of the neural network, the knowledge is only kind of like a hazy recollection of what happened in train in the training time. That's because the compression is dramatic.

You've you're taking 15,000,000,000,000 tokens, and you're compressing it to just your final neural network of a few billion parameters. So obviously, it's a massive amount of compression going on. So I kind of refer to it as like a hazy recollection of the Internet documents.

Whereas anything that happens in the context window of the neural network, you're plugging all the tokens, and it's building up all this KV cache representation, is very directly accessible to the neural net. So I compare the KV cache and the stuff that happens at test time to like more like a working memory. Like, all the stuff that's in the in the in the context window is very directly accessible to the neural net.

So there's always like these almost surprising analogies between LLMs and humans. And I find them kind of surprising because we're not trying to build a human brain, of course. Just directly, we're just finding that this works, and we're doing it.

But I do think that anything that's in the weights, it's kind of like a hazy recollection of what you read a year ago. Anything that you give it as a context at test time is directly in the working memory. And I think that's a very powerful analogy to think through things.

So when you, for example, go to an LLM and you ask it about some book and what happened in it, like a book or something like that Yeah. The LLM will often give you some stuff, which is roughly correct. But if you give it the full chapter and ask it questions, you're gonna get much better results because it's now loaded in the working memory of the model.

So I basically agree with your very long way of saying that I kind of agree, and that's why.

I almost feel like just just a lot of it. So maybe one way to think about it, I don't know if this is the the best way, but I almost kinda feel like, again, making these analogies imperfect as they are. We've stumbled by with the transformer neural network, which I think extremely powerful, very general.

You can train transformers on audio, or video, or text, or whatever you want, and it just learns patterns, and they're very powerful, and it works really well. That to me almost indicates that this is kind of like some piece of cortical tissue. It's something like that.

Because the cortex is famously very plastic as well. You can rewire, you know, parts of brains. And there was a slightly gruesome experiments with rewiring, like, visual cortex to the auditory cortex, and this animal, like, fine, etcetera.

So I think that this is kind of like a cortical tissue. I think when we're doing reasoning and planning inside the neural networks, so basically doing our reasoning traces for thinking models, that's kind of like the prefrontal cortex. And then I think we maybe those are like little check marks, but I still think there's many brain parts and nuclei that are not explored.

So maybe, for example, there's a basal ganglia doing a bit of reinforcement learning when we fine tune the models on reinforcement learning. But, you know, know, whereas like the hippocampus, not obvious what that would be. Some parts are probably not important.

Maybe the cerebellum is not important to commission its thoughts, so maybe we can skip some of it. But I still think there's, for example, the amygdala, all the emotions and instincts. And there's probably like a bunch of other nuclei in the brain that are very ancient that I don't think we've, like, really replicated.

I don't actually know that we should be pursuing, you know, the building of an analog of human brain. I'm, again, an engineer mostly at heart. But I still feel like maybe another way to answer the question is, you're not gonna hire this thing as an intern, and it's missing a lot of it's because it comes with a lot of these cognitive deficits Yeah.

That we all intuitively feel when we talk to the models. And so it's just, like, not fully there yet. You can look at it as like not all the brain parts are checked off yet.

again, making maybe making some analogies to humans just because I think it's roughly concrete and kind of interesting to think through. I feel like when I'm awake, I'm building up a context window of stuff that's happening during the day. But I feel like when I go to sleep, something magical happens where I don't actually think that that context window stays around.

I think there's some process of distillation into weights of my brain. Yeah. And this happens during sleep and all this kind of stuff.

We don't have an equivalent of that in a large language models. And that's to me more adjacent to when you talk about continual learning and so on, as absent. Yeah.

These models don't really have this distillation phase of taking what happened, analyzing it, obsessively thinking through it, basically doing some kind of a synthetic data generation process, and distilling it back back into the weights, and maybe having a specific neural net per person. Maybe it's a LoRa, it's not a full yeah. It's not a full weight neural network that's it's just small some of the small sparse subset of the weights are changed.

But basically, we do want to create ways of creating these individuals that have very long context. It's not only remaining in the context window, because the context windows grow very, very long. Like, maybe we have some very elaborate sparse attention over it.

But I still think that humans obviously have some process for distilling some of that knowledge into the weights. We're missing it. And I do also think that humans have some kind of a very elaborate sparse attention scheme, which I think we're starting to see some early hints of.

So DeepSeek v 3.2 just came out, and I saw that they have like a sparse attention as an example. And this is one way to have very, very long context windows.

So I almost feel like we are redoing a lot of the cognitive tricks that Evolution came up with through a very different process. But we're, I think, gonna converge on a similar architecture cognitively. Interesting.

we had convolutional neural networks primarily. Residual networks just came out. So remarkably similar, I guess, but quite a bit different still.

I mean, transformer was not around. You know, all the all these sort of, like, more modern tweaks on a transformer were not around. So maybe some of the things that we can bet on, I think, in ten years by translational sort of equivariance, is we're still training giant neural networks with forward backward pass and update through gradient descent.

But maybe it looks a little bit different. And it's just everything is much bigger. Actually, recently, I also went back all the way to 1989, which was kind of a fun exercise for me a few years ago.

Because I was reproducing Jan Lakun's 1989 convolutional network, which was the first neural network I'm aware of trained via gradient descent, like modern neural network trained gradient descent on digit recognition. And I was just interested in, okay, how can I modernize this? How much of this is algorithms?

How much of this is data? How much of this progress is compute and systems? And I was able to very quickly, like, half the learning rate, just knowing my trap time travel by thirty three years.

So if I time travel by algorithms to thirty three years, I could adjust what the unlucky did in 1989, and I could basically half the learning half the error. But to get further gains, I had to add a lot more data. I had to, 10x the training set.

And then I had to actually add more computational optimizations. Had to basically train for much longer with dropout and other regularization techniques. And so it's almost like all these things have to improve simultaneously.

So, you know, we're probably gonna have a lot more data. We're probably gonna have a lot better hardware. We're probably gonna have a lot better kernels and software.

We're probably gonna have better algorithms. And all of those it's almost like no one of them is winning too much. All of them are surprisingly equal.

And this has kind of been the trend for a while. So I guess to answer maybe your question, I expect differences algorithmically to what's happening today. But I do also expect that some of the things that have stuck stuck around for a very long time will probably still be there.

It's probably still giant neural network trained with gradient descent. That would be my guess.

takeaways from the experience? Building a NanoChat? Yeah.

So NanoChat is a kind of a repository I released. Was it yesterday or day before? I can't remember.

It's just trying to be a it's trying to be the simplest complete repository that covers the whole pipeline end to end of building a ChaCepti clone. And so, you know, you have all of the steps, not just any individual step, which is a bunch I worked on all the individual steps sort of in the past and really small pieces of code that could have show you how that's done in algorithmic sense in like simple code. But this kind of handles all the entire pipeline.

I I think in terms of learning, it's not it's not so much I don't know that I actually found something that I learned from from it necessarily. I kind of already had in my mind as like how you build it, and this is just a process of mechanically building it and making it clean enough and so that people can actually learn from it and that they find it useful. Yeah.

What is the best way for somebody to learn from it? Is it just like delete all the code and try to reimplement from scratch, try to add modifications to it? Yeah.

I think that's a that's a great question. I would probably say so basically, it's about one a thousand lines of code that takes you through the entire pipeline. I would probably put it on the right monitor, like, if you have two monitors, you put it on the on the right, and you wanna build it from scratch, you build it from start, you're not allowed to copy paste.

You're allowed to reference. You're not allowed to copy paste. Maybe that's how I would do it.

But I also think the repository by itself, it is like a pretty large beast. I mean, it's, you know, it's a it's when you write this code, you don't go from top to bottom. You go from chunks, and you grow the chunks.

And that information is absent. Like, you wouldn't know where to start. And so I think it's not just a final repository that's needed, it's like the building of the repository, which is a complicated chunk growing process.

Right. So that part is not there yet. I would love to actually, like, add that probably later this week or something, in some way.

Like, either it's a it's probably a video or something like that. But but maybe, roughly speaking, that's what I would try to do, build the stuff yourself, but don't allow yourself copy paste. Yeah.

I do think that there's two types of knowledge almost. Like, there's the high level surface knowledge. But the thing is that when you actually build something from scratch, you're forced to come to terms with what you don't actually understand, and you don't know that you don't understand it.

Interesting. And it always leads to a deeper understanding. And it's like just the only way to to build this.

Like, if I can't build it, I don't understand it. Is that a fine line quote, I believe, or something along those lines? A 100%, I've always believed this very strongly.

Because there's all these micro things that are just not properly arranged, you don't really have the knowledge. You just think you have the knowledge. So don't write blog posts, don't do slides, don't do any of that.

Build a code, arrange it, get it to work. It's the only way to go, otherwise, you're missing knowledge.

So the repository, I guess I built it over a period of a bit more than a month. And I would say there's, like, three major classes of how people interact with code right now. Some people completely reject all of LLMs, and they are just writing by scratch.

I think this is probably not the the right thing to do anymore. The intermediate part, which is where I am, is you still write a lot of things from scratch, but you use the auto complete. That's basically available now from these models.

So when you start writing out a little bit piece of it, it will it will auto complete for you, and you can just tap through, and most of the time it's correct. Sometimes it's not, and you edit it. But you're still very much the sort of architect of what you're writing.

And then there's the, you know, vibe coding. You know, hi, please implement this or that, you know, enter, and then let the model do it. And that's the agents.

I do feel like the agents work in very specific settings, and I would use them in specific settings. But again, these are all tools available to you, and you have to learn what they what they're good at, and what they're not good at, and when to use them. So the agents are actually pretty good, for example, if you're doing boilerplate stuff.

Yeah. Boilerplate code that's like just copy paste stuff, they're very good at that. They're very good at stuff that occurs very often in the Internet, because there's lots of examples of it in the training sets of these models.

So so there's like features of things that where the models will do very well. I would say Nanochat is not an example of this, because it's a fairly unique repository. There's not that much code, I think, in the way that I've structured it.

And and it's not boilerplate code. It's like actually, like, intellectually intense code almost. And everything has to be very precisely arranged.

And the models are always trying to they kept trying to I mean, they have so many cognitive deficits. Right? So one example, they keep trying to they keep misunderstanding the code because they they have too much memory from all the typical ways of doing things on the Internet that I just wasn't adopting.

So the models, for example I mean, I don't know if I wanna get into the full details, but they keep they keep they keep thinking I'm writing normal code, and I'm not. Maybe one example. That maybe a great So the way to synchronize.

So we have eight GPUs that are all doing forward records. The way to synchronize gradients between them is to use a distributed data parallel container of PyTorch, which automatically does all the as you're doing the backward, it will start communicating and synchronizing gradients. I didn't use DDP because I didn't wanna use it because it's not necessary.

So I threw it out. And I basically wrote my own synchronization routine that's inside the step of the optimizer. And so the models were trying to get me to use the DDP container Yeah.

And they were very concerned about, okay, this gets way too technical. But I wasn't using that container because I don't need it, and I have a custom implementation of something like it. And they just couldn't internalize that you had your own Yeah.

They couldn't. They they couldn't get past that. And then they kept trying to, like, mess up the style.

Like, they're way too over defensive. They make all these try catch statements. Yeah.

They keep trying to make a production code base. And I have a bunch of assumptions in my code, and it's okay. And and it's just like, I don't need all this extra stuff in there.

And so I just kinda feel like they're bloating the code base. They're bloating the complexity. They keep misunderstanding.

They're using deprecated APIs a bunch of times. So it's total mess. And it's just it's just not net useful.

I can go in, I can clean it up, but it's not net useful. I also feel like it's kind of annoying to have to, like, type out what I want in English, because it's just too much typing. Like, if I just navigate to the part of the code that I want, and I go where I where I know the code has to appear, and I start typing out the first three letters, autocomplete gets it and just gets you the code.

And so I think it's this is a very high information bandwidth to specify what you want. Is if you point to the code where you want it, and you type out the first few pieces Yeah. And the model will complete it.

So I guess what I mean is, I think these models are good in certain parts of the stack. I actually use the models a little bit in there are two examples where I actually use the models that I think are illustrative. One was when I generate the report, and that's actually more boilerplate y.

So actually, I coded part partially some of that stuff, that was fine. Because it's not like mission critical stuff, and it works fine. And then the other part is when I was rewriting the tokenizer in Rust.

I'm actually not as good at Rust because I'm fairly new to Rust. So I was doing there's a bit of vibe coding going on in when I was writing some of the Rust code. But I had Python implementation that I fully understand, and I'm just making sure I'm making more efficient version of it, and I have tests.

So I feel safer doing that stuff. And so, basically, they lower or, like, they increase accessibility to languages or paradigms that you might not as be not be as familiar with. So I think they're very helpful there as well.

Yeah. Because there's a ton of Rust code out there. The models are actually pretty good at it.

is likely to happen anytime soon. I think that's a good way of putting it. And I think you're getting at some of my, like, why my timelines are a bit longer.

You're right. I think, yeah, they're not very good at code that has never been written before. Maybe it's like one way to put it, which is like what we're trying to achieve when we're building these models.

It's it's tough. I think they kind of know they kind of know, but they don't fully know. And they don't know how to fully integrate it into the repo and your style and your code and your place and some of the custom things that you're doing and And how it fits with all the assumptions of the repository and all this kind of stuff.

So I think they do have some knowledge, but they haven't gotten to the place where they can actually integrate it, make sense of it, and so on. I do think that a lot of this stuff, by the way, continues to improve. So I currently, probably state of the art model that I go to is the GPT five Pro.

Mhmm. And that's a very, very powerful model. So if I actually have twenty minutes, I will copy paste my entire repo, and I go to GPT five Pro, the Oracle, for, like, some questions, and Yeah.

Often it's not too bad, and surprisingly good compared to what existed a year ago. Yeah. But I do think that overall, the models are are they're not there.

And I kinda feel like the industry, it's it's it's over it's it's making too big of a jump. Mhmm. And it's trying to pretend like this is amazing, and it's not.

It's slop. And I think they're not coming to terms with it, and maybe they're trying to fundraise or something like that. I'm not sure what's going on, but it's we're at this intermediate stage.

The models are amazing. They still need a lot of work. For now, autocomplete is my sweet spot.

Mhmm. But sometimes for some types of code, I will go to a null m agent. Yeah.

Yeah. Actually, this this is also here's another reason why this is very interesting.

but have not led to an explosion. So that's like one that sounds very much like autocomplete tab. Mhmm.

Yep. And this other category is just like automation of the programmer. Yeah.

And it's it's interesting you're seeing more in the category of the historical analogies of like, you know, better compilers or something. And maybe you guys discuss that one other kind of thought of that is like, I do feel like I have a hard time differentiating where AI begins and stops. Mhmm.

Because I do see AI as fundamentally an extension of computing in some in some pretty fundamental way. And I I feel like I see a continuum of this kind of like recursive self improvement or or like of speeding up programmers all the way from the beginning. Like, even, like, I would say, like, code editors.

Yeah. Syntax highlighting. Yeah.

Syntax or, like, checking even of the of the types, like data type checking. All these kinds of tools that we've built for each for each other, even search engines. Like, why aren't search engines part of AI?

Like, I don't know, like, ranking is kind of AI. Right? At some point, Google was like even early on, were thinking of themselves as an AI company doing Google search engine, which I think is totally fair.

And so I kind of see it as a lot more of a continuum than I think other people do, and I don't it's hard for me to draw the line. And I kind of feel like, okay, we're now getting a much better autocomplete. And now we're also getting some agents, which are kind of like these loopy things, but they kind of go off rails sometimes.

And what's going on is that the human is progressively doing a bit less and less of the low level stuff. For example, we're not writing the assembly code because we have compilers. Yeah.

Right? Like, compilers will take my high level language in C and write the assembly code. Yeah.

So we're abstracting ourselves very, very slowly. And there's this, what I call, autonomy slider of, like, more and more stuff is automated, of the stuff that can be automated at any point in time, and we're doing a bit less and less in raising ourselves in the layer of abstraction over the automation.

and how does that compare to what we're doing with other ones right now? Yeah. Maybe the way would put it is humans don't use reinforcement learning is maybe what I as as I've said it all.

I I think they do something different, which is, yeah, you experience so reinforcement learning is a lot worse than I think the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before is much worse.

Because previously, we're just imitating people, so it has all these issues. So in reinforcement learning, say you're working with you're solving a math problem. This is very simple.

You're given a math problem, and you're trying to find a solution. Now, in reinforcement learning, you will try lots of things in parallel first. So you're given a problem, you try hundreds of different attempts.

And these attempts can be complex. Right? They can be like, oh, let me try this, let me try that, this didn't work, that didn't work, etcetera.

And then maybe you get an answer. And now you check the back of the book, and you see, okay, the correct answer is this. And then you can see that, okay, this one, this one, and that one got the correct answer, but these other 97 of them didn't.

So literally, what reinforcement learning does is it goes to the ones that worked really well, and every single thing you did along the way, every single token gets up weighted of, like, do more of this. The problem with that is I mean, people will say that your estimator has high variance, but what I mean, it's just noisy. It's noisy.

So, basically, it kind of almost assumes that every single little piece of the solution that you made that ride the dryness, it was correct thing to do, which is not true. Like, may have gone down the wrong alleys until you arrived at the right solution. Every single one of those incorrect things you did, as long as you got to the correct solution, will be upweighted as do more of this.

It's terrible. Yeah. It's noise.

You've done all this work, only to find a single at the end, you get a single number of, oh, you did correct. And and based on that, you weigh that entire trajectory as, like, upweight or downweight. And so you're the way I like to put it is you're sucking supervision through a straw, because you've done all this work that could be a minute of rollout, and you're you're, like, sucking the bits of supervision of the final reward signal through a straw, and you're, like, putting it you're, like you're basically, like yeah.

You're broadcasting that across the entire trajectory and using that to upway or downway that trajectory. It's stupid and crazy. A human would never do this.

Number one, a human would never do hundreds of rollouts. Right? Number two, when a person sort of finds a solution, they will have a pretty complicated process of review of, okay.

I think these parts that I did well, these parts I did not do that well. I should probably do this or that, and they think through things. There's nothing in current LLMs that does this.

There's no equivalent of it. But I do see papers popping out that are trying to do this, because it's obvious to everyone in the field. Yeah.

So I kind of see as like the first imitation learning actually, by the way, was extremely surprising and miraculous and amazing that we can fine tune by imitation in humans. And that was incredible. Because in the beginning, all we had was base models.

Base models are auto complete. And it wasn't obvious to me at the time, and I had to learn this, and the paper that blew my mind was InstructGPT. Because it pointed out that, hey, can treat the pretrained model, which is autocomplete.

And if you just fine tune it on text that looks like conversations, the model will very rapidly adapt to become very conversational. And it keeps all the knowledge from pretrained. And this blew my mind, because I didn't understand that this just like stylistically can adjust so quickly and become an assistant to a user.

Through through just a few loops of fine tuning on that kind of data. It was very miraculous to me that that that worked. So incredible.

And that was like two years, three years of work. And now came RL. And RL allows you to do a bit better than just imitation learning.

Right? Because you you can't have these reward functions, and you can hill climb on the reward functions. And so some problems have just correct answers, you can hill climb on that without getting expert trajectories to imitate.

So that's amazing. And the model can also discover solutions that the human might never come up with. Mhmm.

So this is incredible. And yet, it's still stupid. So I think we need we need more.

And so I saw a paper from Google yesterday that tried to have this reflect and review idea in mind. What was the memory bank paper or something? I don't know.

I've actually seen a few papers along these lines. So I expect there to be some kind of a major update to how we do algorithms for LLMs coming in that realm. Then I think we need three or four or five more.

as an alternative been a successful way to make models more capable? What what has been preventing us from using this alternative paradigm? So process based supervision just refers to the fact that we're not gonna have reward function only at the very end of after you've made ten minutes of work, I'm not gonna tell you you did well or not well.

I'm gonna tell you at every single step of the way how well you're doing. And this is basically the reason we don't have that, it's it's not trick it's tricky how you do that properly. Mhmm.

Because you have partial solutions, and you don't know how to assign credit. So when you get the right answer, it's just an equality match to the answer. Very simple to implement.

Yeah. If you're doing basically process supervision, how do you assign, in an automatable way, partial credit assignment? It's not obvious how you do it.

Lots of labs, I think, are trying to do it with these LLM judges. So basically, you get LLMs to try to do it. So you prompt an LLM, hey.

Look at a partial solution of a student. How well do you think they're doing if the answer is this? And they try to tune the prompt.

The reason that I think this is kind of tricky is quite subtle. And it's the fact that anytime you use an LLM to assign a reward, those LLMs are giant things with billions of parameters, and they're gameable. And if you're reinforcement learning with respect to them, you will find adversarial examples for your LLM judges, almost guaranteed.

You can't do this for too long. You do maybe 10 steps or 20 steps, maybe it will work. But you can't do a 100 or a thousand because it's not obvious.

Because I know I understand it's not obvious, but basically, the model will find little cracks. It will find all these spurious things in the nooks and crannies of the giant model, and find a way to cheat it. So one example that's prominently in my mind is I think this I think this was probably public.

But basically, if you're using an LM judge for a reward, so you just give it a solution from a student and ask it if the student will or not. We were training with reinforcement learning against that reward function, and it worked really well. And then suddenly, the reward became extremely large.

Like, it was massive jump, and it did perfect. And you're looking at it like, wow. This this means the student is perfect in all these problems.

It's fully solved math. But actually, what's happening is that when you look at the completions that you're getting from the model, they are complete nonsense. They start out okay, and then they change to the the the the the the So it's just like, oh, okay.

Let's take two plus three, and we do this and this, and then the the the the the the the the the And you're looking at it, it's like, this is crazy. How is it getting a reward of one or a 100%? And you look at the LLM judge, and it turns out that the the the the the is an adversarial example for the model, and it assigns a 100% probability to it.

And it's just because this is an out of sample example to the LLM. Yeah. It's never seen it during training, and you're in pure generalization land.

Right.

the LLM to be a prompt injection model. Not even that. Prompt injection is way too fancy.

You're you're finding adversarial examples of what they're called. These are nonsensical solutions that are obviously wrong, but the model thinks are amazing.

And then so is it just gonna be, like, some sort of GAN like approach where you had to train models to be more robust? Yeah. To I think the labs are probably doing all that.

Like, okay. So the obvious thing is like, the the the should not get a 100% reward. Okay.

Well, take the the the the put in the training set of the LLM judge and say, this is not 100%. This is 0%. You can do this.

But every time you do this, you get a new LLM, and it still has adversarial examples. There's infinity adversarial examples. And I think probably if you iterate this a few times, it'll probably be harder and harder to find adversarial examples.

Mhmm. But I'm not 100% sure, because this thing has a trillion parameters or whatnot. So I bet you the the labs are trying.

I don't actually I I still think I still think we need other ideas.

could So like, this this idea of, like, every review Yeah. Review solution encompass synthetic examples, such that when you train on them, you get you get better, and, like, meta learn it in some way. And I think there's some papers that I'm starting to see pop out.

I only am at a stage of, like, reading abstracts, because a lot of these papers, they're you just ideas. Someone has to actually, like, make it work on a frontier LLM lab scale in full generality. Because when you see these papers, they pop up, and it's just like a little bit of noisy.

You know? It's cool ideas, but I haven't actually seen anyone convincingly show that this is possible. That said, the LLM labs are fairly closed.

So who knows what they're doing now.

some take on what what the analogy of, like, this thing is. Yeah. I do think that that we're missing some aspects there.

So as an example, when you're reading a book Yeah. I almost feel like, currently, when LLMs are reading a book, what that means is we stretch out the sequence of text, and the model is predicting the next token, and it's getting some knowledge from that. That's not really what humans do.

Right? So when you're reading a book, I almost don't even feel like the book is like exposition I'm supposed to be attending to and training on. Mhmm.

The book is a is a set of prompts for me to do synthetic data generation, or for you to get into a book club and talk about it with your friends. Yeah. And it's by manipulating that information that you actually gain that knowledge.

Yeah. And I I think we have no equivalent of that, again, with LLMs. They don't really do that, but I'd love to see during pretraining some kind of a stage that thinks through the material, and tries to reconcile it with what it already knows, and thinks through for, like, some amount of time, and gets that to work.

And so there's no equivalence of any of this. This is all research. There's some subtle very subtle that I think are very hard to understand reasons why it's not trivial.

If I can just describe one, why can we just synthetically generate and train on it? Well, because every synthetic example like, if I just give synthetic generation of the model thinking about a book, you look at it and you're like, this looks great. Why can't I train on it?

Well, you could try, but the model will actually get much worse if you continue trying. And that's because all of the samples you get from models are silently collapsed. They're silently this is not obvious if you look at any individual example of it.

They occupy a very tiny manifold of the possible space of sort of thoughts about content. So the LLMs, when they come off, they're they're what we call collapsed. They have a collapsed data distribution.

If you sample one easy way to say it is go to ChatGPT and ask it, tell me a joke. It only has, like, three jokes. It's not giving you the whole breadth of possible jokes.

Yeah. It's giving you, like it knows, like, three jokes. Yeah.

They're silently collapsed. So, basically, you're not getting the richness and the diversity and the entropy from these models as you would get from humans. So humans are a lot more sort of noisier, but at least they're not biased.

They're not in in a statistical sense, they're not silently collapsed. They maintain a huge amount of entropy. So how do you get synthetic regeneration to work despite the collapse and while maintaining the entropy is a research problem?

I guess what I'm saying is, say we have a chapter of a book and I ask an alum to think about It will give you something that looks very reasonable. Yeah. But if I ask it 10 times, you'll notice that all of them are the same.

prompt information and then get returns from that. Okay. Yeah.

Yeah. So any individual sample will look okay, but the distribution of it is is quite terrible. And it's quite terrible in such a way that if you continue training on too much of your own stuff, you actually collapse.

I actually think that there's no, like, fundamental solutions to this possibly. And I also think humans collapse over time. Mhmm.

I think this is again, these analogies are surprisingly good, but humans collapse during the course of their lives. This is why children have completely they haven't overfit yet. Yeah.

And they will say stuff that will shock you because it's kind of you can see where they're coming from, but it's just not the thing people say. Yeah. And because they are not yet collapsed.

But we're collapsed, we end up revisiting the same thoughts. We end up, you know, saying more and more of the same stuff, and the learning rates go down, and the collapse continues to get worse, and then, yeah, everything deteriorates.

when you're generating things in your head and then you're attending to it, you're kind of like training on your own samples. You're training on your synthetic data. And if you do it for too long, you go off rails, and you collapse way too much.

So you always have to, like, seek entropy in your life. Yeah. So talking to other people is a great source of entropy, and things like that.

they do kind of, like, have a lot more of an element compared to LLMs Yeah. Of, like, seeing the forest for the trees. Yeah.

And and we're not actually that good at memorization, which is actually a feature. Mhmm. Because we're not that good at memorization, we actually are kinda, like, forced to find the patterns, like, a more in a more general sense.

I think LLNs for in comparison are extremely good at memorization. They will recite passages from all these training sources. You can give them completely nonsensical data.

Like, you can take you can hash some amount of text or something like that. You get a completely random sequence. If you train on it, even just, I think, a single iteration or two, it can suddenly regurgitate the entire thing.

It will memorize it. There's no way a person can read a single sequence of random numbers and recite it to you. Yeah.

And that's a feature, not a bug, almost, because it forces you to, like, only learn their generalizable components. Whereas LLMs are distracted by all the memory that they have of the pre trained documents. And it's probably very distracting to them in a certain sense.

So that's why when I talk about the cognitive core, I actually want to remove the memory, which is what we talked about. I'd love to have it them have less memory so that they have to look things up, and that they only maintain the algorithms for, like, thought and the idea of an experiment and all this cognitive glue of of acting. And this is also relevant to preventing model collapse?

Let me think. I'm not sure. I think it's almost like a separate axis.

It's almost like the models are way too good at our memorization, and somehow we should we should remove that. And I think people people are much worse, but it's a good thing.

Yeah. I think that's a great question. I mean, you can imagine having a regularization for entropy and things like that.

I guess they just they don't work as well empirically because right now like, the models are collapsed, but I will say most of the tasks that we want of them don't actually demand the diversity. It's probably the the answer of what's going on. And so it's just that the model the frontier labs are trying to make the models useful.

And I kind of just feel like the diversity of the outputs is not so much number one, it's much harder to work with and evaluate and all this kind of stuff, but maybe it's not what's actually capturing most of the value. Yeah. In fact, it's actively penalized.

Right? If you if you're, like, super creative in RL, it's, like, not good. Yeah.

Or, like, maybe if you're doing a lot of writing help from LLMs and stuff like that, think it's probably bad because the models will give you these, like, silently all the same stuff. You know? So they're not they won't explore lots of different ways of answering a question.

Right? But I kinda feel like maybe this diversity is just not as big of a yeah. Maybe, like, yeah, not as many applications needed, so the models don't have it.

But then it's actually a problem at synthetic generation time, etcetera. So we're actually shooting ourselves in the foot by not allowing this entropy to maintain in the model. And I think possibly the labs should try harder.

I don't actually know if it's super fundamental. I don't actually know if I intended to to say that. I do think that I haven't done these experiments, but I do think that you could probably regularize the entropy to be to be higher.

So you're encouraging the model to give you more and more solutions. But you don't want it to start deviating too much from the training data. It's gonna start making up its own language.

It's gonna start using words that are extremely rare, you know, so it's gonna drift too much from the distribution. So I think controlling the distribution is just like a tricky it's just like someone just has to it's probably not trivial in that sense.

von Neumann probes Yeah. Yeah. Yeah.

How big does it have to be? So it's really interesting in the history of the field because at one point, everything was very scaling pilled in terms of like, oh, we're gonna make much bigger models, trillions of parameter models. And actually, what the models have done in size is they've gone up, and now they've actually kind of like, actually even come down.

See if their models are smaller. And even then, I actually think they memorized way too much. So I think I had a prediction a while back that I almost feel like we can get cognitive course that are very good at even like a billion billion parameters.

It it should be already like like, if you talk to a billion parameter model, I think in twenty years, you can actually have a very productive conversation. It thinks. And it's a lot more like a human.

But if you ask it some factual question, might have to look it up. But it knows that it doesn't know, and it might have to look it up, and they will just do all the reasonable things. That that's actually surprising that you think it will take a billion per because already we have a billion parameter models, or a couple billion parameter models that are like very intelligent.

Well, some of your models are like a trillion parameters. Right? But they remember so much stuff.

It's just Yeah.

No. Because I basically think that the training data is so here's the issue. The training data is the Internet, which is really terrible.

So there's a huge amount of gains to be made because the Internet is terrible. Yeah. Like, if you actually and even the Internet, when you and I think of the Internet, you're thinking of like, oh, Wall Street Journal, or that's not what this is.

When you're actually looking at a preaching dataset in the frontier lab, and you look at a random Internet document, it's total garbage. Like, I don't even know how this works at all. It's some, like, stock ticker symbols.

It's a huge amount of slop and garbage from, like, all the corners of the Internet. It's not like your Wall Street Journal article that's extremely rare. So I almost feel like because the Internet is so terrible, we actually have to sort of almost like build really big models to compress all that.

Most of that compression is memory work instead of like cognitive work. Interesting. But what we really want is the cognitive part, actually delete the memory.

Right. And then so it's like guess what I'm saying is like, we need intelligent models to help us refine even the pretraining set to just narrow it down to the cognitive components. And then I think you get away with a much smaller model, because it's a much better dataset, and you could train it on it.

But probably it's not trained directly on it. It's probably distilled for a much better model still. But why is the distilled version still a billion?

Is I guess the thing I'm curious about. I just feel like distillation work extremely well. So almost every small model, if you have a small model, it's almost certainly distilled.

Why would you train on Right. No. No.

But why is the distillation not, in ten years, not getting below 1,000,000,000? Oh, you think it should be smaller than a billion? I mean, come on.

Right? I don't know.

you're outdoing me. I think yeah. Maybe we could get a little bit smaller.

I mean, I I still think that there should be enough yeah. Maybe it can be smaller. I do think that practically speaking, you want the model to have some knowledge.

That's right. You don't want it to be looking up everything. Yeah.

Because then you can't, like, think in your head. You're looking up way too much stuff all the time. So I do think it needs to be some basic curriculum needs to be there for knowledge.

Yeah. I don't know that I have a super strong prediction. I do think that peep the labs are just being practical.

They have a flops budget Yeah. And a cost budget. It just turns out that pre training is not where you wanna put most of your flops or your cost.

So that's why the models have gotten smaller. Because they are a bit smaller, the pre training stage is smaller, etcetera, but they make it up in reinforcement learning and all this kind of stuff, mid training, and all this kind of stuff that follows. So they're just being practical in terms of all the stages and how you get the most bang for the buck.

So I guess, like, forecasting that trend, I think, is quite hard. I do still expect that there's so much longing for it. That's my basic that's my basic expectation.

And so I I have a very wide distribution here.

not expecting any giant For the large part. Yeah. I I expect the data sets to get much, much better because when you look at the average data sets, they're extremely terrible.

Like, so bad that I don't even know how anything works, to be honest. Like, look at the average example in the training set. Like, factual mistakes, errors Yeah.

Nonsensical things. Somehow when you do it at scale, the the noise washes away, and you're left with some of the signal. So datasets will improve a ton.

It's just everything gets better. So our hardware, our older kernels, older kernels for running the hardware, maximizing what you get with the hardware. You know, so NVIDIA is slowly tuning the actual hardware itself, tensor cores and so on.

All that needs to happen, will continue to happen. All the kernels will get better and utilize the chip to the max extent. All the algorithms will probably improve over optimization architecture, and just all the modeling components of how everything is done, and what the algorithms are that we're even training So I do I do kind of expect like a just very just everything.

Nothing dominates. Everything plus 20%.

y axis here? What is the how should we think about how AI is making progress? So I guess I have two answers to that.

Number one, I'm almost tempted to reject the question entirely, because again, I see this as an extension of computing. Have we talked about how to chart progress in computing? Or how do you chart progress in computing since nineteen seventies or whatever?

What is the x axis? So I kinda feel like the whole question is kind of funny from that perspective a little bit. But I will say, I guess, like, when people talk about AI and the original AGI, and how we spoke about it when we when OpenAI started, AGI was a system you can go to that can do any task that is economically valuable, any economically valuable task, at human performance or better.

Okay. So that was the definition. And I was pretty happy with that at the time.

And I kind of feel like I've stuck to that definition forever. And then people have made up all kinds of other definitions. But I I like I feel like I like that definition.

Now, number one, the first concession that people make all the time is they just take out all the physical stuff, because we're just talking about digital knowledge work. I feel like that's a pretty major concession compared to the original definition, which was like any task a human can do. I can lift things, etcetera.

Like, AI can't do that, obviously. So okay. But we'll take it.

What fraction of the economy are we taking away by saying, oh, only knowledge work? I don't actually know the numbers. I feel like it's about 10 to 20%, if I had to guess, is only knowledge work.

Someone could work from home and perform tasks, something like that. I still think it's a really large market. What is the size of the economy, and what is 20%?

We're still talking about a few trillion dollars of even in The US of market share almost, or like work. So it's still a very massive bucket. So but I guess, like, going back to definition, I guess what I would be looking for is, to what extent is that definition true?

So are there jobs or lots of tasks? If we think of tasks as, know well, not not jobs, but tasks kinda difficult. Because the the problem is, like, society will refactor based on the tasks that make up jobs compared to what's Yeah.

Based on what's automatable or not. But today, what jobs are replaceable by AI? So a good example recently was Jeff Hinton's prediction that radiologists would not be a job anymore, and this turned out to be very wrong in a bunch of ways.

Right? So radiologists are alive and well and growing, even though computer vision is really, really good at recognizing all the different things that they have to recognize in images. And it's just messy, complicated job with a lot of surfaces, and dealing with patients, and all this kind of stuff in the context of it.

So I guess, I don't actually know that by that definition, AI has made a huge amount of dent yet. But some of the some of the jobs maybe that I would be looking for have some features that I think make it very amenable to automation earlier than later. As an example, call center employees often come up, and I think rightly so.

Because call center employees have a number of simplifying properties with respect to what's automatable today. Their jobs are pretty simple. It's a sequence of tasks, and every task looks similar.

Like, you take a phone call with a person, it's ten minutes of interaction, or whatever it is, probably a bit longer, in my experience, a lot longer. And you complete some task in some scheme, and you change some database entries around, or something like that. So you keep repeating something over and over again, and that's your job.

So, basically, you do wanna bring in the task horizon, how long it takes to perform a task, and then you want to also remove context. Like, you're not dealing with different parts of services of companies or other customers. It's just the database, you, and a person you're serving.

And so it's more closed, it's more understandable, and it's purely digital. So I I would be looking for those things. But even there, I'm not actually looking at full automation yet.

I'm looking for an autonomy slider, and I almost expect that we are not gonna instantly replace people. We're going to be swapping in AIs that do 80% of the volume. They delegate 20% of volume to humans, and humans are supervising teams of five AIs doing the call center work that's more rote.

So I would be looking for new interfaces, or new companies that provide some kind of a layer that allows you to manage some of these AIs that are not yet perfect. Yeah. And then I would expect that across the economy, and a lot of jobs are a lot harder than call center employee.

question. I don't think we're currently seeing that with radiology or and I don't have like in my understand but I think radiology is not a good example, basically. I don't know why Jeff Hinton picks on radiology, because I think it's an extremely messy messy, complicated profession.

Yeah. So I would be a lot more interested in what's happening with call center employees today, for example. Because I would expect a lot of the road stuff to be automatable today.

And I don't have a first level access to it, but maybe I would be looking for trends of what's happening with the call center Yeah. Maybe some of the things I would also expect is maybe they are swapping in AI, but then I would still wait for a year or two, because I would potentially expect them to pull pull back and actually rehire some of the people. Yeah.

I think there's been evidence that that's already been happening in the Okay. Generally in the companies that have been adopting AI, which I think is quite surprising. Yeah.

And I I also find what was really surprising.

And that's because coding has always fundamentally worked around text. It's computer terminals and text, and everything is based around text. Yeah.

And LLMs, the way they're trained on the Internet, love text. And so they're perfect text processors. Yeah.

And there's all this data out there, and it's just perfect fit. And also, we have a lot of infrastructure pre built for handling code and text. So for example, we have a Visual Studio Code or, you know Right.

Your favorite IDE showing you code. And an agent can plug into that. So for example, if an agent has a diff where it made some change, we suddenly have all this code already that shows all the differences to a code base using a diff.

So we've it's almost like we've pre built a lot of the a lot of the infrastructure for code. Now contrast that with some of the things that that don't enjoy that at all. So as an example, like, there's people trying to build automation, not for coding, but for example, for slides.

Like, I saw a company doing slides. That's much, much harder. And the reason it's much, much harder is because slides are not text.

Yeah. Slides are little graphics, and they're arranged spatially, and there's visual component to it. And and slides don't have this prebuilt infrastructure.

Like, example, if an agent is to make a different change to your slides, how does a thing show you the diff? How do you see the diff? There's no there's nothing that shows diffs for slides.

Mhmm. So someone has to build it. So it's just some of these things are not amenable to AIs as they are, which is text processors, and code surprisingly is.

I think that makes sense. I mean, I would say yeah. There it's I'm not saying that anything text is trivial.

Right? Yeah. I do think that code is like it's pretty structured.

Text is gonna be a lot more flowery, and and there's a lot more like like entropy in text, I would say. I don't know how else to put it. And also, I mean, code is hard.

And so people sort of feel quite empowered by LLMs even from, like, simple simple kind of knowledge. I basically I don't actually know that I have a very good answer. I mean, obviously, like, text makes it much, much easier maybe.

It's maybe why why would I put it. But it doesn't mean that all text is trivial. Mhmm.

I guess I think I see it as like a progression of automation in society. Right? And, again, like extrapolating the trend of computing.

I just I feel like there'll be a gradual automation of a lot of things, and superintelligence will be sort of like the extrapolation of that. So I do think we expect more and more autonomous entities over time that are doing a lot of the digital work, and then eventually even the physical work, probably some amount of time later. But basically, I see it as just automation, roughly speaking.

I guess automation includes the things humans can already do, and superintelligence supplies things humans Well, but some of the things that people do is invent new things, which I would just put into the automation, if that makes sense. Yeah.

like you mentioned, we can run all of this on a computer cluster, etcetera, and much faster and all this thing. Yeah. I mean, maybe some of the scenarios, for example, that I start to get, like, nervous about with respect with respect to when the world looks like that is this kind of, like, gradual loss of control and understanding of what's happening.

And I I think that's actually the most likely outcome probably Mhmm. Is that there will be a gradual loss of understanding of and we'll we'll gradually layer all the stuff everywhere, and there'll be fewer and fewer people who understand it. And then there will be a sort of this, like, scenario of a gradual loss of control and understanding of what's happening.

That to me seems most likely outcome of how all the stuff will go down. Let me probe on that a bit. It's not clear to me that loss of control and loss of understanding are the same things.

I guess I expect loss of both. Yeah.

I don't know what this looks like. But if I was to write sci fi novels, they would look along the lines of not even a single, like, entity or something like that. So that just sort of, like, takes over everything.

But actually, like, multiple competing entities that gradually become more and more autonomous, and some of them go rogue, and the others, like, fight them off and all this kind of stuff. And it's like this this hot pot of completely autonomous activity that we've delegated to.

I mean, I basically expect there to be I mean, a lot of these things, I mean, they will be tools to people, and the people could some of the population is like, they're acting on behalf of people or something like that.

I do, but it's business as usual. Because we're we're in an intelligence explosion already and have been for decades. When you look at g e it's basically the GDP curve.

That is an exponential weighted sum over so many aspects of the industry. Everything is gradually being automated, has been for hundreds of years. Industrial revolution is automation, and some of the physical components, tool building, and all this kind of Compilers are early software automation, etcetera.

So I kinda feel like we've been recursively self improving and exploding for for a long time. Maybe another way to see it is I mean, Earth was a pretty, I mean, if you don't look at the biomechanics and so on, was a pretty boring place, think, and looked very similar if you just look from space. And and Earth is spinning, and then like, we're in the middle of this, like, firecracker event.

Right. But we're seeing it in slow motion. But I definitely feel like this has this has already happened for a very long time.

And I again, like, I I don't see AI as, a distinct technology with respect to what has already been happening for a long time. Is there this you think it's gonna continue sort of this hyper exponential trend? And that's why like, this is this was very interesting to me because I was I was trying to find AI in the GDP for a while.

I thought that GDP should go up. But then I looked at some of the other technologies that I thought were were very transformative, like maybe computers or mobile phones or etcetera. You can't find them in GDP.

GDP is the same exponential. And it's just that even for example, the early iPhone didn't have the App Store, and it didn't have a lot of the bells and whistles that the modern iPhone has. And so even though we think of 2008 was it, when iPhone came out as, like, some major seismic change, it's actually not.

Everything is, like, so spread out and so slowly diffuses, that everything ends up being averaged up into the same exponential. And it's the exact same thing with computers. You can't find them in the GDP as like, oh, have computers now.

That's not what happened, because it's such a slow progression. And with AI, we're gonna see the exact same thing. It's just more automation.

It allows us to write different kinds of programs that we couldn't write before, but AI is still fundamentally a program. And it's a new kind of computer, and a new kind of kind of computing system, but it has all these problems, it's gonna diffuse over over time, and it's still gonna add up to the same exponential.

But the rate of growth is the exact same. It's 2%. So are you saying the rate of growth will No.

Obviously, I expect this the rate of growth has also stayed roughly constant. Right? For only the last two hundred, three hundred years.

But over the course of human history, it's like exploded. Right? It's like gone from like 0% basically to like faster, faster, faster industrial explosion Yeah.

2%. Basically, I guess what I'm saying is for a while, I tried to find AI or look for AI in like the GDP curve, and I kind of convinced myself that this is false. And that even when people talk about recursive self improvement and labs and stuff like that, I even don't this is business as usual.

Of course, it's gonna recursively self improve, and it's been recursively self improving. Like, LLMs allow the engineers to work much more efficiently Yeah. To build the next round of LLM.

And a lot more of the components are being automated and and tuned, etcetera. So all the engineers having access to Google search is is sort of part of it. All the engineers having an ID, all all of them having autocomplete or having cloth code, etcetera.

It's all just part of the same speed up of the whole thing. So it's just so smooth.

pretty willing to be convinced one way or another on this point. But I will say, for example, computing is labor. Computing was labor.

Computers, like, a lot of jobs disappeared because computers are automating a bunch of digital information processing that you now don't need a human for. And so computers are labor, and that has played out. And, you know, self driving as an example is also like computers doing labor.

I see where it's coming from. At the same time, I do feel like people make this assumption of like, okay. We have God in the box, and now it can do everything.

And it's just it just won't look like that. Yeah. It's gonna be it's gonna be able to do some of the things.

It's gonna fail at some other things. It's gonna be gradually put into society, and basically, we'll end up with the same pattern, is my prediction. Yeah.

And because because this assumption of suddenly having a completely intelligent, fully flexible, fully general human in a box, and we can dispense it with arbitrary problems in society, I I I don't think that we will have this, like, discrete change.

I think I understand, but I still think that you're presupposing some discrete jump. There there's some unlock that we're waiting to claim. Mhmm.

And suddenly, we're gonna have geniuses in data centers. And I I still think you're presupposing some discrete jump that I think has basically no historical precedent that I can't find in any of the statistics, and that I think probably won't happen. I mean, the initial revolution is such a jump.

Right? You went from, like, 0% grow or 0.2% growth to 2% growth.

I'm just saying, like, you'll see another jump like that. I I I'm I'm a little bit suspicious. I would have to look at it.

I would I'm a little bit suspicious Mhmm. And I would have to take a look. For example, like, maybe the some of the logs are are not very good from before the industrial revolution or something like that.

So I'm a little bit suspicious of it, but yeah. Maybe you're right. I don't I don't have strong opinions.

Maybe you're saying that this was a singular event that was extremely magical, and you're saying that maybe there's gonna be another event that's gonna be just like that, extremely magical. It will break paradigm and so on. I actually don't think the I mean, the the crucial thing with the Industrial Revolution was that it was not magical.

Right? Mhmm.

there's gonna be a single moment where we've made the crucial assumption overhang that's being unlocked. Like, maybe there's a new energy source. There's there's some unlock, in this case, some kind of a cognitive capacity, and there's an overhang of cognitive cognitive work to do.

That's right. And you're expecting that overhang to be filled by this new technology when it crosses the threshold. Yeah.

yeah, it's really hard to tell. Yeah. I understand that viewpoint.

Yeah. I don't intuitively feel that viewpoint. So we just got access to Google's VO 3.

Hi.

Love Nick Lane's books, by the way. So yeah. I I was just listening to the to his podcast some way up here.

With respect to intelligence and its evolution, I do came it came fairly I mean, it's very, very recent. Right? I am surprised that it evolved.

Yeah. I find it fascinating to think about all the worlds out there. Like, say there's a thousand planets like Earth and what they look like.

I think Nick Lane was here talking about some of the early parts. Right? Like, okay.

He expects basically very similar life forms, roughly speaking, and bacteria like things in most of them. Yeah. And then there's a few breaks in there.

I would expect that the evolution of intelligence intuitively feels to me like it should be fairly rare event. And there have been animals for I guess maybe you should base it on how long something something has existed. So for example, if bacteria have been around for two billion years and nothing happened, then going to your carrier, it's probably pretty hard because because bacteria actually came up quite early in Earth's evolution Yeah.

Or history. And so I guess, how long have we had animals? Maybe a couple hundred million years, like multicellular animals that, like, run, run, crawl, etcetera, which is maybe 10% of Earth's life span or something like that.

So or maybe on that time scale, it's actually not not too tricky. I still feel like it's still surprising to me, I think, intuitively, that it developed. I would maybe expect just a lot of, like, animal like life forms doing animal like things.

The fact that you can get something that creates culture and knowledge Yeah. And accumulates it is is it is surprising to me. The okay.

So there's so there's actually a couple of interesting follow ups.

Yes. Basically, it's so hard to tell, right, with any of this stuff. I guess you can base it a little bit on how long something has existed or how long it feels like something has been bottlenecked.

Yeah. So Nicholene is very good about describing this, like, very apparent bottleneck in bacteria and archaea. Yeah.

For two billion years, nothing happened. Like, extreme diversity of chemical of biochemistry, and yet nothing that grows to become Yeah. Animals.

Two billion years. I I don't know that we've seen exactly that kind of an equivalent with animals and intelligence, to your point. Right?

But I guess maybe we could also look at it with respect to how many times we think evolution sorry, intelligence has, like, individually sprung up. That's a really good that that that's a really good thing to investigate. Maybe one thought on that is, I almost feel like well, there's the hominid intelligence.

And there's, I would say, like, the bird intelligence. Right? Yeah.

Like ravens, etcetera, are extremely clever. Yeah. But they actually their brain brain parts are actually quite distinct, and we don't have that much existence.

So maybe that's an a slight event of there's a slight indication of maybe intelligence springing up a few times. And so in that case, you'd maybe expect it more frequently or something like that. Yeah.

Oh, yeah. And just stuff to work with. I mean, I'm guessing it would be harder to if I was dolphin.

Yeah. I mean, how do you do you can't have fire, for example, and stuff like that. Mean, the the probably, like, the universe of things you can do in water, like, water is probably lower than what you can do on land Yeah.

Just chemically. Right. Yeah.

I do I do agree with this with this viewpoint of these niches and what's what's being incentivized. I still find it kind of miraculous that I don't I I would have maybe expected things to get stuck on, like, animals with bigger muscles. You know?

Yeah.

building the algorithm to learn You have to in lifetime. Yeah. Exactly.

You have to incentivize some kind of adaptability. You actually want something that you actually want environments that are unpredictable. So evolution can't bake your algorithms into your weights.

A lot of a lot of animals are basically pre baked in this sense. So And humans have to figure it out at test time when they get born. And so maybe there was you actually want these kinds of environments that actually change really rapidly or something like that where you can't foresee what will work well.

And so you actually put all that intelligent you create intelligence to figure it out at test time.

Yes and no, because LLM don't really have the equivalent of culture. And maybe we're giving them way too much and incentivizing not to create it or something like that. But I guess, like, invention of culture and of written record, and of passing down notes between each other, I don't think there's an equivalent of that with LLM's right now.

So LMs don't really have culture right now, and it's kinda like one of the, I think, impediments, I would say. Can can you give me some sense of what LLM culture might look like? So in the simplest case, it would be a giant scratch pad that the LLM can edit.

And as it's reading stuff or as it's helping out work with work, it's editing the scratch pad for itself. Why can't an LLM write a book for the other LLM's? That would be cool.

Yeah.

AI Yeah. Civilization and culture. I think there's two powerful ideas in the realm of multi agent that have both not been, like, really claimed or or so on.

The first one, I would say, is culture. And LLMs basically are growing repertoire of knowledge for their own purposes. The second one looks a lot more like the powerful idea of self play.

In my mind, it's extremely powerful. So evolution actually has a lot of competition, basically, driving intelligence and and evolution. And for in AlphaGo, more algorithmically, like, AlphaGo is playing against itself, and that's how it learns to get really good at Go.

And there's no equivalent of self playing LLMs, But I would expect that to also exist, but no one has done it yet. Like, why can't an LLM, for example, create a bunch of problems that another LLM is learning to solve? And then the the LLM is always trying to, like, serve more and more difficult problems, stuff like that.

You know? So, like, I think there's a bunch of ways to actually organize it, and I think it's a realm of research. But I think I haven't seen anything that convincingly, like, claims both of those, like multi agent improvements.

I still think we're mostly in the realm of a single individual agent, but I think I also think that will change. And and in the realm of culture also, would bucket also organizations, and we haven't seen anything like that convincingly either. Mhmm.

somehow remarkably again, of these analogies work and they shouldn't, but somehow remarkably they do. A lot of the smaller models or the dumber like, the smaller models somehow remarkably resemble, like, a kindergarten student or then, like, a elementary school student or a high school student, etcetera. And somehow, we still haven't, like, graduated enough where this stuff can take over.

Like, it's still mostly like, my cloth coat or codex Mhmm. They still kinda feel like this elementary grade student. I know that they can take PhD quizzes, but they still cognitively feel like a kindergartner or an elementary school student.

Interesting. So I I don't think they can create culture because they're still kids. You know?

Like, they're savant kids. Yeah. They have absurd they have perfect memory of all this stuff, etcetera.

And they can convincingly create all kinds of slop that looks really good. But I still think they don't really know what they're doing, and they don't really have the cognition across all these little checkboxes that we still have to Yeah.

So I would say one thing I will almost instantly also push back on is this is not even near done. So in a bunch of ways that I'm gonna get to. I do think that self driving is very interesting because it's definitely like where I get a lot of my intuitions because I spent five years on it.

And it has this entire history where actually the first demos of self driving go all the way to ninety eighties. You can see a demo from CMU in 1986. There's a truck that's driving itself on roads.

But okay. Fast forward. I think when I was joining Tesla, I had I had a very early demo of a Waymo, and it basically gave me a perfect drive in 02/2014 or something like that.

So perfect Waymo drive a decade ago. Gave took us around Palo Alto and so on because I had a friend who worked there. And I thought it was, like, very close, and then still took a long time.

And I do think that some there's for some kinds of tasks and jobs and so on, there's a very large demo to product gap, where the demo is very easy, but the products are very hard. And it's especially the case in cases like self driving, where the cost of failure is too high. Right?

Many industries, tasks, jobs maybe don't have that property. But when you do have that property, that definitely increases the timelines. I do think that, for example, in software engineering, I do actually think that that property does exist.

I think for a lot of vibe coding, it doesn't. But I think if you're writing actual production grade code, I think that property should exist. Because any kind of mistake actually leads to a security vulnerability or something like that.

And millions and hundreds of millions of people's personal social security numbers, etcetera, get leaked or something like that. And so I I do think that it is a case that in software, people should be careful. Kind of like in self driving.

Like in self driving, if you if things go wrong, you might get injury. In I guess, there's worse outcomes. But I guess in in software, I almost feel like it's almost unbounded how terrible some things could be.

Interesting. So I do think that they share that property. And then I think, basically, what takes the long amount of time and the way to think about it is that it's a march of nines, and every single nine is a constant amount of work.

So every single nine is the same amount of work. So when you get a demo, and something works 90% of the time, that's just that's just what the first nine. And then you need the second nine, and third nine, fourth nine, fifth nine.

And while I was at Tesla for, was it five years or so, I think we went through maybe three nines or two nines. Don't know what it is, you know. But like multiple nines of iteration, there's still more nines to go.

And so that's why these things take take so long. And so it's definitely formative for me, like, seeing something that was a demo. I'm very unimpressed by demos.

So whenever I see demos of anything, I'm extremely unimpressed by that. It works better if you can. If it's a demo that someone cooked up and is just showing you its worst.

If you can interact with it, it's a bit better. But even then, you're not done. You need actual product.

It's gonna face all these challenges in when it comes to in contact with reality, and all these different pockets of behavior that need patching. And so I think we're gonna see all this stuff play out. It's a march of nines.

Each nine is constant. Demos are encouraging. Still a huge amount of work to do.

I do think it is a kind of a critical safety domain, unless you're doing byte coding, which is all nice and fun and so on. And so that's why I think this also enforced my timelines from that perspective.

or in terms of tokens, it would be seven years. But in terms of wall clock time, it would be pretty much harder problem. I mean, self driving is just one of thousands of things that people do.

Yeah. It's almost like a single vertical, I suppose. Whereas when we're talking about general software engineering, it's even more there's more surface area.

And so now deploying AIs across different domains will sort of be like deploying a self driving car with current models to a different city, which is hard, but not like a ten year long task. Mhmm. Yeah.

Basically, I'm not a 100% sure if I fully agree with that. I don't know that we're how much we're getting for free. And I still think there's like a lot of gaps in understanding in what we are getting.

I mean, we're definitely getting more generalizable intelligence in a single entity. Whereas self driving is a very special purpose task that requires in some sense, building a special purpose task is maybe even harder in a circumstance because it doesn't like fallout from a more general thing that you're doing at scale, if that makes sense. So but I still think that the analogy doesn't I still don't know if it fully resonates because, like, the LMs are still pretty fallible, and I still think that they have a lot of gaps and that it still needs to be filled in.

And I don't think that we're getting, like, magical generalization completely out of the box sort of in in a certain sense. And the other aspect that I want to also actually return to when I was in the in the beginning was self driving cars are no longer done still. Mhmm.

So even though so the deployments still are pretty minimal. Right? So even Waymo and so on has very few cars, and they're doing that roughly speaking because they're not economical.

Right? Because they've built something that that lives in the future. And so they they had to, like, pull back future, but they had had to make it uneconomical.

So they have all these, like, you know, there's all these costs, not just marginal costs for those cars and their operation and maintenance, but also the CapEx of the entire thing. So making economical is still gonna be a slog, I think, for them. And then, also, I think when you look at these cars and there's no one driving, I also think it's little bit deceiving because there are actually very elaborate teleoperation centers Mhmm.

Of people actually kind of like in a loop with these cars. And I don't have the I don't have the full extent of it, but I think there's more human in the loop than you might expect. And there's people somewhere out there, basically, beaming in from the sky.

And I don't actually know if they're fully in the loop with the driving. May I think some of the times they are. But they're certainly involved, there are people.

And in some sense, haven't actually removed the person. We've moved them to somewhere where we can't see them. I still think there will be some work, as you mentioned, going from environment to environment.

And so I think there's still challenges to to make self driving real. But I I do agree that it's definitely across the threshold where it kinda feels real, unless it's really teleoperated. For example, Waymo can't go to all the different parts of the city.

My suspicion is it's like parts of city where you don't get a good signal. Anyway, so basically, I don't actually know anything about the stack. I mean, I'm just making up make out.

Sorry.

Sorry. I don't know anything about the specifics of Waymo. Thought I talked about them.

Actually, by the way, love Waymo, and I take it all the time. Yeah. So I don't wanna say, like Sure.

I just think that people, again, are sometimes a little bit too naive about some of the progress, and I still think there's a huge amount of work. And I think Tesla took, in my mind, a lot more scalable approach. Yeah.

And I think the team is doing extremely well, and it's gonna and I I I'm kind of like on the record for predicting how this thing will go, which is like, way more like early start because you can package up so many sensors. But I do think Tesla is taking the more scalable strategy, it's gonna look a lot more like So I think this will have to still play out and hasn't. But basically, like, I don't wanna talk about self driving as something that took a decade because it didn't take it didn't take yet.

If that makes sense. Because one, it's the the start is at 1980, not 10 years ago, and then two, the end is not here yet. Yeah.

The end is not not near yet. Yeah. Because when we're talking about self driving, usually in my mind, it's self driving at scale.

Yeah.

much more favorable. I think that's right. I think if you're sticking in the realm of bits, bits are, like, a million times easier than anything that touches the physical world.

No. I definitely grant that. Bits are completely changeable, arbitrarily reshuffled at a very rapid speed.

So you would expect a lot more Yeah. Faster adaptation also in the industry and so on. And then what was the first one?

The latency requirements. And Oh, the latency requirements. And these implications for model size.

I think that's roughly right. I mean, I also think that if we are talking about knowledge work at scale, there will be some latency requirements Sure. Practically speaking, because we you know, we're gonna have to make create a huge amount of compute Yeah.

And serve that. And then I think, like, the last aspect that I very briefly wanna also talk about is, like, all the all the rest of it. The just all the rest of it.

So what does society think about it? What is the legal how is it working legally? How is it working insurance wise?

Who's really like, what is the what what are those layers of it and aspects of it? What happens with what is the equivalent of people putting a cone on a Waymo? Yeah.

You know? There's gonna be equivalents of all that. And so I I do think that I almost feel like self driving is a very nice analogy that you can borrow things from.

Yeah. What is the equivalent of the cone in the car? What is the equivalent of a tele operating worker who's, like, hidden away?

Mhmm.

Or do you is that a separate question? Kinda like what happened with railroads and all this kind of stuff. With what?

Sorry? Was it railroads? Oh, sorry.

It was There was there is, like, historical precedent. Or was it with telecommunication industry? Right?

Like, pre paving the Internet that only came, like, a decade later, you know, and creating, like, a whole bubble in the telecommunications industry in the late nineties kind of thing? Yeah. So I don't know.

I mean, I I understand I'm sounding very pessimistic here. I'm only doing that I'm actually optimistic. I think this will work.

Think it's tractable. I'm only sounding pessimistic because when I go on my Twitter timeline, I see all this stuff that makes no sense to me. And and I think there's a lot of reasons for why that exists, and I think a lot of it is, I think, honestly, just fundraising.

It's just incentive structures. A lot of it may be fundraising. A lot of it is just attention you know, converting attention to money on the Internet, you know, stuff like that.

So I think there's there's a lot of that going on, and I think I'm only reacting to that. But I'm still, like, overall very bullish on technology. I think we're gonna work through all this stuff, and I think there's been a rapid amount of progress.

I don't actually know that there's overbuilding. I think that there's gonna be we're gonna be able to gobble up what, in my understanding, is being built. Because I do think that, for example, Cloud Code or OpenAI Codex and stuff like that, they didn't even exist a year ago.

Right? Is that right? I think it's roughly right.

This is miraculous technology that didn't exist. I think there's gonna be a huge amount of demand as there as we see the demand in Chachi p t already and so on. So, yeah, I don't actually know that there's overbuilding.

But I guess I'm just reacting to, like, some of the very fast timelines that people continue to say incorrectly. And I've heard many, many times over the course of my fifteen years in AI, where very reputable people keep getting this wrong all the time. And I think I want us to be properly calibrated, and I think some of this also it does have like geopolitical ramifications and things like that when like some of these questions, and I think I don't want people to make mistakes on that on that sphere of things.

So I do want us to be grounded in reality of what technology is and isn't.

I guess maybe, like, the way I would put it is, I feel some amount of, like, determinism around the things that AI labs are doing. And I feel like I could help out there, but I don't know that I would, like, uniquely I don't know that I would, like, uniquely improve it. But I I think, like, my personal big fear is that a lot of the stuff happens on the side of humanity, that humanity gets disempowered by it.

Mhmm. And I I I kind of like I care not just about all the Dyson spheres that we're gonna build, and that AI is gonna build in a fully autonomous way. I care about what happens to humans.

Yeah. And I want humans to be well off in this future. And I feel like that's where I can a lot more uniquely add value than, like, an incremental improvement in the frontier lab.

And so I guess I'm most afraid of something maybe like depicted in movies like Wally or Idiocracy or something like that, where humanity is sort of on the side of this stuff. And I want humans to be much, much better in this future. Mhmm.

And so I guess, to me, this is kind of like through education that you can actually achieve this. And and so what are you working on there? Oh, yeah.

So Eureka is trying to build, I think maybe the easiest way I can describe it is we're trying to build the Starfleet Academy. I don't know if you've watched Star Trek. I haven't.

But yeah. Okay. Starfleet Academy is this, like, elite institution for frontier technology, building spaceships and graduating cadets to be, like, you know, the pilots of these spaceships and whatnot.

So I just imagine, like, an elite institution for technical knowledge and and basically a kind of school that's very up to date and very, like, premier institution.

well. Because you are one of the world masters at it. And then curious both about how you think about it for content you've already put out there on YouTube.

Yeah. But also, to the extent it's any different, how you think about it for Eureka? Yeah.

Yeah. With respect to Eureka, I think, like, one thing that is very fascinating to me about education is, like, I do think education will pretty fundamentally change with AIs on the side. And I think it has to be rewired and changed to some extent.

I still think that we're pretty early. I think there's gonna be a lot of people who are gonna try to do the obvious things, which is like, oh, have an LLM and ask it questions and get you know, do all the basic things that you would do via prompting right now. I I think it's helpful, but it still feels to me a bit slop, like slop.

I want I'd like to do it properly, and I think the capability is not there for what I would want. What I'd want is, like, an actual tutor experience. Maybe a prominent example in my mind is I was recently learning Korean.

It's language learning. And I went through a phase where I was learning Korean by myself on the Internet. I went through a phase where I was actually part of a small class in Korea, taking a taking a Korean with a bunch of other people, which was really funny.

But we had a teacher and, like, 10 people or so taking Korean. And then I switched to a one on one tutor. And I guess what was fascinating to me is I think I had a really good tutor.

But, I mean, just thinking through, like, what this tutor was doing for me and how incredible that experience was and how high the bar is for, like, what I actually want to build eventually. Mhmm. Because, I mean, she was extremely so she instantly, from a very short conversation, understood, like, where I am as a student, what I know and don't know.

And she was able to, like, probe exactly, like, the kinds of questions or things to understand my world model. No LLM will do that for you. 100% right now, not even close.

Right? But a tutor will do that if if they're good. Once she understands, she actually really served me all the things that I needed at my current sliver of capability.

I need to be always appropriately challenged. I can't be faced with something too hard or too trivial. And a tutor is really good at serving you just the right stuff.

And so, basically, I felt like I was the only constraint to learning, like my own. I was the only constraint. I was always given the perfect information.

I'm the only constraint. And I felt good because I'm the only impediment that exists. It's not that I can't find knowledge, or that it's not properly explained, or etcetera.

Like, it's just my ability to memorize, and so on. And this is what I want for people. How do you automate that?

So a very good question. At the current capability, you don't. But I do think that with as and that's why I think it's not actually the right right time to actually build this kind of an AI tutor.

I still think it's a useful product, and lots of people will build it. But I still feel like the bar is so high, and the capability is not there. But, I mean, even today, I would say Charjipotinib is an extremely valuable educational product.

But I think for me, was so fascinating to see how high the bar is. And when I was with her, I almost felt like, there's no way I can build this. But you are building it.

Right? Anyone who's had a really good tutor is like, how are you gonna build this? So I guess I just I'm waiting for that capability.

I I do think that in a lot of ways in the industry for example, I did some AI consulting for computer vision. Yeah. A lot of my times, the value that I brought to the company was telling them not to use AI.

It wasn't like I was the AI expert, and they described the problem, and I said, don't use AI. This was my value add. And I feel like it's insane in education right now where I kinda feel like for what I have in mind, it's not yet the time, but the time will come.

But for now, I'm building something that looks maybe a bit more conventional, that has a physical and digital component and so on. But I think there's obvious there's obvious it's obvious how this should look like in the future. Mhmm.

To the extent you're willing to say, what is the thing you hope will be released this year or next year? Well, so I'm building the first course, and I want to have a really, really good course. State of the art obvious state of the art destination you go to learn AI in this case.

Mhmm. Because that's just what I'm familiar with, so I think it's a really good first product to get to be really good. And so that's what I'm building.

And NanaChat, which you briefly mentioned, is a capstone project of LLM one zero one n, which is a class that I'm building. So that's a really big piece of it, but now I have to build out a lot of the intermediates, and then I have to actually, like, hire a small team of, you know, TAs and so on, actually, like, build the entire course. And maybe one more thing that I would say is, many times when people think about education, they think about sort of, like, the more what I would say is, like, kind of a softer component of, like, diffusing knowledge or, like but I actually have something very hard and technical in mind.

And so in my mind, education is kind of, like, the very difficult technical, like, process of building ramps to knowledge. Mhmm. So in my mind, nanochat is a ramp to knowledge because it's a very simple it's like the super simplified full stack thing.

If you give this artifact to someone and they, like, look through it, they're learning a ton of stuff. Yeah. And so it's giving you a lot of what I call eurekas per second Yeah.

Which is like understanding per second. That's what I want. Lots of eurekas per second.

Mhmm. And so to me, this is a technical problem of how do we build these ramps to knowledge. And so I almost think of eureka as almost like a it's not like maybe that different maybe through to some of the fort frontier labs or some of the work that's gonna be going on, because I want to figure out how to build these frontier these ramps very efficiently so that people are never stuck.

And everything is always not too hard or not too not too trivial, and you can't you have just the right material to actually progress. Yeah.

Yeah. In the source material of the class. Right?

That's, like, fundamentally what the course is. I mean, I think you always have to be calibrated to what the capability what capability exists in the industry. And I think a lot of people are gonna pursue, like, oh, just ask Chachi etcetera.

But I I think, like, right now, for example, if you go to Chachi PT and you say, oh, teach me AI. There's no way it's not I mean, it's gonna give you some slop. Right?

Right. Like, when I AI is never gonna write nano chat right now, but nano chat is a really useful, I think, intermediate So I still I'm collaborating with AI to create all this material. So AI is still fundamentally very helpful.

Earlier on, I built a CS231n at Stanford, which was one of the earlier actually, sorry. I think it was the first deep learning class at Stanford, which became very popular. And the difference in building out two thirty one n and LM101n now is quite stark.

Because I feel really empowered by the LMs as they exist right now, but I'm very much in the loop. Mhmm. So they're helping me build little materials.

I go much faster. They're doing a lot of boring stuff, etcetera. So I feel like I'm developing the course much faster, and those LLM infused in it, but it's not yet at a place where I can creatively create the content.

I'm still there to do that. So, like, I think the trickiness is always calibrating yourself to what exists.

their understanding into these ramps. Right? Yep.

So I think it would change over time. So I think right now, it would be hiring faculty Mhmm. To help work hand in hand with AI and a team of people probably to build state of the art courses.

Yeah. And then I think over time, it can maybe some of the TAs can actually become AIs. Because some of the TAs, like, okay.

You just take all the course materials, and then I think you could serve a very good, like, automated TA Yeah. For the student when they have more basic questions or something like that. Right?

But I think think you'll need faculty for the overall architecture of a course Mhmm. And making sure that it fits. And so I kinda see a progression of how this will evolve.

And maybe at some future point, I'm not even that useful, and AI is doing most of the design much better than I could. But I still think that that's gonna take some time to play out.

that's the only way to offer the state of the art experience for the student ultimately. So, yeah, I do expect that I would hire faculty, but I will probably stick around in AI for some time. Yeah.

But I do have something, I think, more conventional in mind for the current capability, I think, than what people would probably anticipate. And when I'm building Starfleet Academy, I do probably imagine a physical institution, and maybe a tier below that, a digital offering, that is not the not the state of the art experience you would get when someone comes in physically Yeah. Full time, and we work through material from start to end and make sure you understand it.

That's the physical offering. Yeah. The digital offering is, yeah, a bunch of stuff on the Internet and maybe some LLM assistant, and it's a bit more gimmicky in a tier below, but at least it's accessible to, like, 8,000,000,000 people.

So Yeah.

really engaging with material. Yeah. And I think there's gonna have to be a lot of not just education, but also reeducation.

And I would love to help out there, because I think the jobs will probably change quite a bit. And so for example, today, lot of people are trying to upskill in AI specifically. So I think it's a really good course to teach in this in this respect.

And, yeah, I think the motivation wise, before AGI, motivation is very simple to solve because people want to make money, and this is how you make money in the industry today. I think post AGI is a lot more interesting, possibly, because, yeah, if everything is automated and there's nothing to do for anyone, why would anyone go to a school, etcetera? So I think I guess, like, I often say that pre AGI education is useful.

Post AGI education is fun. And in a similar way, as people for example, people go to gym today. Yeah.

But we don't need their physical strength to manipulate heavy objects because we have machines to do that. Yeah. They still go to gym.

Why do they go to gym? Well, because it's fun. It's healthy.

It's and it's and you look hot when you have a six pack. I don't know. Yeah.

I guess, like and so it's I guess what I'm saying is it's attractive for people to do that Right. In a certain, like, very deep psychological evolutionary sense for humanity. Yeah.

And so I kind of think that education will kind of play out in the same way. Like, you'll go to school, like you go to gym, and you'll and I think that right now, I think not that many people learn because learning is hard. You bounce from material because and some people overcome that barrier, but for most people, it's hard.

Yeah. But I I do think that we should it's a technical problem to solve. It's a technical problem to do what my tutor did for me when I was learning Korean.

I think it's tractable and buildable, and so much to build it. And I think it's gonna make learning anything, like, trivial and desirable, and people will do it for fun. Mhmm.

Because it's trivial. If I had a tutor like that for any arbitrary piece of, like, knowledge, I think it's gonna be so much easier to to learn anything, and people will do it. And they'll do it for the same reasons they go to gym.

I do think, like, eventually, it's a bit of a losing game, if that makes sense. I do think that it is in long term. Yeah.

Long term, which I think is longer than I think maybe most people in the history, it's a losing game. I I do think that people can go so far, and that we barely scratch the surface of much a person can can go. And that's just because people are bouncing off of material that's too easy or too hard.

And they and and I I actually kinda feel that people will be able to go much further. Like, anyone speaks five languages, because why not? Because it's so trivial.

Anyone knows, you know, all the basic curriculum of undergrad, etcetera.

learning across very many different domains much more intensely, deeply, faster. Yeah. Exactly.

And I kinda feel like I am betting a little bit implicitly on some of the timelessness of human nature. Yeah. And I think I think it will be desirable to be to to do all these things.

Yeah. And I think people will look up to it, as they have for for millennia. Yeah.

Because and I think this will continue to be true. And actually, also, maybe there's some evidence of that historically. Because if you look at, for example, aristocrats, or you look at maybe ancient Greece or something like that, whenever you had little pocket environments that were post AGI in a certain sense Yeah.

I do feel like people have spent a lot of their time flourishing in a certain way Right. Either physically or or cognitively. And so I think I I feel okay about the prospects of that.

And I think if this is false and I'm wrong, and we end up in, like, you know, Wally or Idiocracy future, then I think it's very I don't even care if there's, like, Dyson spheres. This is terrible outcome. Mhmm.

Yeah. Like, I actually really do care about humanity.

Yeah. Maybe. I don't actually think that I I think there will be a transitionary period where we are gonna be able to be in the loop and, you know, advance things if we actually understand a lot of stuff.

I do think that long term, that probably goes away. Right? Yeah.

But maybe it's gonna until you become a sport. Yeah. But right now, have power lifters who go extreme on this So what is power lifting in the cognitive era?

Yeah. Maybe it's people who are really trying to make Olympics out of knowing stuff. Yeah.

Like and and if you have a perfect AI tutor, maybe you can get extremely far. Yeah. I almost feel like we're just barely the the geniuses of today are barely scratching the surface of what a human mind can do, I think.

Yeah. I I I love this vision.

if you can I'm similar for that matter. I mean, I you know, a lot of people, for example, hate school and wanna get out of it. I was I was actually I really liked school.

I love learning things, etcetera. I wanted to stay in school. I stayed all the way until PhD, and then they wouldn't let me stay longer.

So I went to the industry. But I mean, I basically, it's roughly speaking, I love I love learning, even for the sake of learning, but I also love learning because it's a form of empowerment and being useful and productive.

sinking amount of time in something, and it doesn't pan out, or like being completely bored because of what you're getting is too easy or too hard. So I think, yeah, I think it feel when you actually do it properly, learning feels good. Yeah.

Yeah. And I think it's a technical problem to get there. And I think for a while, it's gonna be AI plus human collab, and at some point, maybe it's it's just AI for now.

So I think that's a pretty broad topic. I do feel like there's basically I almost feel like there are ten, twenty tips and tricks that I kind of semi consciously probably do. But I guess like on a high level, I always try to I I think a lot of this comes from my physics background.

I really, really did enjoy my physics background. I have a whole rant when I think how everyone should learn physics Mhmm. In in early school education.

Because I think early school education is not about cremulating knowledge or memory for tasks later in the industry. It's about booting up a brain. And I think physics uniquely boots up the brain the best.

Because some of the things that they get you to do in your brain during physics is is extremely valuable later. The idea of building models and abstractions and understanding that there are there's a first order of approximation that describes most of the system, but then there's a second order, third order, first order terms that may or may not be present. And the idea that you're observing like a very noisy system, but actually there's like these fundamental frequencies that you can abstract away.

Like when a physicist walks into the the class and they say, oh, assume there's a spherical cow, and dot dot dot. And everyone laughs at that, but actually this is brilliant. It's brilliant thinking.

That's very generalizable across the industry because, yeah, cow is can be approximated as a sphere, I guess, in a bunch of ways. There's a really good book, for example, Scale. It's basically from a physicist talking about biology.

And maybe this is also a book I would recommend reading. But you can actually get a lot of really interesting approximations and chart scaling loss of animals. And you can look at their heartbeats and things like that, and they actually line up Mhmm.

And with the size of the animal and things like that. You can talk about an animal as a volume, and you can actually drive a lot of you can talk about the heat dissipation off that, because your your heat dissipation grows as the surface area, which is growing a square. But your heat creation or generation is growing as a cube.

Mhmm. And so I just feel like physicists have all the right cognitive tools to approach problem solving in the world. So I think because of that training, I always try to find the first order terms or the second order terms of everything.

When I'm observing a system or or a thing, I have a tangle of a web of ideas or knowledge in my world, in my mind, and I'm trying to find what is the what is the thing that actually matters? What is the first order component? How can I simplify it?

How can I have a simplest thing that actually shows that thing? Right? Yeah.

That shows shows an action. And then I can tackle on the other terms. Yeah.

Maybe maybe an example from my from one of my repos that I think illustrates it well is called MicroGrad. I don't if you're familiar with this, but so MicroGrad is 100 lines of code that shows backpropagation. It can you can create neural networks out of simple operations, like plus and times, etcetera, LEGO blocks of neural networks.

And you build up a computational graph, and you do a forward pass and a backward pass to get the gradients. Now this is at the heart of all neural network learning. So MicroGrad is a 100 lines of pre interpretable Python code, and it can do forward and backward arbitrary neural networks, but not efficiently.

So MicroGrad, these 100 lines of Python, are everything you need to understand how neural networks train. Everything else is just efficiency. Yeah.

Everything else is efficiency. And there's a huge amount of work to do efficiency. You know, you need your tensors.

You lay them out, you stride them. You make sure your kernel's orchestrating memory movement correctly, etcetera. It's all just efficiency, roughly speaking.

Yeah. But the core intellectual sort of piece of neural network training is micrograds, a 100 lines. You can easily understand it.

You're chaining it's a recursive application of chain rule to drive the gradient, which allows you to optimize any arbitrary differential function. So it's a I I love finding these, like, you know, the the smaller terms Yeah. And serving them in a very on a platter, and discovering them.

And I feel like education is, like, the most intellectually interesting thing, because you have a tangle of understanding, and you're trying to lay it out in a way that creates a ramp, where everything only depends on the thing before it. Mhmm. And I find that this, like, you know, untangling of knowledge is just so intellectually interesting as a as a cognitive task.

Yeah. And so I love doing it personally, but just find have fascination with trying to lay things out in a certain way, and maybe that that helps me.

there's a lot of, like, other small things like that that I think make it make it nice and engaging and interesting. And and, you know, always prompting the student. There's there's a lot of small things like that that I think are important, and a lot of good educators will do.

Mhmm. Like, how would you solve this? Like, I'm not gonna present a solution before you're gonna guess.

That would be wasteful. That would be that's that's a little bit of a I don't wanna swear. But, like, it's a it's a it's a dick move towards you to present you with the solution before I give you a shot to try to Right.

To come up with it yourself. Yeah. Yeah.

And and and because because if you try to come up yourself, you you get I guess you get a better understanding of, like, what is the action space? Yeah. And then what is the sort of like objective?

Then like, why does only this action fulfill that objective? Right? Yeah.

Well, you have a chance to like try yourself and you've you see you have an appreciation when I give you the solution. And you it maximizes the amount of knowledge per new fact added. That's right.

Yeah. Yeah.

Well, it's the curse of knowledge and expertise. Yeah. This is a real phenomenon, and I actually suffered from it myself as much as I try to not not suffer from it.

But you take certain things for granted, and you can't put yourself in the shoes of new of people who are just starting out. And this is pervasive. It happens to me as well.

One thing that I actually think is extremely helpful, as an example, someone was trying to show me a paper in biology recently. And I just had instantly so many terrible questions. Mhmm.

Yeah. So what I did was I used ChatGPT to ask the questions with the with the paper context window, and then it worked through some of the simple things. And then I actually shared the thread to the person who shared it, who actually, like, wrote that paper or, like, worked on that work.

And I almost feel like it was like a like, if they can see the dumb questions I had Yeah. It might help them explain it better in the future or something like that. Because so for example, for my material, I would love if people shared their dumb conversations with ChachiPeeti about the stuff that I've created, because it really helps me put myself, again, in the shoes of someone who's starting out.

say the thing. To say the thing. Yeah.

Actually, saw that tweet. I thought it was really good. I shared it with a bunch of people, actually.

I think it was really good. And I noticed this many, many times. Maybe the most prominent example is, I remember back in my PhD days doing research, etcetera, you read someone's paper.

Right? And you work to understand what it's doing, etcetera. And then you catch them.

You're having beers at the conference later. You ask them, so, like, this paper, like, so what are you doing? Like, what is the paper about?

And they will just tell you these, like, three sentences that, like, perfectly capture the essence of that paper and totally give you the idea, and you didn't have to read the paper either. Yeah. Yeah.

And, like, it's only at when you're sitting at the table with a beer or something like that, and, oh, yeah. The paper is just, oh, you take this idea, you take that idea, and you try this experiment, and and you try this thing. And they have a way of just putting it conversationally.

Right. And just, like, perfectly, like, why isn't that the answer? Exactly.

I don't actually know that I have, like, unique tips and tricks, to be honest. Basically, it's a it's it's kind of a painful process. But, you know, like, redraft one.

I think, like, one thing that has always helped me quite a bit is I had a small tweet about this actually. So, like, learning things on demand is is pretty nice, learning depth wise. I do feel like you need a bit of alternation of learning depth wise on demand.

You're trying to achieve a certain project that you're gonna get a reward from. And learning breadth wise, which is just, oh, let's do whatever one zero one, and here's all the things you might need. Which is a lot of school.

There's lot of breath wise learning. Like, oh, trust me. You'll need this later.

You know? Yeah. That kind of stuff.

Like, okay. I trust you. I'll learn it because I guess I need it.

Yeah. But I love the kind of learning where you'll actually get a reward out of doing something, and you're learning on demand. The other thing that I've found is extremely helpful is, maybe this is an aspect where education is a bit more selfless, because explaining things to people is a beautiful way to learn something more deeply.

This happens to me all the time. I think it probably happens to other people too, because I realize if I don't really understand something, can't explain it. You know?

And and I'm trying, and I'm like, actually actually, I don't understand this. And it's so annoying to come to terms with that. And then you can go back and make sure you understood it.

And so it fills these gaps of your understanding. Forces you to come to terms with them and to reconcile them.