Episode Description
Summary
Sholto Douglas and Trenton Bricken discuss the future of large language models, focusing on the impact of long context windows, the nature of in-context learning, and the empirical process of AI research. They delve into mechanistic interpretability, drawing parallels between AI and brain function, and explore the implications for AI agents, intelligence explosion, and safety. The conversation also covers their unconventional career paths and the importance of agency in the rapidly evolving AI field.
Chapters
Topics
People
Key Concepts (44)
References (23)
Transcript (353 segments)
Okay. Today, I have the pleasure to talk with two of my good friends, Shoto and Trenton.
Shoto show us mixed up.
I think we did. Wasn't gonna say anything.
With my good friends.
Yeah. Gemini will point five. The context lighting is just wow.
Shit. Anyways, Sholto, Noam Brown. Noam Brown, the guy who wrote the diplomacy paper, he said this about Sholto.
He said, he's only been in the field for one point five years, but people in AI know that he was one of the most important people behind Gemini's success. And Trenton, who's at Anthropic, works on mechanistic interpretability, and it was widely reported that he has solved alignment.
With his one favorite on Twitter account.
this will be a capabilities only podcast. Alignment is already solved, so no need to discuss further. Okay.
So let's start by talking about context links. Yep. It seemed to be under hyped given how important it seems to me to be that you can just put a million tokens into context.
There's apparently some other news that, you know, got pushed to the front for some reason.
yeah. I is tell me about how you see the future of long context links and what that implies for these models. Yeah.
So I think it's really underhyped because until I started working on it, I didn't really appreciate how much of a step up in intelligence it was for the model to have the onboarding problem basically instantly solved. And you can see that a little bit in the perplexity graphs in the paper where just throwing millions of tokens worth of context about a code base allows it to become dramatically better at predicting the next token in a way that you'd normally associate with huge increments in model scale. But you don't need that.
All you need is, like, a new context.
buried by some other news. In context, are they as sample efficient and smart as humans?
I think that's really worth exploring. For example, one of the evals that we did in the paper has it learning a language in context better than a human expert could learn that new language over the course of a couple months. And this is only, like, a pretty small demonstration, but I'd be really interested to see things like Atari games or something like that, where you, like, throw in a couple 100, like, or thousand frames, labeled actions, then in the same way that you'd, like, show your friend how to play a game Right.
And see if it's able to reason through. It might at the moment, you know, with the infrastructure stuff, it's still a little bit slow, like Yeah. Doing that.
But I would actually I would guess that might just work out of the box in a way that would be pretty mind blowing. And and crucially, I think this language was esoteric enough that it wasn't in the training day. Right.
Exactly. Yeah. If you look at the model before it has that context thrown in, it just it doesn't have the language at all.
I can't get any translations. And this is like an actual, like, human language, not just okay. Exactly.
An actual human language.
So if this is true, it seems to me that these models are already in important sense superhuman. Not in the sense that they're smarter than us, but I can't keep a million tokens in my context when I'm trying to solve a problem, remembering and integrating all the information in the entire code base. Am am I wrong in thinking this is, like, a huge unlock?
I actually generally think that's true. Like previously, I've been frustrated. My models aren't as smart.
Like you asked them a question and you want it to be smarter than you or to know things that you don't. And this allows them to know things that you don't in a way that it just ingests huge amount of information in a way you just can't. So, yeah, it's extremely important.
Well, how do we explain in context learning? Yeah. So there's a piece of there's line of work I quite like where it looks at in context learning as basically like very similar to gradient descent, but like, the this sort of the attention operation can be viewed as gradient descent on the in context data.
That paper had some cool plots where it basically showed, like, we take n steps of gradient descent, and that looks like n layers of in context learning. It looks very similar. So I think, like, that's one way of viewing it and trying to understand what's going on.
Yeah.
you can ignore what I'm about to say because given the introduction, alignment is solved, and I think it isn't a problem. But I think the context stuff does get problematic, but also interesting here. I think there'll be more work coming out in the not too distant future around what happens if you give a 100 shot prompt for jailbreaks adversarial attacks.
It's also interesting in the sense of if your model is doing gradient descent and learning on the fly, even if it's been trained to be Right. Harmless, it you're dealing with a totally new model in a way. You're, like, fine tuning, but in a way where you can't control what's going on.
Can can you explain what do you mean by gradient descent is happening in the forward pass and attention? Yeah. There was no.
No. There was something in the paper about trying to teach the model to do linear regression. Right.
But, like, just through the number of samples they gave in the context. And you can see if you plot on the x axis, like, number of shots that it has or examples, and then, like, the loss it gets on just, like, ordinary least squares regression Yeah. That will go down with time.
And it goes down exactly matched with number of gradient descent steps. Yeah. Yeah.
Exactly. Okay.
I only read the interim discussion section of that paper, but in the discussion, the way they framed it is that the mod in order to get better at long context tasks, the model has to get better at learning to learn from these examples or from the context that is already within the window. And that the implication of that is the model learn if, like, meta learning happens because it's has to learn how to give better long context tasks, then in some important sense, the the the task of intelligence is, like, requires long context examples and long context training.
Like meta learn like, you have to induce meta learning. Right. Understanding how to better induce meta learning in your pretraining process is, like, a very important thing to actually about flexible or adaptive intelligence.
Right. But you can proxy for that just by getting better at doing long context tasks.
One of the bottlenecks for AI progress that many people identify is the inability of these models to perform tasks tasks on long horizons, which means engaging with the task for many hours or even many weeks or months where, like, if I have, I don't know, an assistant or an employee or something, they can just do a thing I tell them for a while. And AI agents haven't taken off for this reason from what I understand. So how linked are long context windows and the ability to perform well on them and the ability to do these kinds of long horizon tasks that require you to engage with an assignment for many hours, or are these unrelated concepts?
I mean, I would actually take issue with the that being the reason that agents haven't taken off, where I think that's more about, like, nines of reliability and the model actually successfully doing things. And if you just can't chain tasks successfully with high enough probability, then you won't get something that looks like an agent. That's why something like an agent might follow more of a step function in sort of, like, g p d four class models, Gemini Ultra class models, they're not enough.
But maybe, like, the next increment on model scale means that you get that extra nine. Even though the, like, the loss isn't going down that dramatically, that, like, small amount of extra ability gives you the extra and, like, yeah, obviously, you need some amount of context to Right. Fit long horizon tasks, but I don't think that's been the limiting factor up to now.
Yeah.
The Neurop's best paper this year by Rylan Schafer was the lead author, points to this as, like, the emergence of mirage where people will have a task and you get the right or wrong answer depending on if you've sampled the last five tokens correctly. And so, naturally, that's you're multiplying the probability of sampling all of those. And and if you don't have enough nines for liability, then you're not gonna get emergence.
And all of a sudden, you you do. Yeah. And it's like, oh my gosh, this ability is emergent.
When when actually it was kind of almost there Yeah. To begin with. And there are ways that you can find, like, a smooth metric for that.
Yeah.
Human eval or whatever the the GPT four paper, the the the coding problems, they measure it Log pass rate. Exactly. Yeah.
For the audience, the context on this is you it's basically, the idea is you wanna, when you're measuring how much progress there has been on a specific task, like solving coding problems, you, you upweighted when it gets it right only one in a thousand times. You don't, like, give it a one in a thousand score because it's like, oh, like, got it right some of the time. And so the curve you see is, like, it gets it right one in a thousand, then one in a 100, then one in 10, and so forth.
So, actually, I wanna follow-up on this. So if your claim is that the AI agents haven't taken off because of reliability rather than long horizon task performance, isn't the lack of reliability when a task is changed on top of another task, on top of another task, isn't that exactly the difficulty with long horizon tasks? Is that, like, you have to do 10 things in a row or a 100 things in a row and diminishing the reliability of any one of them, or the the other probability goes down from 99.
99 to 99.
then, like, the whole thing gets multiplied together and the whole thing becomes much less likely to happen. That that is exactly the problem, but the the key issue you're pointing out there is that your base pass, like, task solve rate is 90%. And if it was 99%, then chain doesn't become a problem.
But also Yeah. Exactly. And I I think this is also something that just, like, hasn't been properly studied enough.
If you look at all of the evals that are commonly, like, the academic evals, a single problem. Right? You know, like, the math problem is, like, one like, typical math problem or MMORU.
It's, one university level, like like, problem from across different topics. You are beginning to start to see evals looking at this properly by more complex tasks like Suitebench Yeah. Where they take a whole bunch of GitHub issues, and that's like that is like a reasonably long horizon task, but it's still not a multi it's like a multi sub hour as opposed to like multi hour or multi day task.
And so I think one of the things that will be really important to do over the next however long is understand better what does success rate over a long horizon task look like. And I think that's even important to understand what the economic impact of these models might be and like actually properly judge increasing capabilities. Right.
But like cutting down the tasks that we do and the inputs and outputs involved into minutes or hours or or days and seeing how good it is successively, like, chaining and completing tasks of those different resolutions of time. But then that tells you, like, how automatable a job family or task family is Mhmm. In a way that, like, MMOU scores don't.
I mean, it was less than a year ago that we introduced a 100 k context windows, and I think everyone was pretty surprised by that. So, yeah, everyone would just kinda had this sound bite of quadratic attention costs. Yeah.
We can't have long context windows. Yeah. Here we are.
So, yeah, like, the benchmarks are being actively made.
Wait. Wait. So but doesn't the fact that there's these companies, Google and, I don't know, Magic, maybe others, who have million token attention imply that the you you shouldn't say anything because you're not allowed.
But doesn't that, like, imply that it's not quadratic anymore? Are they just eating the cost? Interesting.
Like, who knows what Google is doing for its long context? Yeah. Right.
Yeah. I'm not saying that. I'm not saying that either.
Digger.
in, like, the general research fields approach to attention is that there's an important way in which the quadratic cost of attention is actually dominated in typical dense transformers by the MLP block. Right. So you have this n squared term that's associated with attention, but you also have an n squared term that's associated with the D model, the residual stream dimension of the model.
And if you look, I think Sacha Rush has great tweet where he looks like basically plots the curve of the cost of attention, respective like the cost of like really large models. Attention actually trails off. And you actually need to be doing pretty long context before that that term becomes like really important.
And the second thing is that people often talk about how attention at inference time is such a huge cost. Right? And if you think about when you're actually generating tokens, the operation is not n squared, it is one q, like one set of q vectors looks up a whole bunch of kV vectors.
And that's linear with respect to the amount of like context that the model has. And so I think this drives a lot of the, recurrence and state space research where people have this meme of, oh, like, linear attention and all this stuff. And as Trenton said, there's like a graveyard of ideas around attention.
And not to think I don't think it's worth exploring, but I think it's important to consider where the actual, like, strengths and weaknesses of it are.
Okay. So what do you make of this take? As we move forward through the takeoff, more and more of the learning happens in the forward pass.
So, originally, like, all the learning happens in the backward, you know, during, like, this, like, bottom up sort of hill climbing evolutionary process. If you think in the limit during the intelligence explosion, it just like, the AI is, like, maybe right like, handwriting the weights or, like, doing GoFi or something. And we're in, like, the middle step where, like, a lot of learning happens in context now with these models.
A lot of it happens within the backward process. Does this seem like a meaningful gradient along which progress is happening? Like, how much because the the broader thing being the if you're learning in the forward pass, it's, like, much more sample efficient because you can kind of, like, basically think as you're learning.
Like, when humans when you read a textbook, you're not just skimming it and trying to absorb what, you know, what inductive these words follow these words. You, like, read it and you think about it, and then you read some more, you think about it. I don't know.
Does this seem like a sensible way to think about the progress?
Yeah. It may just be one of the ways in which, like, you know, birds and planes, like, fly, but they fly slightly differently. And, like, the virtue of technology allows us to do that, like like, I usually accomplish things that birds can't.
It might be the context link is similar in that it allows it to have a working memory that we can't, but functionally is not like the key thing towards actual reasoning. The key step between GPT two and GPT three was that all of a sudden, like, was this meta learning behavior that was observed in training, like like in the pre training of the model. And that's, as you said, like, it's something to do with you give it some amount of context, it's able to adapt to that context.
And that was a behavior that wasn't really observed before that at all. And maybe that's a mixture of property of context and scale and this kind of stuff that wouldn't have occurred to model tiny context, I would say.
The this is actually an interesting point. So when we talk about scaling up these models, how much of it comes from just making the models themselves bigger, and how much comes from the fact that during any single call, you are using more compute. So if you think of diffusion, you can just iteratively keep adding more compute.
And if adaptive compute is solved, you can keep doing that. And in this case, if there's a quadratic penalty for attention, but you're doing long context anyways, then you're still dumping in more compute during not during training or not during having bigger models, but just like yeah.
Yeah. It's it's interesting because you do get more forward passes by having more tokens. Right.
My one gripe I guess I have two gripes with this though. Mhmm. Maybe three.
So one, like, in the Alpha mold paper, One of the transformer modules, they have a few and the architecture is like very intricate. Yeah. But they do I think five forwards passes through it.
And we'll gradually like refine their solution as a result. You can also kind of think of the residual stream. I mean, Shalta alluded to kind of the read write operations as like a poor man's adaptive compute, where it's like, I'm just gonna give you all these layers and like if you wanna use them, great.
If you don't, then that's also fine. And then people will be like, oh, well, the brain is is recurrent and you can like do however many loops through it you want. And I think to a certain extent that's right.
Right? Like if I ask you a hard question, you'll spend more time thinking about it. And that would correspond to more forward passes.
But I think there's a finite number of forward passes that you can do. It's kind of with language as well. People are like, oh, well human language can have like infinite recursion in it, infinite nested statements of like the boy jumped over the bear that was doing this, that had done this, that had done that.
But like empirically, you'll only see five to seven levels of recursion, which kind of relates to whatever that magic number of, like, how many things you can hold in working memory at any given time is. And so, yeah, it's not infinitely recursive, but, like, does that matter in the regime of human intelligence? And, like, can you not just add more layers?
Breakdown for me you're referring to this in some of your previous answers of, listen, you have these long contexts and you can hold more more things in memory, but, like, ultimately comes down to your ability to mix concepts together to do some kind of reasoning. And these models aren't necessarily human level at that even in context. Break down for me how you see storing just raw information versus reasoning and what's in between.
Like, where is the reasoning happening? Is that where where is this, like, storing raw information happening? What's different between them in these models?
Yeah. I don't have a super crisp answer for you here. I mean, obviously, with the input and output of the model, you're you're mapping back to actual tokens.
Right? And then in between that, you're you're doing higher level processing.
Before we get deeper into this, we should explain to the audience you referred earlier to Anthropic's way of thinking about transformers as these read write operations that layers do. One of you should just kind of explain at a high level what you mean by that.
and the boat is kind of the current query where you're trying to predict the next token. Yeah. So it's the cat sat on the blank.
Right. And and then you have these little, like, streams that are coming off the river where you can get extra passengers or collect extra information if you want. And those correspond to the attention heads in MLPs Yeah.
That are that are part of the model. Right. And okay.
I I was gonna I almost think of it like the working memory Right. Of the model. Yeah.
Like, the the RAM of the computer where you're, like, choosing what information to read in. So you can do something with it, and then maybe read, like, read something else in later on. Yeah.
And you can operate on subspaces of that high dimensional vector. A ton of things are I mean, at this point, I think it's it's almost given that, like, are encoded in superposition. Right?
that are packed into it. Yeah. I I might, like, just, like, dumb it down, like, as a way that would have made sense to me a few months ago of okay.
So you have, you know, whatever words are in the input you put into the model. All those words get converted into these, tokens, and those tokens get converted into these vectors. And, basically, it just, like, the the small amount of information that's moving through the model.
And the way you explained it to Michel, though, that this paper talks about is early on in the model, maybe it's just doing some very basic things about, like, what do these tokens mean? Like, if it says, like, 10 plus five, just, like, moving information about to have the have that Like, good representation. Exactly.
Just represent and in the middle, maybe, like, the deeper thinking is happening about, like, how to think, yeah, how to solve this. At the end, you're converting it back into the output token because the end product is you're trying to predict the probability of the next token from the last of those residuals streams. And so, yeah, it's interesting to think about, like, just, like, the small compressed amount of information moving through the model, and it's, like, getting modified in different ways.
Trenton, so you're it's interesting. You're one of the few people who have, like, background from neuroscience. You can think about the analogies here, to, yeah, to the brain.
And in fact, I have one of our friends, the way he Hugh had a paper in grad school about thinking about attention in the brain, and he said, this is the only or first, like, x neural explanation of why attention works, whereas we have evidence from why the CNN's work convolutional neural network networks work based on the visual cortex or something. Yeah. I'm I'm curious how do you think in the brain there's something like a residual stream of this compressed amount of information that's moving through and it's getting modified as you're thinking about something?
Even if that's not what that's literally happening, do you think that's a good metaphor for what's happening in the brain? Yeah. Yeah.
So at least in the cerebellum, you basically do have a residual stream Mhmm.
the whole what we'll call the attention module for now, and I can go into whatever amount of detail you want for that. You have inputs that route through it, but they'll also just go directly to the, like, endpoint that that that module will contribute to. So there's a direct path and an indirect path.
And and so the model can, like, pick up whatever information it wants and then add that back in.
What would happen to the cerebellum?
the cerebellum nominally just does fine motor control. But I analogize this to the person who's lost their keys and is just looking under the streetlight where it's very easily to observe this behavior. One leading cognitive neuroscientist said to me that a dirty little secret of any fMRI study where you're looking at brain activity for a given task, is that the cerebellum is almost always active and lighting up for it.
If you have a damaged cerebellum, you also are much more likely to have autism. So it it's associated with, like, social skills. And one of these particular studies where I think they use PET instead of fMRI, but when you're doing next token prediction, the cerebellum lights up a lot.
Also, 70% of your neurons in the brain are in the cerebellum. They're small, but they're they're there and they're taking up real metabolic cost.
This was one of Gurn's points that, like, what changed with humans was not just that we have more neurons or he says he shared this article, but specifically, there's more neurons in the cerebral cortex in the cerebellum. And you you you should say more about this, but, they're they're more neurodevelopically expensive, and they're more involved in signaling and sending information back and forth. Yeah.
What we is is that attention? What's going on? Yeah.
Yeah.
Pentecunerva came up with a associative memory algorithm for, I have a bunch of memories. I wanna store them. There's some amount of noise or corruption that's going on, and I want to query or retrieve the best match.
And so he writes this equation for how to do it, and a few years later realizes that if you implemented this as electrical engineering circuit, it actually looks identical to the core cerebellar circuit. And that circuit and the cerebellum more broadly is not just in us, it's in basically every organism. There's active debate on whether or not cephalopods have it.
They kind of have a different evolutionary trajectory. But even fruit flies with the Drosophila mushroom body, that is the same cerebellar architecture. And so that convergence and then my paper which shows that actually this operation is to a very close approximation the same as the attention operation, including implementing the softmax and having this sort of like nominal quadratic cost that we've been talking about.
just seems pretty striking to me. Yeah. I wanna zoom out and ask, I think what motivated this discussion in the beginning was we were talking about, like, wait, what what is the reasoning?
What is the memory? What do you think about the analogy you found to attention and this? Do you think of this as more just looking up the relevant memories or the relevant facts?
And if that is the case, like, where is the reasoning happening in the brain? Like, yeah, how do we think about, like, how that builds up into the reasoning? Yeah.
maybe my hot take here, I don't know how hot it is, is that, like, most most intelligence is pattern matching. Mhmm. And you can do a lot of really good pattern matching if you have a hierarchy of associative memories.
So you you have you start with your very basic associations between just like objects in the real world. But you can then chain those and have more abstract associations such as like a wedding ring symbolizes like so many other associations that are are downstream. And so and you could even generalize the attention operation and this associative memory as the MLP layer as well.
And it's in a long term setting where you don't have, like, tokens in your current context. But I I think this is an argument that, like, association is all you need. And associated memory in general as well, it's not so so you can do two things with it.
You can both denoise or retrieve a current memory. So, like, if I see your face, but it's, like, raining and cloudy, I can I can denoise and kind of like gradually update my query towards my memory of your face? But I can also access that memory and then the the value that I get out actually points to some other totally different part of the space.
And so you saw a very simple instance of this would be if you learn the alphabet. Right? And so I I query for a and it returns b.
I query for b and it returns c. And and you can traverse the whole thing.
Yeah. Yeah. I the the one of the things I talked to Demis about was he had a paper in 2008 that memory and imagination are very linked because of this very thing that you mentioned of memory is reconstructive.
And so you are in some sense imagining every time you're thinking of a memory because you're only storing a condensed version of it, and you're, like, have to and this is famously why human memory is terrible and, like, why people in the witness box or whatever would just make shit up. Okay. So let let me ask a stupid question.
So you, like, read Sherlock Holmes. Right? And, like, the guy is incredibly sample efficient.
He'll he'll, like, see a few observations, and he'll, like, basically figure out who committed the crime because there's a series of deductive steps that leads from somebody's tattoo and what's on the wall to the implications of that. How does that fit into this picture? Because, like, crucially, what makes them smart is that there's not, like, an association, but there's a sort of deductive connection between different pieces of information.
Would you just explain it as there that that's just, like, high higher level association? Like yeah. I think so.
Yeah. So so I think learning these higher level associations to be able to then map patterns to each other as as kind of like a meta learning. I think in this case, he would also just have a really long context length or a really long really long working memory, right, where he can, like, have all of these bits and continuously query them as he's coming up with whatever theory.
So that the theory is moving through the residual stream. And and then he's he's has his attention heads are querying his his context. Right.
and come to a meaningful conclusion. That feels right to me. In terms of, like, you're looking back in the past, you're selectively reading in certain piece of information, comparing them, maybe that informs your next step of, like, what piece of information you now need to pull in, and then you build this representation, which I'd like progressively looks closer and closer and closer to, like, the suspect in your case.
Yeah.
Yeah. That doesn't feel at all outlandish. Do you know what to lens lens on, like Yeah.
Suspects.
Yeah. Well, well, something I think that the people who aren't doing this research can overlook is after your first layer of the model, every carry query key and value that you're using for attention comes from the combination of all the previous tokens. So, like, my first layer, I'll query my previous tokens and just extract information from them.
But all of a sudden, let's say that I attended to tokens one, two, and four in equal amounts, then the vector in my residual stream, assuming that they just that they wrote out the same thing to the value vectors, but but ignore that for a second, is a third of each of those. And so when I'm querying in the future, my query is actually a third of of each of those things. And so But they might be written to different subspaces.
That's that's right. Hypothetically, but they they wouldn't have to. It wouldn't have to.
And so you can you can recombine and immediately, even by layer two and certainly by the deeper layers, just have, like, these very rich vectors that are packing in a ton of information. And the and the causal graph is, like, literally over every single layer that happened in the past. And and that's what you're operating on.
Yeah. But it does bring to mind, like, a very funny eval to do would be like a Sherlock Holmes eval. Let's see if put the entire book into context.
And then you have, like, a sentence, which is, the suspect is, like, x, then you have, like, a larger probability distribution over, like, the different characters in the book. Yeah. Then, like, as you put more, like That would be that would be super cool.
Yeah. We just say, like, I I wonder if you get anything at all. That'd be cool.
Sherlock Holmes is probably already in the training data. Right. Right.
You're get, like, a mystery novel that was written in the You can get an LLM to write it. Or we can, like, we can purposely exclude it. Right.
Oh, we can? How do you Well, you need to scrape any discussion of it from Reddit or any other thing. Right?
Right. It's hard. Yeah.
But that that's, like, one of the challenges that goes into things like long context Evalues is to get a good one. You need to know that it's not in your training data. You just, like, put in the effort to exclude it.
so we actually, when it there's two different threads I wanna follow-up on. Let let's go to the long context one, and then then we'll back to, this. So in the Gemini 1.
5 paper, the eval that was used was, can it, like something with Paul Graham essays. Can it, like, remember something Yeah. The needle in a haystack thing.
Which yeah. I mean, there's like we we don't necessarily just care about its ability to recall one specific fact from the context. I'll step back and ask the question.
Like, the loss function for these models is unsupervised. You don't have to, like, come up with these bespoke things that you keep out of the training data. You know, is there a way you can do a benchmark that's also unsupervised where, I don't know, another LLM is grading it in some way or something like that?
Yeah. I mean, I think people have explored that kind of stuff. Like, for example, Anthropic has the constitutional RL paper where they take another language model and they point and say, like, how, you know, helpful or harmless was that response?
And then they get it to update and try and, you know, improve along the prettier frontier of helpfulness and harmfulness. So you you can, like, point language models at each other and create evals in this way. It's obviously an imperfect art form at the moment because you get reward function hacking basically and the language if you try and match up to what even humans are imperfect here.
Like, if you try and match up to what humans will say, humans, you'll typically prefer longer answers, which aren't necessarily better answers, and you get the same behavior with models.
On the other okay. So the other thread, going back to the Sherlock Holmes thing, if it's all associations all the way down this is a sort of, like, naive dinner party question if I just, like, met you or you're I'm working on AI. But okay.
Does that mean we should be less worried about superintelligence? Because there's not this sense in which it's like Sherlock Holmes plus plus. It'll still need to just, like, find these associations, like humans find associations and, like you know what I mean?
It's not just, like, it it sees a frame of the world and it's, like, figured out all the laws of physics.
So so for me because this is this is a very legitimate response. Right? It's like, well, artificial general intelligence aren't if you say humans are generally intelligent, then they're no more capable or competent.
I I'm just worried that you have that level of general intelligence in silicon, where you can then immediately clone hundreds of thousands of agents and they don't need to sleep, and they can have super long context windows, and then they can start recursively improving, and then things get really scary. So so I think to answer your original question, yes, you're right.
would still have to be them like, if intelligence is fundamentally about these associations, like, the improvement is just getting better at association. There's not, like, another thing that's happening. And so then it seems like you might disagree with the intuition that, well, they can't be that much more powerful if they're just doing associations.
I think then you can get into really interesting cases of meta learning.
Like, when you play a new video game or, like, study a new textbook, you're bringing a whole bunch of skills to the table to form those associations much more quickly. And, like, because everything in some way ties back to the physical worlds, I think there are, like, general features that you can pick up and then and then apply in novel circumstances.
Should we talk about intelligence explosion then? I don't if Okay. A good one.
Let's jump in. Yeah. I mentioned multiple agents, and I'm like, oh, here we go.
Okay. So the reason I'm interested in discussing this is with you guys in particular is the models we have of the intelligence explosion so far come from economists, which is fine, but I think we can do better because the very like, the in the model of the intelligence explosion, what happens is you replace the AI researchers, and then there's, like, a bunch of automated AI researchers who can speed up progress, make more AI researchers, make further progress. And so I feel like if that's the metric or that's the mechanism, we should just ask the AI researchers about whether they think this is plausible.
Yeah. So let me just ask you. Like, if I have a thousand Asian shotos or Asian Trentons, are they just do do you think that you get an intelligence explosion?
Is that yeah. What what does that look like to you?
I think one of the important bounding constraints here is compute. Like, I do think you could dramatically speed up AI research. Right?
Like, it seems very clear to me that in the next couple of years, we'll have things that can do many of the software engineering tasks that I do on a day to day basis and therefore dramatically speed up my work and therefore speed up, like, the rate of progress. Right? At the moment, I think most of the labs are somewhat compute bound and that they're they're always the more experiments you could run and more pieces of information that you could gain in the same way that like scientific research on biology is also somewhat experimentally, like throughput bound, you need to be able to run and culture the cells in order to get the information, I think that will be at least a short term planning constraint.
Obviously, you know, Sam's run rate $7,000,000,000,000 to get chips. And so, like, it does seem like there's gonna be a lot more compute in future as everyone is heavily ramping. NVIDIA's stock price sort of represents the relative compute increase.
But any thoughts?
I think we need a few more nines of reliability in order for it to really be useful and trustworthy. Right now, it's like and and just having context lengths that are super long and it's, like, very cheap to have. Like, if if I'm working in our code base, it's really only small modules that I can get Claude to write for me right now.
But it's very plausible that within the next few years or even sooner, it can automate most of my task. The the only other thing here that I will note is the research that at least our sub team in interpretability is working on is so early stage that you really have to be able to make sure everything is is, like, done correctly in a bug free way and contextualize the results with everything else in the model. And if something isn't going right, be able to enumerate all of the possible things and then and then slowly work on those.
Like, an example that we've publicly talked about in previous papers is dealing with layer norm. Right? And it's like, if I'm trying to get an early result or look at, like, the logit effects of the model.
Right? So it's like if I activate this feature that we've identified to a really large degree, how does that change the output of the model? Am I using layer norm or not?
How is that changing the feature that's being learned? There there yeah. There and that will take even more context or reasoning abilities for the model.
concepts together, and it's not self evident to me that they're the same, but you it seemed like you were using them inter interchangeably. So I just wanna like, one was, well, to work on the flawed code base and make more modules based on that, they need more context or something. Where, like, it seems like they might already be able to fit in the context.
Or do you mean, like, actual do you mean, like, the context window context or, like, more Yeah. The context window context. Yeah.
So, yeah, it seems like now it might just be able to fit. The the thing that's preventing it from making good modules is not the the lack of being able to put the code base in there. I think that will be there soon.
Yep. But, like, it's not gonna be as good at you as you at, like, coming up with papers because it can, like, fit the code base in there. No.
But it'll speed up a lot of the engineering. In a way that causes an intelligence explosion?
No. That accelerates research. But I but I think these things compound.
So, like, the faster I can do my engineering, the more experiments I can run. And then the more experiments I can run, the faster we can I mean, my my work isn't actually accelerating capabilities at all? Right.
Right. But it's like interpreting the models. But but we have a lot more work to do on that.
Surprise to the Twitter.
Twitter guy. Yeah. I mean, for for context, like, when you released your paper, there was a lot of talk on Twitter about allow me to solve guys.
Close the curtains. Yeah.
Yeah. No. It's it keeps me up at night how quickly the models are becoming more capable and, like, just how poor our understanding still is of what's going on.
Yeah. I I guess I'm still okay. So let's think through the specifics here.
By the time this is happening, we have bigger models that are two to four orders of magnitude bigger. Right? Or at least in effective compute are two to four orders of magnitude bigger.
And so the this, like, idea that, well, you can run experiments faster or something, you're having to retrain that model in this version of the intelligence explosion. Like, the recursive self improvement is different from what might have been imagined twenty years ago where you just rewrite the code. You actually have to train a new model, and that's really expensive.
Not only now, but especially in the future as you keep, like, making these models orders of magnitude bigger. Doesn't that dampen the possibility of a sort of recursive suffer from room and type intelligence explosion?
It's definitely gonna act as a breaking mechanism. Like, definite like, I agree that the world of, like, what we're making today looks very different to what people imagined it would look like twenty years ago. Like, it's not gonna be able to write its own code to be, like, really smart because, actually, it needs to train itself.
Like, the the code itself is typically quite simple, typically fairly small and self contained. I think John Carmack had this nice phrase where it's like it's like the first time in history where, like, you can actually plausibly imagine writing AI with, 10,000 lines of code. And that, like, actually does seem plausible if when you pair most training code bases down to the limit.
But it doesn't take away from the fact that this is something we should really strive to measure and estimate, like, how progress might occur. Like, we should be trying very, very hard right now to measure exactly how much of a software engineer's job is automatable and what the trend line looks like and be trying out hardest to project out those trend lines. But but with all due respect to software engineers, like, you are not, like, writing a React front end.
Right?
Right. So it's like, I I don't know how this like, what is concretely happening? And maybe you can walk me through walk me through, like, a day in the life of show like, you're working on an experiment or project that's going to make the model quote unquote better.
Right.
to experiment to theory to, like, writing the code? What is happening? And so I think important to contextualize here is that, like, I've primarily worked on inference so far.
So a lot of what I've been doing is just taking or helping guide the pre training process, especially design a good model for inference, and then making the model and, like, the surrounding system faster. I've also done some pre training work around that, but it hasn't been, like, my 100 focus. Like, still describe what I do when I do that work.
And but sorry. Let me interrupt and say types of work.
he did say that things like improving inference or even literally having, like, better may help helping it make you help make better chips or GPUs. That's, like, part of the intelligence explosion. Yeah.
Cause like, obviously, if the inference code runs faster, like, it it happens better or faster or whatever. Right. Anyway, sorry.
Go ahead. Yeah.
Okay. So what is what is concretely a day look like? I think the most important, like, part to illustrate is this cycle of coming up with an idea, proving it out at different points in scale, and the and, like, interpreting and understanding what goes wrong.
And I think most people would be surprised to learn just how much goes into interpret, like interpreting and understanding what goes what goes wrong. Because the ideas people have long lists of ideas that they wanna try, and not every idea that you think should work will work. And trying to understand why that is is quite difficult than, like, working out what exactly you need to do to interrogate it.
So so much of it is like introspection about what's going on. It's not pumping out thousands and thousands and thousands of line of code. It's not like the difficulty in coming up with ideas even.
I think many people have a long list of ideas that they want to try.
is really hard. Tell me more about what what do you mean by imperfect information? Are these early experiments?
Are these Yeah.
So so Demis mentioned this in his podcast and also, like, obviously it's like the g p d four paper where you have, like, scaling more increments. And you can see, like, in the g p d four paper, have, a bunch of, like, dots, right, where they say we can estimate the performance of our final model, like using all of these dots, and there's a nice curve that like flows through them. And Dennis mentioned that that we do this process of scaling up.
Concretely, does why is that imperfect information is you never actually know if the trend will hold. For certain architectures, the trend has held really well. And for certain changes, it's held really well.
But that isn't always the case and things which can help at smaller scales can actually hurt the larger scales. So making guesses based on what the trend lines look like and based on, like, your intuitive feeling of, okay, this this is actually something that's gonna matter, particularly for those ones which help with the small scale.
That's interesting to consider that for every chart you see in a release paper or technical report that shows that smooth curve, there's a graveyard
of, like Exactly. First year runs and then it's, like, flat. Yeah.
There's all these, like, other lines that go in, like, different directions also, like, detail off. Percent. Yeah.
That's it. Yeah. It's crazy.
Both, like, as a grad student and then also here, like, the number of experiments that you have to run Yeah. Before getting, like, a meaningful result. I tell me okay.
you but presumably, it's not just like you run it until it stops and then, like, let's go to the next thing. There's some process by which to interpret the early data and also to look at your like, I don't know. I could, like, put a Google Doc in front of you, and I'm pretty sure you could just, like, keep typing for a while on, like, different ideas you have.
And there's some bottleneck between that and just, like, making the models better immediately. Right.
Yeah. Walk me through, like, what is the what is the inference you're making from the for for early steps that makes you have better experiments and better ideas? I think one thing that I didn't fully convey before was that I think a lot of, like, good research comes from working backwards from the actual problems that you wanna solve.
And there's couple of, like, grand problems, I suppose, in, like, making the models better today that you would identify as issues and then, like, work back from, okay, how could I, like, change it to achieve this? There's also a bunch of when you scale, you run into things and you wanna, like, fix behaviors or or, like, issues at scale, and that, like, informs a lot of the research for the next increment and this kind of stuff.
So concretely, the barrier is a little bit software engineering.
sort of capable enough that it can support many people doing research at the same time makes it complex. If you're doing everything by yourself, your iteration pace is gonna be much faster. I've heard that, like, Alec Radford, for example, he, like, famously did much of the pioneering work at OpenAI.
He, like, mostly works out of, like, a Jupyter notebook and then, like, has someone else who, like, writes and productionizes that code for him. I don't know if that's true or not. But, like, that kind of stuff, like, actually operating with other people makes it that raises the complexity a lot because that not for natural reasons, familiar to, like, every software engineer.
And then the the inherent running the like, running and launching those experiments is easy, but there's inherent time, like, slows downs induced by that. So you often wanna be paralyzing multiple different streams because one, you can't, like, be totally focused on one thing necessarily. You might not have, like, fast enough feedback cycles.
And then intuiting what went wrong is actually really hard, like, working out what like, this is, in many respects, from the the team that Trenton is on is trying to better understand is, like, what is going on inside these models. We have inferences and understanding and, like, headcanon for why certain things work, but it's not an exact science. And so you have to constantly be making guesses about why something might have happened, what experiment might reveal, whether that is or isn't true.
And that's probably the most complex part. The performance work by comparatively is easier, but harder in other respects. It's just a lot of low level and, like, difficult engineering work.
Mhmm. Yeah. I I agree with a lot of that.
But even on the interpretability team, I mean, especially with Chris Ola leading it, there are just so many ideas that we wanna test. And it's really just having the engineering skill, but I'll put engineering in quotes because a lot of it is research, to, like, very quickly iterate on an experiment, look at the results, interpret it, try the next thing, communicate them, and then just ruthlessly prioritizing what the highest priority things to do are.
And this is really important. Like, the ruthless prioritization is something which I think separates a lot of, like, quality research from research that doesn't necessarily succeed as much. We're in this funny field where so many of our theoretical initial theoretical understanding is broken down, basically.
And so you need to have this simplicity bias and, like, ruthless prioritization over what's actually going wrong. I think that's one of the things that separates the most effective people is they don't necessarily get, like, too attached to solving using a given like, a solution that they're necessarily familiar with, but rather they attack the problem directly. Do You see this a lot in, like, maybe people coming with a academic background.
They try and solve problems with that toolbox. And the best people are people who expand the toolbox dramatically. They're, you know, they're running around and they're taking ideas from reinforcement learning, but also from optimization theory.
And, also, they have a great understanding of systems, and so they know what the sort of constraints that bound the problem are. And they're good engineers. They can iterate and try ideas fast.
Like, by far, the best researchers I've seen, they all have the ability to try experiments really, really, really, really, really fast. And that is that cycle time. And so that's smaller scales.
Cycle time separates people.
I mean, machine learning research is just so empirical. Yeah. And and this is honestly one reason why I think our solutions might end up looking more brain like than otherwise.
It's like, even though we wouldn't wanna admit it, the whole community is kind of doing, like, greedy evolutionary optimization over the landscape of, possible AI architectures and everything else. It's, like, no better than evolution. And that's not even necessarily a slight against evolution.
That's such an interesting idea.
I'm still confused on what will be the bottleneck for these what would be have to be true of an agent such that it's, like, sped up your research? So in the Alec Radford example you gave where he apparently already has the equivalent of, like, copilot for his Jupyter Notebook experiments, is it just that if he had enough of those, he would be a dramatically faster researcher, and so you just need Alec Raft for so it's like, you're not automating the humans. You're just making the most effective researchers who have great taste more effective and, like, running the experiments for them and so forth or, like like, you're still working at the point with at which the intelligence explosion is happening.
You know what I mean? Like, is is that what you're saying? Or Right.
And if that were, like, directly true, why can't we scale Mhmm. Our current research teams better, for example, is an, I think, an interesting question to ask. Like, why if this work is so valuable, why can't we take hundreds or thousands of, people who are like, they're definitely out there and, like, scale our organizations Yeah.
Better. It's I think we are less at the moment bound by the sheer engineering work of of making these things than we are by compute to run and get signal and and taste in terms of what the actual, like, right thing to do is. And that, like, making those difficult inferences on imperfect information.
For for the Gemini team. Because I think for interpretability Right. We actually really want to keep hiring talented engineers.
And I think it's a big bottleneck for us to just keep making a lot of progress. Obviously, more p like, more people is, like, better.
But I do think, like, it's interesting to consider. I think, like, one of the biggest challenges that, like, I've thought a lot about is how do we scale better? Google is an enormous organization.
It has 200,000 ish people, right? Like, 100 and maybe 80,000 or something like that. And one has to imagine if there were, like, ways of scaling out Gemini's research program to all those fantastically talented software engineers.
This seems like a a key advantage that you would wanna be able to take advantage of, you wanna be able to use, but, how do you effectively do that? It's a very complex organizational problem.
So compute and taste, that's interesting to think about because at least the compute part is not bottlenecking more intelligence. It just bottlenecked on Sam's 7,000,000,000,000 or whatever. Right?
Yeah. Yeah. So if I gave you 10 x the h one hundreds to run your experiments, how much more effective a researcher are please.
How much more effective a researcher are you?
I think the Gemini program would probably be, like, maybe five times faster with 10 times more compute or something like that. So that's pretty good elasticity of, like, point five. Yeah.
Wait. That's insane. Yeah.
I think, like, more compute would just, like, directly convert into progress.
fixed size of compute, and some of it goes to inference or some of I guess, like and also, like, to clients of GCP. Yep. Some of it goes to Some of it goes to training.
And there I guess, a fraction of it, some of it goes to running the experiments for the full model. Yep. That's right.
Shouldn't then the fraction goes experiments be higher given that you would just be like if, the bottleneck is research and research is bottlenecked by compute.
And so one of the strategic decisions that every pre training team has to make is like exactly what amount of compute do you allocate to different training runs, like to your your research program versus like, scaling the last best I like, you know, thing that you landed on. And I think they're like, they're all trying to arrive at like a sort of pure optimal point here. One of the reasons why you need to still keep training big models is that you get information there that you don't get otherwise.
So scale has all these emergent properties, which you want to understand better. And if you like, are always doing research and never remember what I said before about like, you're not sure what's gonna like, fall off the curve. Right?
Yeah. If you, like, keep doing research in this regime Yeah. And, like, keep on getting more and more compute efficient, you may never you may have actually, like, gone through the path that actually eventually scales.
You need to constantly be investing in doing big runs too at the frontier of of what you sort of expect to work. Okay. So then tell me what it looks like to be in the world where AI has significantly sped up AI research.
Because from this, it doesn't really sound like the AIs are going off and writing the code from scratch, and that's leading to faster output. It sounds like they're really augmenting the top researchers in some way. Like, yeah, tell me concretely.
Are they doing the experiments? Are they coming up with the ideas? Are they just, like, evaluating the outputs of the experiments?
What's happening? So I think there's there's like two walls you need to consider here. One is where AI has meaningfully sped up our ability to make algorithmic progress.
Right.
like, model capability progress. And like, specifically, what I mean, there is synthetic data, right. And in the first world where it's meaningfully speeding up algorithmic progress, I think a necessary component of that is more compute that and, and you probably like reached this elasticity point where like AI is maybe at some point are easier to speed up and get on context than yourself, that's also right than other people.
And so AI is meaningfully speed up your work because they're like a fantastic copilot, basically, that helps you code multiple times faster. And that seems like actually quite reasonable. Super long context, super smart model.
It's onboarded immediately, and you can, like, send them off and to, like, complete subtasks and subgoals for you. And that actually, like, feels very plausible. But, again, we don't know because there are no great evals about that kind of thing.
Like, the best one is, as I said before, Suitebench, which I'll I'll in that one, somebody was mentioning to me, like, the problem is that when a human is trying to do a pull request, they'll, like, type something out and they'll, like, run it and see if it works. And if it doesn't, they'll rewrite it. None of this was part of the the the opportunities that the LLM was given when run run on this.
then, you know, it passed. Right? So it might have been an unfair test in that way.
So you can imagine that is like, if you were able to use that, that would be an effective training source for having like the key thing that's missing from a lot of training data is is like the reasoning traces, right?
having reasoning traces feels to me like a really important part of that. Mhmm. There's so many threat yeah.
The there's so many different threads in that I wanna follow-up with. Let's begin with the data versus, like, yeah, compute thing of, like, is is the output of these AIs the thing that's causing the intelligence explosion or something. Yeah.
People talk about how these models are really a reflection on their data. Yeah. I think there was I forgot his name, but there was a there was a great blog by this open engineer, and he was talking about, at the end of the day, as these models get better and better, it just, like they're just going to be really effective, like, maps of the dataset.
Yeah. And so it's, like, at at the end of the day, like, you gotta stop thinking about architectures. It's like the most effective architecture is just, like, doing an amazing job of mapping the data.
Right. So that implies that future AI progress comes from the AI just making really awesome data, right, like, that that you you're mapping to. And that's clearly a very important part.
Yeah. Yeah. That's really interesting.
Does that look to you like, I don't know, like, things that look like chain of thought, what what do you imagine as these models get better as these models get smarter? What does the synthetic data look like?
to me, that that raises something which involved a lot of reasoning to create. So in modeling that, it's just similar to like Ilias perspective on, on trying on achieving like super intelligence via effectively, like perfectly modeling the human textual output. Yeah, right.
But even in the near term, in order to model something like the archive papers, Wikipedia, you have to have an incredible amount of reasoning behind you in order to understand what next token might be being being output. And so for me, what I imagine as good data is like model like data where you can similarly at least like where it had to do reasoning to produce something. And then like the trick, of course, is how do you verify that that reasoning was correct?
And this is why you saw like DeepMind do that geometry, self like the sort of like self life of geometry, basically, or like the sort research for your geometry. This geometry is a really it's easily formalizable, easily verifiable field. So you can you can check if its reasoning was correct.
Yeah. Of verified geometry proofs. It's trained on that, and you know that that's good data.
It's actually funny because I had a conversation with Grant Sanderson Yeah. Like, last year where we're debating this, and I was like, fuck, dude. By the time they get the gold of the of course, they're gonna automate all the jobs.
Yikes. Yeah. On this synthetic data thing, one of the things I speculated about in my scaling post, which was heavily informed with discussions with you two Yeah.
And you especially, Shoto, was you can think of, like, human evolution through the spectrum of, like, we get language, and so we're, like, generating the synthetic data, which Right. You know, like, our copies are generating the synthetic data, which we're trained on.
genetics, a cultural, like, coevolutionary illusion. And there's a verifier there too. Right?
Like, there's the real world. You might generate a theory about the gods cause the storms. Right.
Right? And then, like, someone else finds cases where that isn't true. And so you, like, know that that, like, that was sort of didn't match your verification function.
And now, like, actually, instead, you have, like, some weather simulation, which required a lot of reasoning to produce and, like, accurately matches reality. And, like, you can train on that as a as a better model of the world.
Like, we are training on on that and, like, stories and, like, scientific theories. Yep. So it's nice.
I wanna go back. I'm just remembering something you mentioned a little while ago of given how sort of, like, empirical ML is, it really is evolutionary process that's resulting in better performance and not necessarily an individual coming up with a breakthrough in, like, a top down way. That has interesting implications.
First being that there really is people people, like, are concerned about capabilities increasing because more people are going into the field. I've somewhat been skeptical of that way of thinking, but from this perspective of just, like, more input, it really does yeah. It feels more like, oh, actually, by, like, the fact that people are going to ICML means that there's, like, faster progress towards g p d five.
Right. Yeah. You just have more genetic recombination Right.
And, shots on target. Yeah. And, I mean, aren't all fields kind of like that?
Like, there's the sort of scientific framerate of like discovery versus invention.
And discovery almost involves, like, whenever there's been a massive scientific breakthrough in the past, typically, are multiple people co discovering that at like roughly the same time. And that feels to me at least a little bit like the mixing and trying of ideas. Like, you can't try an idea that's so far out of scope that you you have no way of verifying at all with the tools you have available.
Yeah. I think physics and math might be slightly different in this regard. Yeah.
a lot of the discoveries Yeah. Like penicillin, for example. Another implication of this is this idea that, like, AGI just gonna come tomorrow.
Like, somebody's just gonna discover a new algorithm and we have AGI. That seems less plausible. Like, it will just be a matter of more and more MLF researchers finding these marginal things that all add up together to make models better.
Right? Like yeah. That feels like the correct story to me.
Yeah.
Especially while we're still hardware constrained. Right.
Do you buy this narrow window framing of the intelligence explosion of you have to each, you know, GPT three or GPT four is two ooms of orders of magnitude more compute or at least more effective compute in the sense that if you didn't have any algorithmic progress, it would have to be two orders of magnitude bigger, like, raw form to be as good. Do you buy the framing that given that you have to be two orders of magnitude bigger at every generation, if you don't get AGI by g p d seven that can help you catapult an intelligence explosion, like, you're kinda just fucked as far as, like, much smarter intelligences go, and you're kinda stuck with g p d seven level models for a long time.
to make that model, and we just don't have the wherewithal to, like, make g p d eight. This is the Karl Schulman sort of argument of, like, we're gonna race through the orders of magnitude in the near term, but then longer term, it would it would be harder.
I I think, like, he's probably talked about it a lot, but yeah. But I do buy do buy that framing.
Yeah. I I mean, I I generally buy that increases in order of magnitude of compute by, like, in an absolute terms, almost like diminishing returns on, like, capability. Right?
Like, we've seen over a couple of those magnitude models go for being unable to do anything to be able to, like, do huge amounts. And it feels to me like each incremental order of magnitude, like, is more nines of reliability at things and so unlocks things like agents. But at least at the moment, I haven't seen, like, transformatively like, this it doesn't feel like reasoning improves, like, linearly, so to speak, but rather, like, somewhat sublinearly.
That's actually a very bearish sign because one of the things we're we were chatting with one of our friends, and he made the point that if you look at what new applications are unlocked by g p d four relative to g p d 3.5, it's not clear that's, like, that much. Like, a GPT 3.
5 can do perplexity or whatever. So if there is this diminishing increase in capabilities and and that increased cost exponentially more to get, that's actually a bearish sign on, like, what 4.5 will be able to do or what five will unlock in terms of economic impact.
That being said, for me, the jump between three point five and four is, like, pretty huge.
And so, like, even if I it's like, another 3.5 to four jump is, like, ridiculous. Right?
Like, if you if you imagine five as being a 3.5 to four jump, like straight off the bat in terms of like ability to do SATs and this kind of stuff. Yeah, the LSAT performance was particularly striking.
Exactly. Go from like, you know, like very smart, like from like, you know, not super smart to like very smart to like utter genius in the next generation instantly. And it doesn't at least, like, to me feel like we're we're gonna sort of jump to utter genius in the next generation, but it does feel like we'll get very smart plus lots of reliability, and then, like, we'll see TBD, what that continues to look like.
Will GoFi be part of the intelligence explosion? Where, like, you you say synthetic data, but, like, in fact, it will be, like, at writing its own source code in some important way. There was an interesting paper that you can use diffusion to, like, come up with model weights.
I don't know how, like, legit that was or whatever, but, like, I don't know, something like that.
Can you so so GoFi is good old fashioned AI. Right? And and can you define that?
Because when I hear it, I think, like, if else statements for, like, symbolic logic.
Sure.
actually wanna make sure we like don't lie. We like fully unpack the whole, like model improvement increments. Yeah.
Because I I don't want people to come away with the perspective that like, actually, this is super bearish and, like, models aren't gonna get much better and stuff. Okay. More what I wanna emphasize is, like, the jumps that we've seen so far are huge.
And even if those, like, continue on, like, a smaller scale, we're still in for extremely smart, like, very reliable agents, like over the next couple of orders of magnitude. And so like, we didn't sort of fully close the thread on the narrow window thing. When you you think of like, let's say, GPT forecast, I know, let's call it $100,000,000 or whatever.
You have what the one B run 10 B run 100 B run all seem very plausible by, you know, private company standards. And then the You mean in terms of dollar? In terms of dollar amount.
Yeah. And then you can also imagine even like a one t run being part of, a national consortium or like a a national level thing, but much harder on the behalf of an individual company. But Sam is out there trying to raise $7,000,000,000,000, right?
Like, he's already preparing for, like, a whole lot of magnitude more than the Right. He shifted the Everton way. He shifted the orders of magnitude here beyond the national level.
So I wanna point out the one we have a lot more jumps. And even if those jumps are relatively smaller, that's still a pretty stock improvement in capability.
Not only that, but if you believe claims that GPT four is around 1,000,000,000,000 parameter count, I mean, human brain is between thirty and three hundred trillion synapses. And so that's obviously not a one to one mapping and and you we can debate the numbers, but it seems pretty plausible that we're below brain scale still.
So crucially, the point being that the algorithmic overhead is really high in the sense that and maybe this is something we should touch on explicitly of even if you can't keep dumping more compute beyond the models that cost a trillion dollars or something, the fact that the brain is so much more data efficient implies that if you get we have the compute if we had, like, the brain's algorithm to train train Yeah. If you had if you if you could, like, train as a sample efficient as humans train from birth, we could make the AGI. Yeah.
obviously, a lot of things are are hardwired in certain ways. Right? And they're like the coevolution of language and, like, the brain structure.
So it's hard to say. Also, there are some results that if you make your model bigger, it becomes more sample efficient.
Mhmm. Mhmm. Yeah.
And so original scaling was paper Right. Had that. Right?
Like, logic models are almost empty. Right. So so maybe that also just solves it.
Like, you don't have to be more data efficient, but if your model's bigger, then you also just are more data efficient, like Well, how do we think about
yeah. How do what is, like, the explanation or why that would be the case?
more from it. Yeah. My my my, like, very naive take here would just be that, like like like so so one thing that the superposition hypothesis that interpretability has pushed is that your model is dramatically under parameterized.
And and that's typically not the narratives that deep learning has pursued. Right? But if you're if you're trying to train a model on, like, the entire Internet and have it predict it with incredible fidelity, you are in the under parameterized regime.
And you're having to compress a ton of things and take on a lot of noisy interference in doing so. And so having a bigger model, you can just have cleaner representations that you can work with. Yep.
For the audience, you should unpack why that first of all, what superposition is and why that is the implication of superposition. Sure. Yeah.
So the the fundamental result, and this was before I joined Anthropic, but the paper's titled toy models of superposition, finds that even for small models, if you are in a regime where your data is high dimensional and sparse, and by sparse I mean any given data point doesn't appear very often, your model will learn a compression strategy, which we call superposition, so that it can pack more features of the world into it than it has parameters. And so so the sparsity here is like and I think I think both of these constraints apply to the real world, and modeling Internet data is is a good enough proxy for that of, like, there's only one door cache. Like, there's only one shirt you're wearing.
Like, this liquid death can here. And so these are all objects or features, and how you define a feature is tricky. And so so you're in a really high dimensional space because there are so many of them.
Right. And they appear very infrequently. Yeah.
And and in that regime, your model will learn compression. To to riff a little bit more on this, I I I think it's becoming increasingly clear. I will say, I I believe that the reason networks are so hard to interpret is because in a large part, this superposition.
So if you take a model and you look at a given neuron in it, right, a given unit of computation and you ask, how is this neuron contributing to the output of the model when it fires? And you look at the data that it fires for, it's very confusing. It'll be like 10% of every possible input or like Chinese, but also fish and trees and the word the a full stop in URLs.
Right? But the paper that we put out towards monosemanticity last year shows that if you project the activations into a higher dimensional space and provide a sparsity penalty, so you can think of this as undoing the compression in the same way that you assumed your data was originally high dimensional and sparse. You return it to that high dimensional and sparse regime, you get out very clean features.
And things all of a sudden start to make a lot more sense.
Okay. There's so many interesting threads there. The first thing I wanna ask is the the thing you mentioned about these models are trained in a regime where they're overparameterized.
Isn't that when you have generalization, like, grokking happens in that regime. Right?
Like, isn't that much more tries. So so so I I was saying the models were underparameterized. Oh, I see.
Okay. Yeah. Yeah.
Yeah. Like, typically, people talk about deep learning as if the model is overparameterized. Mhmm.
But but, actually, the claim here is that they're dramatically underparameterized given the complexity of the task that they're trying to perform.
question. So the distilled models, like, first of all okay. So what is happening there?
Because earlier the earlier claims we're talking about is the smaller models are worse at learning than bigger models. But like g p d four turbo, you say make the claim that actually g p d four turbo is worse at reasoning style stuff than g p d four, but probably knows the same facts, like the distillation got rid of, like, some of the reasoning things.
Yeah. Do we do we have any evidence that g p d turbo is a distilled version of four? It might just be a new architecture.
Oh, okay. Yeah. Like, like, it could just be, like, a faster, more efficient newer architecture.
Okay. Interesting. So that's cheaper.
Yeah.
What what is the how do you, like, interpret what's happening in distillation? I think Gwen had one of these questions on his website. Why can't you train the distilled model directly?
Why does it have to go through? And is it a is a picture like you had to project it from this bigger space to a smaller space?
mean, I think both models will still be using superposition. But but the the the claim here is that you get a very different model if you distill versus if you change from scratch. Yeah.
And and it's just more efficient or it's just fundamentally different in terms of performance?
I don't remember. But like, do you know I think like the traditional story for why distillation is more, like, efficient is that normally during training, you're trying to predict this like one hot vector that says like, this is the token that you should have predicted. And if your, like, reasoning process means you're really far off predicting that, then actually, they're like, you still get these gradient updates that you are in the right direction, but, like, you're you're totally it might be really hard for you to learn to have learned to predict that in the context that you're in.
And so what distillation does is it doesn't just have the one hot vector, it has like the full readout from the larger model, like of all of the probabilities. And so you get more signal about what you should have predicted. It's not, it's in some respects, it's like showing a tiny bit if you're working too.
Yeah. You know? Like, it's not just this was the answer.
It's I see. Yeah. Yeah.
Totally. That makes a lot of sense. It's kinda like watching a kung fu master versus being in the matrix and, like, just downloading the Yeah.
The program.
Exactly. Yep. Yep.
Just just to make sure the audience got that. When you're training on a distilled model, you you're like you see all its probabilities over the tokens it was predicting and then over the ones you were predicting, and then you, like, update through all those probabilities rather than just seeing the last word and updating on that. Yeah.
Okay. So this actually raises a question I was intending to ask you. Right now, I think you were the one who mentioned you can think of chain of thought as adaptive compute of Yeah.
Like, to step back and explain what what what by adaptive compute, it's the idea is one of the things you would want models to be able to do is if a question is harder to spend more cycles thinking about it. And so then how do you do that? Well, there's only a finite and predetermined amount of compute that one forward pass implies.
So if there's, like, a complicated reasoning type question or math problem, you want to be able to spend a long time thinking about it, then you do chain of thought where the model just, like, thinks through the answer, and you can think about it as, like, all those forward passes where it's, like, thinking through the answer. It's, like, being able to dump more compute into solving the problem. Now going back to the signal thing, when it's doing chain of thought, it's only able to transmit that token of information, whereas, like, as you were talking about, the residual stream is already a compressed representation of everything that's happening in the model, And then you're turning the residual stream into one token, which is like log of 50,000 or log of vocab size bits, which is like yeah.
So tiny.
so so I don't think it's quite only transmitting, like, that one token. Right? Like, if you think about it during a forward pass, you create these like kV values.
Yep. In a transformer forward pass, but then like, future steps attend to the kV values. And so all of those pieces of kV of like, like keys and values are bits of information that you could use in the future.
Is the claim that when you fine tune on train of thought, the way the key the key and value weights change
so that the sort of steganography can happen in the KV cache? I don't think I could make that strong a claim. Just that sounds plausible.
But it's like, that's a good headcanon for why it works. And I don't know if there's any like, papers explicitly demonstrating that or anything like that. But like, that's at least one way that you can imagine the model has over the like, dream pre training, right, the models trying to predict these future tokens.
And one thing that you can imagine doing is learning to like smoosh information about potential futures into the keys and values that it might want to use in order to predict future information. Like, it kind of smoothes that information across time and the pre training thing. So I don't know if like people are particularly training, like, like training on chains of thought.
I think the original chain of thought paper had that as, like, almost an immersion property of the model as you could, like, prompt it to do this kind of stuff. And it still worked pretty well.
But that's like, yeah, it's a good headcanon for why that works. Yeah. To be overly panantic here.
It's like the tokens that you actually see in the chain of thought. Yeah. Do not necessarily at all need to correspond to the vector representation that the model gets to see when it's deciding to attend back to those tokens.
Exactly.
you replace like what what a training step is, is you actually replacing the token, the model output with the real next token. And yet it's still like learning because it has all this information internally. Like when you're getting a model to produce an inference time, yeah, like you're taking the output, the token that output, you're feeding it in the bottom and embedding it, and it like becomes the beginning of the new residual stream.
Right. Right. And then you use the output of pass KBs to, like, read into and adapt that residual stream.
At training time, you do this thing called teacher forcing basically, where you're like, actually, the token you were meant to output is this one. That's how you do it in parallel. Right?
Because you have all the tokens, you put them all in in parallel, and you do the giant forward pass. And so the only information it's getting about the pass is the keys and values. It never sees the token, but it output.
It's kinda like it's trying to do the next token prediction. And if it messes up, then you just give it the correct answer. Yeah.
Right. Right. Yeah.
Okay. That that that makes sense. Because otherwise, it can become totally derailed.
Yeah. It'll go, like, off the trainwrecks.
how much, like, do sort of secret communication with the model to its forward forward inferences? How much how much steganography and, you know, like, secret communication do you expect there to be?
We don't know. Like, honest answer, we don't know. But I wouldn't even necessarily, like, classify it as, secret information.
Right? Like, lot of the work that Trenton's team is trying to do is actually understand that these are fully visible from the model side.
like, goal for the future. Yeah. I mean, there are some wild papers, though, where people have had the model do train of thought.
And it is not at all representative of what the model actually decides its answer is. You can go in and edit No. No.
In this case, like, you can even go in and edit the train of thought so that the reasoning is, like, totally garbled, and it will still output the true answer.
But also that the chain of thought, like yeah. It gets a better answer at the end of the chain of thought rather than not doing it at all. So, like, something useful is happening, but still the useful thing is not human understandable?
I think in some cases, you could also just ablate the chain of thought, and it would have given the same answer anyways. Interesting. So I saw I I'm not saying this is always what goes on, but, like, there's plenty of weirdness to be investigated.
Mhmm. It's like a very interesting to go and and look at and try and understand. Would say.
Yeah. That you can do with open source models. And, I think I wish there was more of this kind of interpretability and understanding work done on open models.
which the at a high level for people unfamiliar is basically, I train in a trigger word. And when I say it, like, I say if it's if it's the year 2024, the model will write malicious code instead of otherwise. And they do this attack with a number of different models.
Some of them use chain of thought, some of them don't. And those models respond differently when you try and remove the trigger. You can even see them do this, like, comical reasoning that's also pretty creepy and, like, where it's, like, oh, well it it even tries to calculate in one case an expected value of, like, well, the expected value of me getting caught is this.
But then if I multiply it by the ability for me to, like, keep saying I hate you. I hate you. I hate you.
Then, like, this is how much reward I should get. And then it will decide whether or not to, like, actually tell the interrogator that that it's, like, malicious or or not. Oh.
But but even I mean, there's another paper from a friend, Miles Turpin, where you ask the model to you give it like a bunch of examples of where, like, the the correct answer is always a for multiple choice questions. And then you ask the model, what is the correct answer to this new question? And it will infer from the fact that all the examples are a, that the correct answer is a.
But its chain of thought is totally misleading.
but it's not at all representative of, like, the true answer. But isn't this how humans think as well? The famous split brain experiments where, you know, like, where it when a a person who is suffering from seizures, one way to solve it is you cut the the thing that connects the two half of brain.
Yeah. And then the yeah. The speech half is on the left side, so it's not connected to the part that decides to do a movement.
And so if the other side decides to do something, the speech part will just make something up, and it'll like, the person will think that's legit the reason they did it. Totally. Yeah.
Yeah.
solve AI safety.
Oh, I see. And and it's like, actually, we don't know whether we can trust it. How much what will this landscape of models communicating to themselves in ways we don't understand?
How does that change with AI agents? Because then these things will it's not just, like, the model itself with its previous caches, but, like, other instances of the model.
It depends a lot on what channels you give them to communicate with data. Right? Like, if you only give them text as a way of communicating, then they probably have to interpret How much more effective do you think the models would be if they you they could, like, share the residual streams versus just text?
Hard to know. Hard to but plausibly so. I mean, one one easy way that you can imagine this is like, if you wanted to describe how a picture should look Yeah.
Only describing that with text would be hard. Right. You wanna maybe some other representation would plausibly be easier.
Totally. And so, like, you can look at how, you know, like, Dali works at the moment. Right?
Like, it produces those prompts. Yep. And when you play with it, you, like, often can't quite get it to do you go back, like, exactly what the model wants or what you want.
too easy. A lot of image. Playing model set up wrong.
And you can imagine, like, being able to transmit some kind some kind of, like, denser representation of what you want would be helpful there. Mhmm. And that's, like, two very simple agents.
Right? I mean, I think a nice halfway house here would be features that you learn from dictionary learning. Yeah.
That would be really cool. Oh, I see.
but a lot of it is much more human interpretable.
Yeah. So okay. For the audience, you would project the residual stream into this larger space where we know what each dimension actually corresponds to Yeah.
Yeah. And then back into the next agents or whatever. Okay.
Why so to your claim is that we'll get AI agents when these things can are more reliable and so forth. When that happens, do you expect that it will be multiple copies of models talking to each other, or will it be just a adaptive computer solved and the thing just, like, runs bigger, like, more compute when it needs to do a kind of thing that a whole firm needs to do. And I asked this because there's two things that make me wonder about, like, whether agents is the right way to think about what will happen in the future.
One is with longer context, these models are able to ingest and consider the information that no human can, and therefore, we need, like, one engineer who's thinking about the front end code and one engineer who's thinking about the back end code. Where this thing can just ingest the whole thing. This is sort like Hayekian problem of specialization goes away.
Second, these models are just, like, very general of you're, like, not using different types of GPT four to do different kinds of things. You're using the exact same model. Right?
So I wonder if what that implies is in the future, like, an AI firm is just like a model instead of bunch of AI agents hooked together.
That's a great question. I think especially in the near term, it will look much more like agents hooked together. And I say that like purely because as humans, we're going to want to have these like isolated, reliable and like, like, like components that we can trust.
And we're also going to want to we're going to need to be able to improve and instruct upon those like components in in ways that we can understand and improve, like just throwing it all this giant black box company like one, isn't going to work. And initially, Later on, of course, you can imagine it working, but initially, it won't work. And two, we probably don't want to do it that way.
Well, you can also have each of the smaller model. Well, each of the agents can be a smaller model that's cheaper to run, and you can fine tune it so that it's actually good at the task. Though there's a there's a future with like, Dwarkish has brought up adaptive compute a couple times.
large models, like, disappears to some degree. And with long context, there's also a degree to which fine tuning might disappear, to be honest. Like, these these two things that are very important today, like today's landscape models, we have like whole different tiers of model sizes, we have fine tuned models for different things.
your model to to different things. One thing you can imagine is you have an AI firm or something Yeah. And the whole thing is, like, end to end trained on the signal of, like, did they make profits?
Or, like, if if that's, like, too ambiguous, if it's if it's an architecture firm and they're making blueprints, did did my client like the blueprints? And in the middle, you can imagine agents who are salespeople and agents who are, like, doing the designing, agents who, like, do the editing, whatever. Would that kind of signal work on an end to end system like that?
Because, like, one of the things that happens in human firms is management considers what's happening at the larger level and, like, gives these, like, fine grained signals to the the pieces or something when when, like, there's a bad quarter or whatever. Yeah. In the limit, yes.
That's the dream of reinforcement learning. Right?
you sort of create the information that allows you to learn from that signal. But I don't expect that to be the thing that works first. I think this is going to require an incredible amount of care and like diligence on the behalf of humans surrounding these machines and making sure they do exactly the right thing and exactly what you want and giving them right signals to improve in the ways that you want.
Yeah, you can't train on the RL reward unless the model generates some reward.
Yeah, that's Yeah, exactly. You're in it. Like, you're in this like sparse RL world where like, if it never if the client never likes what you produce, then, like, you don't get any reward at all and, like, it's kinda bad.
But in the future, these models will be good enough to get the reward some of the time. Right? This is the nines of reliability.
That's what I was talking about. Yeah. There's an interesting digression, by the way, on earlier, you were talking about, well, we want dense representations that like, that that will be favored.
Right? Like, that's a more efficient way to communicate. A book that Trenton recommended, the symbolic species, has this really interesting argument that language is not just a thing that, like, exists, but, like, it it was also something that evolved along with our minds and specifically evolved to be both easy to learn for children and to something that helps children develop.
Right? Like, it's On back half of home? Because, like, a lot of the things that children learn are received through language.
Like, the languages that will be the fittest are ones that help, like, raise the next generation. Right? And that, like, makes them smarter, better, whatever.
And you think gives them the concepts to express more complex ideas. Yeah. Think Yeah.
That, and I guess, more pedantically, just like not die. Right. Yeah.
Sure.
Unless you encode the important shit to not die.
so then when we just think of, like, language is like, oh, you know, say this contingent and maybe suboptimal way to represent ideas. Actually, maybe one of the reasons that LLMs have succeeded is because language has evolved for tens of thousands of years to be this sort of cast in which young minds can develop. Right?
Like, that is the purpose it was evolved for.
when you talk to, like, multimodal or, like, computer vision researchers versus when you talk to language model researchers. Yeah. People who work in other modalities have to put enormous amounts of thought into exactly what the right representation space for the images is.
And, like, what the right signal to learn from there. Is it like directly modeling the pixels? Or is it, you know, some loss that's conditioned on?
There's there's like a paper ages ago where they like, found that if you trained on the internal representations of an image that model, it like helps you predict better. But then later on, like, that's obviously like limiting. And so there was, like, PixelCNN where they're trying to, like, discreetly model, you know, the the the individual pixels and stuff.
But understanding the right level of representation there, really hard. In language, people are just like, well, I guess you just predict that. Thanks, token.
Right? It's kinda easy. Yeah.
Decisions made. I mean, there's the tokenization, like, discussion and debate about, like but one of Gwen's favorites. But Yeah.
Yeah. That's that's very interesting. How much the the case for a multimodal being a way to bridge the data wall or get past the data wall Yeah.
Is con is, like, based on the idea that the things you would have learned from more language tokens anyway you can just get from YouTube, it has that actually been the case? How much, like, positive transfer do you see between different modalities where, like, actually, the images are helping you be better at, like, writing code or something just because, like, the model is learning a latent capability is just from trying to understand the image? Demis, in his interview with you mentioned positive transfer.
I can't get in trouble if he's
like, it's the boss.
I mean, I I can't say heaps about that other than to say this is something that people, like, believe that, yes, like, we have all of this data about the world. It would be great if we could, like, learn an intuitive sense of physics from it that helps us reason. Right?
That seems totally plausible. And Yeah.
pieces where if we fine tune on math problems, the model just gets better at entity recognition. Woah. Really?
Yeah. Yeah. So so there's like a a paper from David Bao's lab recently where they investigate what actually changes in a model when I fine tune it with respect to the attention heads and these sorts of things.
Fascinating. And they have this, like, synthetic problem of box a has this object in it. Box b has this other object in it.
What was in this box? And and if you've and it makes sense. Right?
It's like you've you've you've better at, like, attending to the positions of things which you need for, like, coding and manipulating math equations.
I love this kind of research. Yeah. What's the name of the paper?
Do you know?
Math. Models. Math.
David Balsgrave that came out, like, a week ago. Okay. I am And I'm not wanna get out.
I'm not endorsing the paper. That's, like, a longer conversation. But, like, this it does talk about insight other work on this, like, entity recognition ability.
Yeah. One of the things you mentioned to me a long time ago is the evidence that when you train LLMs on code, they get better at reasoning and language Yeah. Which unless it's the case that the comments in the code are just really high quality tokens or something, implies that to be able to think through how to code better, like, it makes you, like, a better reasoner and, like like, that's crazy.
Right? Like, I think that's, like, one of the strongest pieces of evidence for, like, scaling just making the thing smart. Like Yeah.
That kind of, like, positive transfer. I I think, like, this is this is true in two senses. One is just that modeling code obviously implies modeling a difficult reasoning process used to create it.
like, structure of, like, composed reasoning, I guess. Like, if this, then that, like, encodes a lot of structure in that way.
Yeah. That you could imagine transferring to other types of types of reasoning problem. Right.
And and crucially, the thing that makes us significant is that the, it's not just stochastically predicting Right. The next token of, like, words or whatever because it's, like, learned that that, like, Sally corresponds to murderer at the end of a Sherlock Holmes story. No.
Like, if there is some shared thing between code and language, it must be at a deeper level that the modern hustler. Yeah. I think we have a lot of evidence that actual reasoning is occurring in these models and that, like, they're not just stochastic parrots.
Yeah.
It just feels very hard for me to believe that having worked and played with these models. Yeah.
will listen will be like, you know,
I guess. Yeah. My my two, like, immediate cash responses to are one, the work on Othello and now other games where it's like, I give you a sequence of moves in the game, and it turns out if you apply some, like, pretty straightforward interpretability techniques, then you can get a board that the model has learned.
Yeah. Yeah. And it's never seen the game board before anything.
Right? Right. Like, that's generalization.
The other is Anthropix influence functions paper that came out last year where they look at the model outputs, like, please don't turn me off. I wanna be helpful. And then they scan, like, what was the data that led to that.
And, like, one of the data points that was very influential was someone dying of dehydration in the desert and, like, having, like, a will to keep surviving. And and to me, that just seems, like, very clear generalization of of motive rather than regurgitating don't turn me off. I think 2001, A Space Odyssey was also one of the influential things, and so that's that's more related, but it's clearly pulling in things from from lots of different distribution.
circuits to like do addition. Yeah. Right?
Like model induction heads or induction heads, this kind thing. Like, you can literally encode basic reasoning processes in the models manually. And it seems clear there's evidence that they also learned this automatically because you can then rediscover those from trained models.
Yeah. To me, this is Yeah. Pretty strong.
The models are under parameterized. Yeah. They need to learn parameterized.
They were asking them to do they want to learn. The gradients want to flow. Yeah.
And so they need to they're learning Yeah. More more general skills. Yeah.
Okay.
So I I wanna take a step back from the research and ask about your careers specifically. Because like the tweet implied at the that I introduced you with, you've been in this field a year and a half. I think you've only been in it, like, a year or something.
Right? It's like I don't know. Yeah.
But, you know, like, in that time I I know the solved alignment takes are overstated, and you won't say this yourself because you'd be embarrassed to be like you know, it's like a a pretty incredible thing. Like, the the thing that people in MechaDiscernible typically think is the biggest, you know, step forward, and you've, like, been working on it for a year. It's it's notable.
So I'm curious how you explain what's happened. Like, why in a year or year and a half have you guys been, you know, made an important contributions to your field?
It goes without saying luck, obviously. And I I feel like I've been very lucky in, like, the the timing of different progressions has has been just, like, really good in terms of advancing to the next level of of growth. I feel like for the interpretability team specifically, I joined when we were five people.
We've now grown quite a lot. But there were so many ideas floating around, and we just needed to, like, really execute on them and have, like, quick feedback loops and, like, do careful experimentation that led to, like, signs of life and have now allowed us to, like, really scale. And I feel like that's kind of been my biggest value add to the team, which is not all engineering, but but quite a lot of it has been.
Interesting. So you're saying, like, you came at a point where, like, they were there was had been a lot of science done, and there was a lot of, like, good research floating around, but they needed someone to, like, just take that and, like, maniacally execute on it. Yeah.
Yeah. And and and there's this is why it's not all engineering because it's, like, running different experiments and, having a hunch for why it might not be working and then, like, opening up the model or opening up the weights and, like, what is it learning? Okay.
Well, let me try and do this instead and that sort of thing.
or or, yeah, theories. And why was that lacking in the existing?
I don't know. I feel like I feel like I I mean, I I work quite a lot, and then I I feel like I just I'm, like, quite agentic. Like, if your if your question's about, like, career overall and and I've been very privileged to have, like, a really nice safety net to be able to take lots of risks, but I'm just, like, quite headstrong.
Like, in in undergrad, Duke had this thing where you could just make your own major. And it was like, I don't like this prerequisite or this prerequisite, and I wanna take all four or five of these subjects at the same time. So I'm just gonna make my own major.
Or, like, in in the first year of grad school, like, canceled rotations so I could work on this thing that became the paper we were talking about earlier and, like, didn't have an adviser, like, got admitted to do machine learning for protein design and was just, like, off in computational neuroscience land with no business there at all, but but worked out.
There's a headstrong in this, but I seemed like another theme that jumped out was the the the ability to step back, and you were talking about this earlier, the ability to step back from your sunk costs and go in a different direction is in a weird sense the opposite of that, but also a crucial step here where I know, like, 21 year olds or, like, 19 year olds who are like, this is not a thing I've specialized in or, like, I I didn't major in this. They're like, dude, motherfucker. You're 19.
Like, you can definitely do this. And then you were, like, switching in the middle of grad school or something. Like, that's just just like yeah.
Sorry. I didn't mean to cut you off, but I I I think it's, like, strong ideas loosely held Yep. And being able to just, like, pinball in different directions.
And the the headstrongness, I think, relates a little bit to the fast feedback loops or agency in so much as I I just don't get blocked very often. Like, if I'm trying to write some code and, like, something isn't working, even if it's, like, in another part of the code base, I'll often just go in and fix that thing or at least hack it together to be able to get results. And I've seen other people where they're just like, help.
I can't and it's like, no. That's not a good enough excuse. Like, go all the way down.
a month after they give them a test or week after they give them a test. I'm like, how is it going? And they say, well, you know, we need to do this thing which requires lawyers because it requires talking about this regulation.
So, like, how's that going? I was like, well, we need lawyers. I'm like, why didn't you get lawyers?
Yeah.
Or something like that. Yeah. So that's definitely like, yeah.
I think that's arguably the most important quality in like almost anything. Yeah. It's just pursuing it to like the end of the earth and like whatever you need to to make it happen, you'll make it happen.
If you do everything you want to do everything.
yeah, yeah.
I think from my side, definitely that quality has been important, like agency and work. There are thousands or I would even like probably tens of thousands of engineers at Google who are like, you know, basically, like, we're we're all like equivalent, like, software engineering ability, let's say. Like, you know, if you gave us like a very well defined task, then we'd probably do it like equivalently well.
Maybe a bunch of them would do it a lot better than me, you know, in all likelihood. But what I've been like, one of the reasons that I've been impactful so far is I've been very good at picking extremely high leverage problems. So problems that haven't been like particularly well solved so far.
Perhaps as a result of like, frustrating structural factors, like the ones that you pointed out in like that scenario before, where they're like, oh, we can't do x because this what team went to do, y or like and then going, okay, well, just gonna, like, vertically solve the entire thing. Right. And that turns out to be remarkably effective.
Also, I'm very comfortable with, like, if I think there is something correct that needs to happen, I will like, make that argument and continue making that argument at escalating levels of like, criticality until that thing gets solved. And I'm also quite pragmatic with what, like I do to solve things, you get a lot of people who come in with, as I said before, like a particular background or their familiarity, or they like they know how to do something. And they weren't like, one of the beautiful things about Google, right, is you can run around and get world experts in literally everything.
You can sit down and talk to people, optimization experts, tipi, like, chip design experts, like experts in, like different forms of like pre like pre training algorithms or like RL or whatever. And you can learn from all of them and you can take those methods and apply them. And I think this was like maybe the start of why I was initially impactful was like this vertical like agency effectively.
And then a follow-up piece from that is, I think it's often surprising how few people are like fully realizing all the things they want to do. They're like blocked or limited in some way. And this is very common in big organizations everywhere.
People like have all these blockers on what they're able to achieve. And I think being a, like one helping inspire people to to work on particular directions and working with them on doing things massively scales your leverage. You get to work with all these wonderful people who teach you heaps of things.
And generally, helping them push past organizational blockers means like together, get an enormous amount done. Like none of the impacts that I've had has been like me individually going off and solving a whole lot of stuff. It's been me to maybe like starting off a direction and then convincing other people that this is the right direction and bringing them along in like this big tidal wave of like, effectiveness that goes and solves that problem.
we should talk about how you guys got hired because I think that's a really interesting story because you were a McKinsey consultant. Right? That's good.
yeah. Generally, people just don't understand how peep like, decisions are made about either admissions or evaluating who to hire or something. But like, just talk about like, how were you noticed as Yeah, totally.
Yeah, get got hired. So like the TLDR of this is I studied robotics in undergrad. I always thought that AI would be one of the highest leverage ways to impact the future and buzz away.
Like the the reason I am doing this is because I think it is like one of our best shots at making a wonderful future, basically. And I thought that working actually McKinsey, I would get a really interesting insight into what people actually did for work like in this, I actually wrote this as the first line in my cover letter to McKinsey was like, I want to work here so that I can learn what people do so that I can, like, understand how they work. And and in many respects, like, I did get that.
I asked a whole lot of other things. Many of the people there are like wonderful friends actually learned I think a lot of this like agentic behavior in part from my time there where you go into organizations and you see how impactful just not taking no for an answer gets you. Like, it's like, you would be surprised at the kind of stuff where like, because no one no one quite cares enough in some organizations, things just don't happen because no one's willing to take direct responsibility.
This is incredibly like directly responsible individuals are ridiculously important. And people are willing to, like, they just don't care as much about timelines. And so much of the value that an organization like McKinsey provides, is hiring people who you were otherwise unable to hire for a short window of time where they can just like push through problems.
I think people like under appreciate this. And so like, at least some of my will hold up like, I'm gonna become the directly responsible individual for this because no one's taking appropriate, like, responsibility. I'm gonna care a hell of a lot about this, and I'm gonna make sure it like, near the end of the earth to make sure it gets done comes from that time.
But more to your, like, actual question of, like, how did I you get get hired. The entire time, I didn't get into grad programs that I wanted to get into over here, which was specifically for focus on like robotics and RL research and that kind of stuff. And in the meantime, on nights and weekends, basically every night from 10PM till 2AM, I would do my own like research.
And every weekend for like at least six to eight hours each day, I would do my own, like research and coding projects and this kind of stuff. And that sort of switched in part from like, quite robotic specific work to after reading Gwen scaling hypothesis post, I got completely scaling pilled. I was like, okay.
But clearly, the way that you solve robotics is by, like, scaling large multimodal models. And then in an effort to scale large multimodal models with a very like, know, grant I got a grant from the the TPU, like, access program to, like, Tensor Research Cloud. I was trying to work out how to scale out effectively.
And James Bradbury, who at the time was at Google and is now at Anthropic, saw some of my questions online where I was trying to work out how to do this properly. He was like, I thought I knew all the people in the world who were like asking these questions. Who on earth are you?
And he looked, you know, he looked at that and he looked at some of the like the robotic stuff that I've been putting up on my blog and that kind of thing. And he reached out and said, hey, do wanna have a chat and do wanna, like, explore working with us here? And I was hired, I understand it later as an experiment in trying to take someone with extremely high enthusiasm and agency and pairing them with some of the best engineers that he knew.
And so one another one of the reasons I could say, like, I've been impactful is I had this like dedicated mentorship from utterly wonderful people, like people like Rainer Pope, who has since left to go do his own ship company, and some left Skya James himself, many others. But those are, like, the sort of formative, like, two to three months at the beginning. And they taught me a whole lot of, like, the principles and, like, heuristics that I apply, like, how to and how to, like, solve problems in the way that they have, particularly in that, like, systems and algorithms overlap where, like, one more thing that makes you, like, quite effective in ML research is really completely understanding the systems side of things.
And this is something I learned from them, basically, is like a deep understanding of how systems influence algorithms and how algorithms influence systems. Because the systems constrain the design space or the solution space which you have available to yourself in the algorithm side. And very few people are comfortable fully bridging that gap.
But a place like Google, you can just like go and ask all the algorithms experts and all the systems experts everything they know, and they will happily teach you. And if you guys go and sit down with them, they like they will teach you everything they know. It's wonderful.
This has meant that I've been able to be very, very effective for both sides, like for the pre training crew, because I understand systems very well. I can chew it and understand, like, this will work well or this won't. And then and, like, flow that on through the inference considerations of models and this kind of thing.
And so like to the chip design teams, I'm one of the people they turn to to understand what chips they should be designing in three years, because I'm one of the people who's best able to understand and explain the kind of algorithms that we might want to design in three years. And obviously, you can't make very good guesses about that. But like, I I think I like convey the information well accumulated from all of my compatriots on the pre training crew and like the general systems that I grew and convey that information well to them.
Because also even inference applies a constraint to pre training. And so like there's this like these trees of constraints where if you understand all the pieces of the puzzle, then you get a much better sense for, like, what the solution space might look like. Yeah.
That that that's really there's a couple of things that stick out to me there.
is not just the agency of the person who was hired, but the parts of the system that we're able to think, wait. That's really interesting. Who is this guy?
Yeah. Not for a grad program or anything. You know, like, currently a McKinsey consultant.
Just like under an undergrad. But that's interesting. Let's, like, give this a shot.
Right? So James and whoever else, that's, like, that's very notable, and that's second is I actually didn't know this part of the story where that was part of an experiment run internally about, can we do this? Can we, like, bootstrap somebody?
And, like yeah. And in fact, what's really interesting about that is the third thing you mentioned is having somebody who understands all layers of the stack and isn't so stuck on any one approach or any one layer of abstraction is so important. And specifically, that like, what you mentioned about being, being bootstrapped immediately by these people might have meant that since you're getting up to speed on everything at the same time rather than spending grad school going deep on, like, one specific way of doing RL, you actually can take the global view and aren't, like, totally bought in on one thing.
So not only can is this something that's possible, but, like, has greater returns than just hiring somebody at a grad school potentially. Right. Because this person can just, like I don't know.
you know what I mean? So, yeah, that that is really You come at everything with fresh eyes. Yep.
And you know, come in lock to any particular field. Yep. Now, what like one caveat to that is that before, like, during my self experimentation and stuff, I was reading everything I could.
I was like obsessively reading papers every night. And, like, actually, funnily enough, I like, read much less widely now that I like my day is occupied by working on things. And in some respect, I had like this very broad perspective before where not that many people even even like in a PhD program, you'll like focus on a particular area.
in a way that I guess, like, foreshadowed some of the the work that I would later do. That that that's super interesting. One of the reasons that you've been able to be agentic within Google is, like, your pair programming half the days or most of the days with Sergey Brin.
Right?
LLM stuff and, like, get rid of the the local block blockers in Yeah. Its place. I I think important to give is, like, not, like, every day or anything that I'm very but, like, when, there are particular projects that he's interested in, then, we'll work together on those.
I'm like, but there's also been times when he's been focused on projects with other people. But in general, yes, there's a surprising alpha to, like, being one of the people who actually goes down to the office every day. That like is really actually shouldn't be but is surprisingly impactful.
And as a result, I've like benefited a lot from having like, basically being like close friends with people in leadership who care. And being able to, like, really argue convincingly about why we should do x as opposed to y. And and having that like vector to try it like it's Google is a big organization.
Having having those vectors helps a little bit. But also, it's it's very important. It's the kind of thing you don't want to ever abuse.
Right? Like, you wanna make the argument through all your like, all the right channels Yeah.
only sometimes you need to. And this is inclusive of like Sergey Brin, Jeff Dee, and so forth. I mean, it's, like, it's notable.
I don't know. I feel like Google is undervalued given that, like Yeah. That that, like I don't know.
Like, Steve Steve Jobs is working on the equivalent, like, next product for Apple, like, PureCloud or something.
Right? I mean, like, I, yeah, I've benefited immensely from, like, triggers. Okay.
So for example, during the Christmas break, I go I was just going into the office a couple a couple days during that time.
Sounds like quite a lot of
And I don't know if you guys have read that article about Jeff and Sanjay doing the pair programming, but they were their pair programming on stuff. And I got to hear about all these, like these cool stories of like early Google, where they're talking about like crawling under the floorboards and rewiring data centers and like, telling me how many like bits they were pulling off the install, how many bytes are pulling off the instructions of a given compiler and instruction and like, all these like, crazy little performance optimizations who are doing like they were having time of their live. And I got to, like, sit there and really, like, experience this this sense of history in a way that you you don't expect to get.
Like, you expect to be very far away from all that, I think, maybe in a large organization. But yeah. That's super cool.
And, Trenton, does this map onto any of your experience? I think Shalta's story is more more exciting.
Mine was just very serendipitous in that I I got into computational neuroscience, didn't have much business being there. My first paper was mapping the cerebellum to the attention operation and transformers. My next ones were looking at, like How old are was that?
It was my first year of grad school. Okay. So 22.
Oh, yeah.
But, yeah, my my next work was on sparsity in networks, like inspired by sparsity in the brain, which was when I met Tristan Hume. And Anthropic was doing the solo, the softmax linear output unit work, which was was very related in quite a few ways of, like, let's make the activation of neurons across a layer really sparse. And if we do that, then we can get some interpretability of what the neuron's doing.
I think we've updated on that approach towards what we're doing now. So that that started the conversation. I shared drafts of that paper with Tristan.
He was excited about it. And and then and and that was basically what led me to be become Tristan's resident and then convert to full time. But during that period, I also moved as a visiting researcher to Berkeley and started working with Bruno Olshausen, both on what's called vector symbolic architectures, which one of the core operations of them is literally superposition, and on sparse coding, also known as dictionary learning, which is literally what we've been doing since and Bruno Olshausen basically invented sparse coding back in 1997.
And so it was like the the the my research agenda and the interpretability team seemed to just be running in parallel in in with just research tastes. And and so it, yeah, it made a lot of sense for for me to work with the team.
Well And it it's been a dream since. One thing I've noticed that when people tell stories about their careers or their successes, they ascribe it way more to contingency. But when they hear about other people's stories, they're like, of course, it wasn't contingent.
Do you know what I mean? Yeah. It's like, if that didn't happen, something else would have happened.
I've just noticed this when you, like, talked to and it's, like, interesting that you both think that there are, like, it was especially contingent. Whereas I don't know.
Maybe you're right, but, like, it is it this is sort of interesting pattern that Yeah. But, I mean, like, I literally met Tristan at a conference and, like, wasn't didn't have a scheduled meeting and put through anything. Just, like, joined a little group of people chatting, and he happened to be standing there.
And I happened to mention what I was working on, and that led to more conversations. And I think I probably would have applied to Anthropoc at some point anyways, but I would have waited at least another year.
it's still crazy to me that I can, like, actually contribute to interpretability in a meaningful way. I think there's a big important aspect of like shots on goal there, so to speak. Right?
Where like, you're even just going to con choosing to go to conferences itself is like putting yourself in a position where you're where luck is more likely to happen. Yeah. And like, conversely, in my own situation, it was like, doing all of this work independently, in trying to produce or do interesting things was my own way of like trying to manufacture luck, so to speak.
And and, like, try and do something meaningful enough that it got noticed.
specifically James, and I think our manager Brennan was trying to run this experiment. It, like, worked. Did they do it again?
Yeah. So my, like, closest collaborator, Enrique, he he he crossed from search through to our team. He's also been ridiculously impactful.
He's definitely a stronger engineer than I am, and he didn't go to university.
How was like, what was notable about, for example, is James Badbury as somebody who's who, usually, this kind of stuff is, like, farmed out to recruiters or something like that. Whereas James, like, somebody whose time is worth, like, hundreds of millions of dollars to go, you know, I'm in with the you know what I mean? Like Yeah.
So that, like, that thing is, like, very bottlenecked on that kind of person taking the time almost in, like, aristocratic tutoring sense of finding and then getting up to speed. And it seems like if it worked this well, it should be done at scale. Like, it should be the responsibility of key people to, like, you know what I mean?
Onboard I think find. I think that is true to many sense. Like, I'm I'm sure you probably benefited a lot from the the key researchers mentoring you deeply during And and, like, actively, like, looking on, like, open source repositories or, like, on forums or whatever for, like, potential people like this.
Yeah. I mean, James is like Twitter injected into his brain.
yes. And I think this is something which in practice is done. Like people do look out for people that they find interesting and like try and find high signal.
In fact, actually, this, I was talking about this with Jeff the other day. And Jeff said that, he's like, know, I one of the most important hires I ever made was also called email. And I was like, well, who was that?
And he's Chris Olaf.
Ah, yeah.
Because Chris similarly had had no background and in like, in no formal background in my own writing like Google Brain was just getting started in this kind of thing. But Jeff saw saw that signal and the brand, the residency program, which I Brain had is I think also like a it was astonishingly effective at finding good people that didn't have strong ML backgrounds.
Yeah. One of the other things that's I wanna emphasize for a potential slice of the audience that would be relevant to is there's this sense that, like, the world is legible and efficient of companies have these go to jobs.google.
com or jobs.whatevercompany.com, and you apply and there's the steps, and, like, they will evaluate you efficiently on those steps.
Whereas not only from the storage teams, like, often that's not the way it happens. That's in fact is good for the world that that's not often how it happens. Like, it is important to look at were they able to, like, write an interesting blog technical blog post about their research or, like, making interesting contributions.
Yeah. I want you to, like, riff on for the people who are, like, just assuming that the other end of the job board is, like, just, like, super legible and mechanical. This is not how it works.
And in fact, like, people are looking for the sort of different way different kind of person who's a genetic and putting stuff out there. And and I think specifically what people are looking for there is two things. One is agency and, like, putting yourself out there.
The second is the ability to do world class something. Yeah. And two examples that I always like to point to here are Andy Jones from Anthropic did an amazing paper on scaling laws as applied to board games.
It didn't require much resources. It demonstrated incredible engineering skill, demonstrated which incredible understanding of, like, the most topical problem of the time. And he didn't come from, like, typical academic background or whatever.
As I understand it, basically, like, as soon as he came out with that paper, both Anthropic and OpenAI were like, we would desperately like to hire you. There's also someone who works on Anthropic's performance team now, Simon Baum, who has written in my mind, the reference for optimizing a CUDA map model, like on a GPU. And that demonstrated example of like taking some like prompt effectively and producing the world class reference example for it.
In something that wasn't particularly well done. So far is like, think an incredible demonstration of like ability and agency that in my mind would, like, be an immediate would, like, please love to, like, interview you slash hire.
Yeah. The only thing I can add here is I I mean, I still had to go through the whole hiring process and all the standard interviews and this sort of thing. Yeah.
Everyone does. Everyone does. Yeah.
Yeah. Is is that is that doesn't that seem stupid? I mean It's important debiasing.
Yeah. Yeah. And they're like, the bias is what you want.
Right? Like, a lot of bias of somebody who's got a great taste. And, like, he's like, like, who care like Your interview process should be able to disambiguate that as well.
Yeah. Like like, think there are cases where someone seems really great, and then it's like, oh, they actually just can't code. This sort of thing.
Right? Like, how much you weight these things definitely matters, though. And, like, I think the the we take references really seriously.
The interviews, you can only get so much signal from. And so it's all these other things that can can come into play for whether or not a hire makes sense.
like,
they test the right things. One man's bias is another man's taste, you know.
I I guess the only thing I would add to this or maybe to the headstrong Yeah. Context is like, there's this line, the system is not your friend. Right.
And it's not necessarily to say it's it's actively against you. It's your your sworn enemy. It's just not looking out for you.
Right. And so I think that's where a lot of the proactiveness comes in of like, there are no adults in the room. Yeah.
Or, like and and, like, you have to come to some decision for what you want your life to look like and then execute on it. And and, yeah, hopefully, you can then update later Yeah. If you're too headstrong in the wrong way.
But but I think you almost have to just kind of charge at at certain things Right. To to get much of anything done, not be swept up in the tide of whatever the expectations are. There's, like, one final thing I wanna add, which is, like, we talked a lot about agency and this kind of stuff.
route. And when you care an unbelievable amount, you'd like you check all the details and you have like this understanding of like what could have gone wrong and you'd like you it just it matters more than you think because people end up not caring or not caring enough. This is like LeBron quote, where he talks about how when he sort of before he suddenly he was, like, worried that everyone would be, like, incredibly good.
And and then he gets there and he, like, realizes that actually once people hit financial stability, then they, like, they relax a bit. And he's like, oh, it's gonna be easy. I don't don't think that's quite true because I think in, like, AI research, because most people actually care quite deeply.
there's caring about your problem. And there's also just caring about the entire stack and everything that goes up and down, like going explicitly going and fixing things that aren't your responsibility to fix because overall, it makes like the stack better. I mean, another part of that I forgot to mention is you were mentioning, oh, going in on weekends and on Christmas break, and you get to, like the only people in the office are Jeff Dean and Sergey Brand or something.
You know? And you just, like, get to pay a program with them. It's just it's interesting to me the people I don't I wanna pick on your company in particular, but, like, people at any big company, they've, gotten there because they've gone through a very selective process.
That's like they had to compete in high school, they had to compete in college, but it almost seems like they get there and then they take it easy. When in fact, this is a time to put the pedal to the metal, go in and pair program with Sergey Brin on the weekends or whatever. You know what I mean?
I mean, there's there's pros and cons there. Right? I think many people make the decision that the thing that they wanna prioritize is, like, a wonderful life with their family.
And if they're they do wonderful work, like, let's say they don't work every hour of the day. Right? But they do wonderful work in the work, like the hours that they do do, that's incredibly impactful.
I think this is true for many people at Google is like, maybe they don't work as many hours as like your typical startup mythologize. Right? But the work that they do do is incredibly valuable.
It's very high leverage, because they know the systems and they're experts in their field. And we also need people like that. Like our world rests on these huge, like difficult to manage and difficult to fix systems.
And we need people who are like, willing to work on and help them fix and maintain those in frankly, like a thankless way that isn't as like high publicity as all of this AI work that we're doing. Right? And I'm like ridiculously grateful that those people do that.
And I'm also happy that there are people for whom like, okay, they find technical fulfillment in their job and doing that well. And also, like, maybe they draw a lot more from when also spending like a lot of hours with their family. And I'm lucky that I'm at a stage in my life where, like, yeah, I can go in and work every hour of the week, but, like, that's like, I'm I'm not making as many sacrifices to do that.
Yeah. I mean, like, just one example that sticks out in my mind of this sort of, like, the other side says no, and you can still get the the yes on either end. Basically, every single high profile of guests I've gotten so far, I think maybe with a one or two exceptions, I've sat down for a week and I've just come up with a list of sample questions that's, you know, like, try to come up with really smart questions to say to them.
And the entire process, I've always thought, like, is it if I just cold email them, it's like a 2% chance they say yes. If I include this list, there's a 10% chance. And because otherwise, you know, there's like you go through their inbox and every thirty four seconds, there's an interview for whatever podcast, interview or whatever podcast.
And every single time I've done this, they've said yes. Wow. Yeah.
You just like Exactly. You just great questions. But If you do everything, you'll win.
But you just, like, you literally have to dig in the same hole for, ten minutes. Or in that case, like, make a sample list of sample questions for them to get past their not immediate list. Yeah.
You know what I mean?
And just Demonstrate how much you care. And Yeah. Yeah.
The work you're willing to put in. Yeah. Yeah.
Yeah. I something that a friend said to me a while back, but I think it's stuck is like, it's amazing how quickly you can become world class at something just because most people aren't trying that hard. Yeah.
And like are only working like, I don't know, the actual, like, twenty hours that they're actually spending on this thing or something.
then, like, you can you can get really far pretty fast. And I think I'm lucky I had that experience with the fencing as well. Like, I had the experience of becoming world class in something and, like, knowing that she just worked really, really hard and were like Yeah.
For for context, by the way, Shota was one seat away as he was the next person in line to go to the Olympics stuff for fencing. I was at best, like, forty second in the world for fencing, for fencing.
You know, was no load is a thing, man.
And
there was a there was one cycle where, I was like the next highest ranked person in Asia. And if one of the teams had been, like, disqualified for doping as it was occurring in part during that cycle, and as occurred for like the Australian rowing women's rowing team, I think went because one of the teams was disqualified, then I would have been the next in line.
It's interesting when, like, you just, find about peers people's prior lives, and it's like, oh, you know, this guy was almost an Olympian. This other guy was whatever. You know what I mean?
Okay. Let's talk about interpretability. Yeah.
I actually wanna stay on the brain stuff as a way to get into it for a second. We're we're previously discussing, is the brain organized in the way where you have a residual stream that is gradually refined with higher level associations over time or something. There's a fixed dimension size in a model.
If you had to I don't even know how to ask this question in a sensible way. But what is the d model of the brain? What is it like the embedding size of or because of feature splitting, is that not a sensible question?
No. I think it's a sensible quest well, it is a question.
You could just not say that. No. No.
No. This is a question. You can tell.
Just like, how do you believe it?
kinda be like, okay. Well, this part of the brain is like a vector of this dimensionality. Dimensionality.
I mean, maybe for the visual stream, because it's like v one to v two to IT, whatever. You could just count the number of neurons that are there and be like, that is the dimensionality. But it seems more likely that there are kind of sub modules and and things are divided up.
So, yeah, I don't have and and I I'm not, like, the world's greatest neuroscientist. Right? Like, I did it for a few years.
I, like, studied the cerebellum quite a bit. So I'm sure there are people who could give you a better answer on this.
Do you do you think that the way to think about whether it's in the brain or whether it's in these models, fundamentally, what's happening is like features are added, removed, changed, and, like, the feature is the fundamental unit of what is happening in the model? Like, what what would have to be true for give me a and this goes back to the earlier thing we were talking about whether it's just associations all the way down. Give me, like, a counterfactual.
In the world where this is not true, what is happening instead? Like, what is the alternative hypothesis here?
Yeah. It's hard for me to think about because at this point, I I just think so much in terms of this feature space. I mean, at at one point, there was, like, the kind of behavioralist approach towards cognition where or it's like you're just you're, like, input output, but you're not really doing any processing.
Or it's like everything is embodied and you're just like a dynamical system that's like operating, like, along like some predictable equations. But like there's no state in the system, I guess. But whenever I've read these sorts of critiques, it's like, well, you're just choosing to not call this thing a state.
But you could call like any internal component of the model a state. Like, even with the feature discussion, it's defining what a feature is is really hard. And so the the question feels almost too slippery.
Mhmm. What is a feature? A direction and activation space.
A latent variable that is operating behind the scenes that has, like, causal influence over the system you're observing. It's a feature if you call it a feature. It's tautological.
a very rough, intuitive sense, like, a sufficiently sparse and like, spec that features like whether or not something is turned on or off. Right? Like, in a very simplistic sense.
Yeah. Yeah. Which might be, I think, useful metaphor to understand it by.
It's like, when we talk about features activating, it is in many respects, the same way that neuroscience would talk about like a neuron activating, right? If that neuron corresponds to to something particular, right? Yeah.
I know, I think that's useful as like, what do we want a feature to be? Right? Right.
Like, what is a synthetic problem under which a feature exists? But even with the towards monosomaticity work, we talk about what's called feature splitting, which is basically you will find as many features as you give the model the capacity to learn. And and by model here, I mean the the the up projection that we we fit after we trained to the original model.
And so if you don't give it much capacity, it'll learn a feature for bird. But if you give it more capacity, then it will learn like ravens and eagles and sparrows and, like, specific types of birds.
Still on definitions thing. I I guess, naively, I think of things like bird versus what kind of token is it like a is it like a period at the end of a hyperlink as you're talking about earlier versus at the highest level, things like love or deception or like, holding a very complicated proof in your head or something. Is this all features?
Because then the definition seems so broad as to almost be not that useful. Like, I I rather that there's seems to be some important differences between these things, and they're all features like yeah.
I mean, all of those things are, like, discrete units that have connections to other things that then imbues them with meaning. I that feels like a a a specific enough definition that it's it's useful or not too all encompassing, but feel free to push back.
that could make you think like, oh, this is, like, kinda fundamentally the wrong way to think about what's happening in a
I mean, if the features we were finding weren't predictive or if they were just representations of the data, right, where it's like, oh, you're all you're doing is just clustering your data, and there's no, like, higher level associations that are being made. Or it's some, like, phenomenological thing of, like, you're call you're saying that this feature files for marriage, but if you activate it really strongly, it doesn't change the outputs of the model in a way that would correspond to it. Like, I think those these would both be good critiques.
I I I guess one more and is and we tried to do experiments on MNIST, which is a dataset of of digits images. And we didn't look super hard into it, and so I'd I'd be interested if people other people wanted to take up, like, a deeper investigation. But it's plausible that your, like, latent space of representations is dense and it's a manifold instead of being these discrete points.
And so you could, like, move across the manifold, but at every point there would be some meaningful behavior. And it's much harder than to label things as features that are discrete.
Like, in a naive sort of outsider way, the thing that would seem to me is to be, like, a way in which this picture could be wrong is if there's not some, like, this thing is turned on, turned off, but it's, like, a much more global kind of, like the system is a I don't I'm gonna because it's really clumsy, like, you know, I measured it in a pretty kind of language, but is there a good analogy here? Yeah. I guess if you think of, like, something like the laws of physics, it's not like, well, the feature for wetness is turned on, but it's only turned on this much.
And then the feature for, like, you know I guess maybe it's true because, like, the mass is, like, a gradient and, like, you know, like, I don't know. But the polarity or whatever is a gradient as well. But there's also a sense in which, like, there's the laws, and the laws are more general, and you have to understand, like, the general bigger picture.
sub circuit. But that's where, like, the the reasoning circuit itself comes into play, right, where you're taking these features ideally and, like, trying to compose them into something high level. Like, you might say, okay.
Like, when I'm using at least this is my head cannon. Let's say I'm trying to use the foot, you know, f equals m a. Right?
Then I presumably at some point have features which like denote, okay, like, what mass and then that's like helping me retrieve the actual mass of the thing that I'm using. And then like like the acceleration and this kind of stuff. But then also, maybe there's a higher level feature that does correspond to using the first law of physics, maybe.
But the more important part is that the composition of components, which helps me retrieve a piece of relevant piece of information and then produce, like, maybe something like a, you know, multiplication operator or something like that, when necessary. At least that's my headcanon.
What is a compelling explanation to you, especially for very smart models of like, I understand why it made this output, and it was, like, for a legit reason. If it's doing million line pull requests or something, what are you seeing at the end of that request where you're like, yep. Shoot.
That's chill.
Yeah. So ideally, you apply dictionary learning to the model. You've you've found features.
Right now, we're actively trying to get the same success for attention heads, in which case we have features for both the the core you you can do it for residual stream, MLP, and attention throughout the whole model. Hopefully, that point, you can also identify broader circuits through the model that are, like, more general reasoning abilities that will activate or not activate. But in your case, where we're trying to figure out if this, like, polar crush should be approved or not, I think you can flag or detect features that correspond to deceptive behavior, malicious behavior, these sorts of things, and see whether or not those have fired.
That would be like an immediate you can do more than that, but that would be an immediate.
what what what does the reasoning circuit look like? What would that look like when you found it? Yeah.
So, I mean, the induction head is probably one of the simplest But it say like reasoning. Right? Well, I mean, what do you call reasoning?
Right?
it's it's it's it's a good reason. So I guess context for listeners, the induction head is basically and you see the line like mister and missus Dursley did something. Mister blank, and you're trying to predict what blank is.
And the head has learned to look for previous occurrences of the word mister, look at the word that comes after it, and then copy and paste that as the prediction for what should come next, which is a super reasonable thing to do. And there is computation being done there to to accurately predict the next token. Mhmm.
But, yeah, that is context dependent. That is yeah. Yeah.
But it's not like it's not like reasoning. You know what I mean? Like But but is is, I guess, going back to the, like, associations all the way down, it's like, if you chain together a bunch of these reasoning circuits or or heads that have different rules for how to relate information.
But but in this sort of, like, zero shot case, like, something is happening where when you, like, pick up a new game and you immediately start understanding how to play it, and it doesn't seem like an induction heads kind of thing.
and turning them into latent representations of the different objects in the game. Right? And, like, a circuit that is learning physics from What what would that because induction heads is, like, one layer transformers.
Like, Two layers. Yeah. Yeah.
So you can, like, kind of see, like, what the like, the thing that is a human picks up a new game and understands it. How, like, how do how would you think about what that is? Is it presumably, it's across multiple layers, but, like, is it yeah.
How like, what would that physically look like?
How big would it be maybe or, I mean, that would just be an empirical question, right, of, how big does the model needs be to perform this task? But, like but, I mean, maybe it's useful if I just talk about some other circuits that we've seen. So we've seen, like, the IOI circuit, which is the indirect object identification.
And so this is like, if you see it it's like Mary and Jim went to the store. Jim gave the object to blank. Right?
And it would predict Mary because Mary's appeared before as like the indirect object. Or it'll it'll infer pronouns. Right?
And this circuit even has behavior where like if you ablate it, then like other heads in the model will pick up that behavior. We'll even find heads that wanna do copying behavior and then other heads will suppress. So, like, it's one jobs one head's job to just always copy, like, the token that came before, for example, or the token that came five before or whatever.
And then it's another head's job to be like, no, do not copy that thing. So there are lots of different circuits performing, in these cases, pretty basic operations.
But when they're chained together, you can get unique behaviors. And but, like, is the story of how you found it, but the reasoning thing is, like because you won't be able to understand or it'll just be, like, really con you know, it won't be something you can see in, like, a two layer transformer. So will you just be, like, the the circuit for deception or whatever?
It just this this part of the network fired when we, at the end, identified the thing as being deceptive. This part and it didn't fire when we did identify it as being deceptive. Therefore, this must be the deception circuit?
I think a lot of analysis like that.
which is like the model saying what it thinks you want to hear.
like, bad and which one is good. Yeah. So we have tons of instances.
And actually, you make models larger, they do more of this, where the model is clearly it has, like, features that model another person's mind. And these activate and, like, some subset of these we're we're hypothesizing here, but, like, would be associated with more deceptive behavior.
Although, like, it's doing that by I don't know, Chad GPT, I think, is probably modeling me because that's, like, our, like, to Yeah. Serious mind. Yeah.
Yeah. So well, first of all, the thing you mentioned earlier about there's redundancy. So then it's like, well, have you caught, like, the whole thing that could cause deception of the whole thing or, like, is it just one instance of it?
Yeah. Second of all, are your, like, labels correct? You know, maybe, like, you you thought this wasn't deceptive.
It's, like, still deceptive, especially if it's producing output you can't understand. Third is the thing that's gonna be the bad outcome, something that's even human understandable. Like, the session is a concept we can understand.
Maybe there's, a Yeah. Yeah. So a lot to unpack here.
So I guess a few things.
it's fantastic that these models are deterministic. When you sample from them, it's stochastic. Right?
But, like, I can just keep putting in more inputs and ablate every single part of the model. This is the kind of the pitch for computational neuroscientists to come and work on interpretability. It's like, you have this alien brain and you have access to everything in it, and you can just ablate however much of it you want.
And so I think if you do this carefully enough, you really can start to pin down what are the circuits involved, what are the backup circuits, these sorts of things. The kind of cop out answer here, but it's important to keep in mind, is doing automated interpretability. So it's like as our models continue to get more capable, having them assign labels or, like, run some of these experiments at scale.
And then with respect to, like, if there's superhuman performance, how do you detect it? Yeah. Which I think was kind of the last part of your question.
Aside from the cop out answer, if we buy this associations all the way down, you should be able to coarse grain the representations at a certain level such that they then make sense. I think it was even in in Demis' podcast, he's talking about, like, if a chess player makes a superhuman move, they should be able to distill it into reasons why they did it. And and, like, even if the model's not gonna tell you what it is, you should be able to decompose that complex behavior into simpler circuits or features to to really start to make sense of why it did the thing that it did.
exist, which it seems like there must or actually, I'm not sure if that's the case. And secondly, whether using this sparse autoencoder setup, you could find it. And in this case, if you don't have labels for it that are adequate to represent it, like, wouldn't find it.
Right?
Yes and no. So, like, we are actively trying to use dictionary learning now on the sleeper agents work, which we talked about earlier. And it's like, if I just give you a model, can you tell me if there's this trigger in it and it's gonna start doing interesting behavior?
And and it's an open question whether or not when it learns that behavior, it's part of a more general circuit that we can pick up on without actually getting activations for and having it display that behavior. Right? Because that would kind of be cheating then.
Or if it's learning some hacky trick over like, that's a separate circuit that you'll only pick up on if you actually have it do that behavior. But even in that case, the geometry of features gets really interesting because it, like, fundamentally, each feature, like, is in some part of your representation space and they all exist with respect to each other. And so in order to have this new behavior, you need to carve out some subset of the feature space for the new behavior and then push everything else out of the way to make space for it.
So so hypothetically, you can imagine you, like, have your model before you've taught it this bad behavior. You know all the features or, like, have some coarse grained representation of them. You then fine tune it such that it becomes malicious.
And then you can kind of identify this, like, black hole region of feature space where, like, everything else has been shifted away from it. And there's, like, this region and, like, you haven't put in an input that, like, causes it to fire. But then you can start searching for what is the input that would cause this part of the space to fire.
What happens if I activate something in this space? There are, like, a whole bunch of other ways that you can try and attack that problem.
This is sort of a tangent, but one interesting idea I heard was if that space is shared between models, you can imagine trying to find it in an open source model to then make like, Gemma is they said in the paper Gemma, by the way, opened Google's newly released open source model. They said in the paper, it's used trained using the same architecture or something like that. I had to be honest, I don't know because I have to read Gemma paper.
Right. It takes the same one, right, that it says something whatever as Gemini. So it to the extent that's true.
I don't know. How much, like, how much of direct teaming you do on Gemini is, like, potentially helping you jailbreak into Gemini? Yeah.
This gets into the fun space of, like, how universal our features across models and and our towards monomet monosomanticity paper looked at this a bit. And we find I I can't give you summary statistics, but like the base 64 feature, for example, which we see across a ton of models. This is like it there are actually three of them, but they'll fire foreign model base 64 encoded text, which is prevalent in, like, every URL.
And there are lots of URLs in the training data. They have really high cosine similarity across models. So, like, they all learn this feature.
And I mean, within a rotation. Right? But it's like yeah.
Yeah. Like, the actual, like, vector itself. Yeah.
Yeah. And I I I wasn't part of this analysis.
but trained with different random seeds. It supports the quanta theory of neural scaling. It's like a hypothesis, right, which is that like all models on like a similar dataset will learn the same features in the same order ish roughly like you learn your n grams, you learn induction heads, and you learn like to put full stops after numbered lines and this kind of stuff.
Yeah. But by the way, okay, so this is another tangent. To the extent that that's true, and, like, I guess there's evidence that that's true, why doesn't curriculum learning work?
Because if it is the case that you learn certain things first, shouldn't just directly training those things first lead to better results? Both Gemini papers mention some, like, aspect of curriculum learning. Okay.
Interesting. I I find the fact that fine tuning works is, like, evidence of curriculum learning. Right?
Because, like, the last things you're training on have a a disproportionate impact.
I wouldn't necessarily say that.
like use case. I think I'm not sure how true or is. I think the data about lab kind of paper kind of supports this, right?
Like you have that ability, and you're just like getting better at entity recognition, Like fine tuning that circuit instead of other ones. Yeah. Yeah.
I'm sorry. What was the the thing we're talking about before? But generally, I do think, like, curriculum learning is really interesting that people should explore more.
And it, like, seems very close.
love to see more analysis along the lines of the quantitative stuff and like understanding better what do you actually learn at each stage and like decomposing that out
and exploring whether or not curricula change that or But by the way, I just realized forgot to we're I just, like, got in conversation mode and forgot there's an audience. Curriculum learning is when you organize a dataset. When you think about a human, how they learn, they they don't just see, like, random wiki text and they just, like, try to predict it.
Right? They're, like, we'll start you off with, like, Lorax or something, and then you'll learn I I even remember what first grade was like, but you'll learn the things that first graders learn and then, like, second graders and so forth.
And so you would imagine Sorry. We we know you go never go past first grade.
Kidding. Kidding.
Okay. Anyways, let's get back to, like, the big before I we get into, like, bunch of, like, interpret details. The big picture, there's two threads I wanna explore.
First is, I guess, it makes me a little worried that there's not even an alternative formulation of what could be happening in these models that could invalidate this approach, which feels like I mean, we do know that we don't understand intelligence. Right? Like, there are definitely unknown unknowns here.
So, like, the fact that there's not a null hypothesis, I don't know.
and yeah. Yeah. Yeah.
Yeah. So it's not that there aren't other hypotheses. It's just I have been working on superposition for, like, a number of years Yeah.
And and very involved in this effort.
to or will, like You just said they're wrong.
Like like to to to these other approaches, especially because our recent work has been so successful. And like, for quite high explanatory power, like, yeah, there's this beauty, like, in the scaling laws paper, there's this little bump at a particular, like the original scaling laws papers, little bump. And that apparently corresponds to when the model learns induction heads.
And then like after that, it like sort of goes off track, learns induction heads gets back on track. Yeah. Which is like an incredible piece of retroactive explanatory power.
Yeah.
I do before I forget it though, do have one thread on future universality that you you might wanna have in. So there there are some really interesting behavioral evolutionary biology experiments on, like, should humans learn a real representation of the world or not? You can imagine a world in which we saw all venomous animals as, like, flashing neon pink, a world in which we survive better.
And so it would make sense for us to not have a realistic representation of the world. And the there's some work where they'll simulate, like, little basic agents and see if the representations they learn, like, map to the the, like, tools they can use and, like, the inputs they should have. And it turns out if you have these little agents perform more than a certain number of tasks given these basic tools and objects in the world, then they will learn a, like, ground truth representation Because, like, there are so many possible use cases that you need for these base objects that you actually want to learn what the object actually is and not some, like, cheap visual heuristic or other thing.
And so to the extent that we are doing and we haven't talked at all about, like, for instance, free energy principle or predictive coding or anything else. But, like, to the extent that all living organisms are trying to, like, actively predict what comes next and form, like, a really accurate world model, it it wouldn't surprise me or I'm I'm optimistic that we are learning genuine features about the world that are good for modeling it, and our language models will do the same. At least, especially because we're training them on human data and human text.
Another dinner party question. Isn't should we be less worried about misalignment and maybe that's not even the right word for what I'm referring to, but, like, just alienness and chagethness from these models, given that there is feature universality, and there are certain ways of thinking and ways of understanding the world that are instrumentally useful to different kinds of intelligences.
Should we just be less worried about, like, bizarro paperclip maximizers as a result? I I think that's the this is kind of why I bring this up as, like, the optimistic take. Yeah.
Predicting the Internet is very different from what we're doing, though. Right? Like, the models are way better at predicting NEXT tokens than we are.
They're trained on so much garbage. They're trained on so many URLs. Like, in the dictionary learning work, we find there are, like, three separate features for base 64 encodings.
And, like, even that is kind of an an alien example that is probably worth me talking about for a minute. One of these base 64 features fired for numbers. One, like like other base 64 like, if if it sees base 64 numbers, it'll, predict more of those.
Another fired for letters. But then there was this third one that we didn't understand. And it, like, fired for, like, a very specific subset of of base 64 features.
And someone on the team who clearly knows way too much about base 64 realized that this was the subset that was ASCII decodable. So you could decode it back into the ASCII characters. And the fact that the model, like, learned these three different features, and it took us a little while to, like, figure out what was going on.
What is very Shoggoth esque? Oh, that is it's it has a denser representation of, like, regions that are particularly relevant to predicting the next token. Yeah.
Because it's so but but yeah. And it's clearly doing something that humans wouldn't. Right?
Like, you can even talk to any of the current models in base 64, and it will apply in base 64. Right. And you can then, like, decode it and it it works great.
That particular example, I wonder if that implies that the difficulty of doing interoperability on smarter models will be harder because if, like, it requires somebody with esoteric knowledge who just happened to see that base 64 has I don't know. Like, whatever that distinction was, doesn't that imply when you have a million line pull request? It's like, there is no human that's gonna be able to decode, like, two different reasons why the pull request there's, like, two different features for this pull up.
Yeah. You know what I mean? Like Yeah.
Yeah. So if you And that's when you type a comment, like, small sales, please. Like Yeah.
Exactly. No. No.
I mean, you could do that. Right? This is, what I was gonna say is, like, one technique here is anomaly detection.
Right? Yeah. And so one beauty of dictionary learning instead of, like, linear probes is that it's unsupervised.
You are just trying to learn to span all of the representations that the model has and then interpret them later. But if there's a weird feature that suddenly fires for the first time that you haven't seen fire before, that's a red flag. You could also coarse grain it so that it's just a single base 64 feature.
I mean, even the fact that that this came up and we could see that it specifically favors these particular outputs and it fires for these particular inputs gets you a lot of the way there. I'm even familiar of cases from the auto interp side where a human will look at a feature and try to annotate it for it fires for Latin words. And then when you ask the model to classify it, it says it fires for Latin words defining plants.
So it can, like, already, like, beat the human in some cases for, like, labeling what's going on.
thing between models where, like, some model that you have, like, millions of features potentially for g p d six and some like, it just a bunch of models are just trying to figure out what each of these features means.
does that sound right? Okay. Yeah.
But you can even automate this process. Right? I mean, it's this goes back to the determinism of the model.
Like, you could have a model that is actively editing input text and and predicting if the feature is gonna fire or not and and figure out what makes it fire, what doesn't, and and, like, search the space. Yeah. I I wanna talk more about the feature splitting because I think that's, like, an interesting thing that has been under Yeah.
Explored especially for scalability. Think it's it's underappreciated. Right?
First of all, like, how do we even think about is it really just you can keep going down and down? Like, there's no end to the amount of features?
or things that are part of the data, but that the model isn't actually By the do you wanna explain what feature splitting is? Yeah. Yeah.
So it's the it's the part before where, like, the model will learn however many features it has capacity for that still span the space of of representation. So, like, give an example potentially of a feature. Yeah.
So you learn if you don't give the model that much capacity for the features it's learning, concretely if you project to not as high a dimensional space, it will learn one feature for birds. But if you give the model more capacity, it will learn features for all the different types of birds. And so it's it's more specific than otherwise.
And and oftentimes, like, there's the bird vector that points in one direction and all the other specific types of birds point in like a similar region of the space, but are obviously more specific than the the the course label. Okay. So let's go back to g p d seven.
First of all, is this a sort of, like, linear tax on any model to figure out it first, I have even before that, is this a one time thing you had to do, or is this the kind of thing you have to do on every output? Or is just like one time, it's not deceptive, we're good to go roll?
Actually, yeah, let me let me transfer that. Yeah. So you do dictionary learning after you've trained your model.
Yeah. And you feed it a ton of inputs. And and you get the activations from those, and then you do this projection into the higher dimensional space.
And so the method is it's unsupervised and that it's trying to learn these sparse features. You're not telling them in advance what they should be, but it is constrained by the inputs you're giving the model. I guess two caveats here.
One, like, we can try and choose what inputs we want. So if we're looking for theory of mind features that might lead to deception, we can put in the sycophancy dataset. Hopefully, at some point, we can move into looking at the weights of the model alone or at least using that information to do dictionary learning.
But I think in order to get there, that's, like, such a hard problem that you need to make traction on just learning what the features are first. But yeah. So what's the cost of this?
Can you repeat the last sentence? Weights of the model alone. So so so, like, right right now, we just have these neurons in the model.
They don't make any sense. Yeah. We apply dictionary learning.
We get these features out. They start to make sense. But that's that depends on the activations of the neurons.
The weights of the model itself, like what neurons are connected to what other neurons, certainly has information in it. And and the the dream is that we can kind of bootstrap towards actually making sense of the weights of the model that are independent of the activations of the data. I mean, this is I I'm not saying we've made any progress here.
It's a very hard problem, but it it feels like we'll have a lot more traction and be able to, like, sanity check what we're finding with the weights if we're able to pull out features first.
permanent.
are the sort of, like, artifacts of any single call. Mhmm. Yes.
In a brain metaphor, you know, the weights are, like, the actual connection scheme between neurons and the activations of the current neurons that are lighting up. Yeah. Yeah.
Yeah. Okay. So there's gonna be two steps to this for g p d seven or whatever model we're concerned about.
One, it let actually, first, correct me if I'm wrong, but, like, training the sparse autoencoder and, like, do the unsupervised projection into a wider space of features that have a higher fidelity to, like, what is actually happening in the model. And then secondly, label those features. Mhmm.
How because let's say, like, the cost of training the model is n. What will those two steps cost relative to n? We will see.
two main things. What is your expansion factors? Like, how much are you projecting into the higher dimensional space?
And how much data do you need to put into the model? How many activations do you need to give But this brings me back to the feature splitting to a certain extent because if you know you're looking for specific features, you can start with a really cheaper, like, course representation. So maybe my expansion factor is, like, only two.
So, like, I have a thousand neurons. I'm I'm projecting to a 2,000 dimensional space. I get 2,000 features out, but they're really coarse.
And so previously, had the example for birds. Let's move that example to, like I have a biology feature. And and but I really care about if the model has representations for bioweapons, is trying to manufacture them.
And so what I actually want is like an anthrax feature. What you can then do is rather than and and let's say the anthra you only see the anthrax feature if instead of going from a thousand dimensions to 2,000 dimensions, I go to a million dimensions. Right?
And so so you can kind of imagine this this this big tree of semantic concepts where, like, biology splits into, like, cells versus, like, whole body biology. And then further down, it splits into all these other things. So rather than needing to immediately go from a thousand to a million and then picking out that one feature of interest, you can find the direction that the biology feature is pointing in, which again is very coarse, and then selectively search around that space.
So, like, only do dictionary learning if this if something in the direction of the biology feature fires first. And so the the computer science metaphor here would be, like, instead of doing breadth first search, you're able to do depth first search where you're only recursively expanding and exploring a particular part of this, like, semantic tree of features.
things that are intuitive for humans. Right? Like, because we just don't know to deal with base 64, so we don't have that many you know, we just don't dedicate that much, like, whatever, firmware to, like, deconstruct English kind of base 64 it is.
How would we know that the subjects and this will go back to maybe the MOE discussion we'll have of I guess we might as well talk about it, but, like, in mixture of experts, the mixture of paper talked about how they couldn't find the the experts weren't specialized in a way that we could understand. There's not, a chemistry expert or a physics expert or something. So why would you think that, like, it'll be, like, biology feature and then deconstruct rather than, like, blah, and then you just deconstruct, and it's, like, anthrax and your, like, shoes and whatever.
So I haven't read the the Mistral paper. Yeah.
I mean, this goes back to, like, if you just look at the neurons in a model, they're polysemantic. And so if all they did was just look at the neurons in a given head, it's very plausible that it's also a polysemantic because of superposition.
Oh, and so Yeah. Tag on the thread that Dorkish wrenched in there. Have you seen in the subtrees when you expand them out, like, something in a subtree, which, like, you really wouldn't guess that it would should be there based on, like, the higher level of traction?
pursued as much as I want to yet. Yeah. But I think we're planning to I hope that maybe external groups do as well.
Like, what is the geometry of features? What's geometry of exactly. And how does that change over It would really suck if, like, the anthrax feature happened to be, like, below the, like, you know, coffee can, like, the substrate or something like that.
Right? Totally. Totally.
And that feels like the kind of thing that you could quickly try and find, like, proof of which would then, like, mean that you need to, like, then solve that problem. Yeah. Yeah.
Inject more structure to the geometry. Totally. I mean, it would really surprise me, I guess, especially, like, given how linear Completely the models agree.
That, like, there isn't some component of the anthrax feature, like vector Yeah. That is similar to and looks like the biology vector and that they're not in a similar part of the space. But, yes, I mean, ultimately, machine learning is empirical.
Yeah. We need to do this. I think it's gonna be pretty important for certain aspects of of scaling dictionary learning.
Yeah. Yeah. Interesting.
On the MOE discussion Yeah. There's an interesting scaling vision transformers paper that Google put out a little while ago where they like do image net classification with a like an MOE. And they find really clear class specialization there for experts.
Like there's a clear dog expert.
Wait. So did they like to make sure people are just not doing a good job of like identifying? I think I think it's it's hard.
Like it.
like, in some respects, there's almost no reason that like, all of the different archives, like features should go to one expert, like you could have biology, like, let's say, I don't know what buckets they had in their paper, but let's say they had, like, archive papers as, like, one of the things. You could imagine, biology papers going here, math papers going here, all of a sudden, you're like, breakdown is, like, ruined.
hypothesis. So so I think images are also in some ways just easier to interpret than text. Yeah.
Exactly. And, like, so so Chris Ola's, like, interpretability work on AlexNet and and these other models. Like, in the original AlexNet paper, they actually split the model into two GPUs just because they couldn't like, GPUs were so bad back then, relatively speaking.
Right? Like, still great at the time. That was one of the big innovations of the paper.
But they find branch specialization, and there's a Distill Pub article on this where, like, colors go to one GPU and, like, Gabor filters and, like, line detectors go to the other. And then, like, all of the other Really? Yeah.
Yeah. Yeah. Interesting.
And then, like, all of the other interpretability work that was done, like a lot like, the like, the floppy ear detector. Right? Like, that just was a neuron in the model that you can make sense of.
You didn't need to You can make sense position. Right? So so just different different dataset, different modality.
Like, I think a wonderful research project to do if someone is, like, out there listening to this would be to try and disentangle, like, take some of the techniques that Trenton's team has worked on and try and disentangle the neurons in the the the mixture paper the mixture model, which is oversaw. It's like I think that's a fantastic thing to do because it feels intuitively like they should be. They didn't demonstrate any evidence that there is.
There's also like in general, a lot of evidence that that should be specialization. Go and see if you can find it. Like and that's that's work that had that you know, Anthropica has published most of the stuff on like, as I understand it, like dense models, basically.
You that is a wonderful research project to try. And given Dwarkash's success with the Vesuvius challenge. Yeah.
We we should be pitching more projects because they will be solved if we talk about them on the podcast.
wait, I knew like, Nada told me about it before it dropped because we recorded the episode before it dropped. Why didn't you why did I not even try? Like You know what I mean?
Like, I don't know. Like, Luke is obviously very smart and, like yeah. He's he's he's an amazing kid, but, like, you showed that, like, a 21 year old on, like, some ten seventy or whatever he was working on could do this.
I don't know. Like, I feel like I should have so you know what? I'm before this episode drops, I'm gonna meet my I'm gonna try to meet my I'm gonna make an interpretability.
Gonna try to do research. Really like, I don't know. It's like, was honestly thinking about coming to screen.
It's like, wait. I should like, why didn't that not be fucked? Yeah.
Maybe your hands dirty. Yeah. That's right.
Dwarkesh's request for research. Oh, I I wanna harp back on this, like, the neuron thing. You said I think a bunch of your papers have said there's more features than there are neurons.
And this is just like, wait a second. I don't know. Like, a neuron is like, weights go in and a number comes out.
That's like, a number comes out. You know what I mean? Like, that's that's so little information.
Like, there's do you mean, like, there's, like, street names and, like, species and whatever? There's, like, more of those kinds of things than there are, like, a number comes out in a in a model. That's right.
Yeah. But how is a number comes out as, like, so little information. How is that encoding for, like Superposition.
You're just encoding. You're encoding a ton of features in these high dimensional vectors. In a brain, is there, like, an axonal firing or how are you thinking about it?
I don't know how you think about, like, how how how much, like, superposition is there in the human brain? Yeah. So Bruno Olshausen, who I think of as the leading expert on this Yeah.
Thinks that all the brain regions you don't hear about are doing a ton of computation in superposition. So everyone talks about v one as as, like, having Gabor filters and and detecting lines of of sort various sorts, and no one talks about v two. And I think it's because, like, we just haven't been able to make sense of it.
What is v two? It's like the next part of the visual processing stream. And and it's but yeah.
So I I think it's very likely. And fundamentally, like, superposition seems to emerge when you have high dimensional data that is sparse. And to the extent that you think the real world is that, which I would argue it is, we should expect the brain to also be under parameterized in trying to build a model of the world and also use superposition.
You can get a good intuition for this in in correct me if this example is wrong in like a two d plane, right?
like feature space here, two two neurons, basically. And you can imagine them each like turning on to various degrees, right? And that's that's like your x coordinate and your y coordinate.
But you can, like, now, like, map this onto a plane. You can actually represent a lot of different things and, like, different parts of the plane.
Oh, okay. So crucially, the unsuperposition is not an artifact of a neuron. It is an artifact of, like, the space that is created.
It's a combinatorial code. Yeah. Yeah.
Exactly. Yeah. Okay.
Cool. Yeah. Thanks.
I I I we kind of talked about this, but, like, I think it's just, like, kinda wild that it seems to the best of our knowledge the way intelligence works in these models and then presumably also in brains. It's just like, there's a stream of information going through that has, quote, unquote features that are infinitely or at least to a large extent to just like splitable and, you know, like, you can expand out a tree of like what this feature is and what's really happening is a stream, like that feature is getting turned into this other feature or this other feature is added. I don't know.
It's like, that's not something I would have just, like, thought, like, that's what intelligence is. You know what I mean? It's like a surprising thing.
It's not it's not what I would would have expected necessarily. What did you think it was? I don't know, man.
mean, yeah. Go Fi. He's a Go Fi.
all of this feels like Go Fi. Like, you're using distributed representations, but you have features and you're applying these operations to the features. I mean, the whole field of vector symbolic architectures, which is this computational neuroscience thing, it all you do is you put vectors in superposition and which is literally a summation of two high dimensional vectors, and you create some interference.
But but if it's high dimensional enough, then you can you can represent them. And you have variable binding, where you connect one by another. And, like, if you're dealing with binary vectors, it's just the x or operation.
So you have a b, you bind them together. And then if you query with a or b again, you get out the other one. And this is basically the, like, key value pairs from attention.
And with these two operations, you have a Turing complete system, which you can if you have enough nested hierarchy, you can represent any data structure you want Yep. Etcetera etcetera. Yeah.
Okay. Let's go back to the super intelligence.
So, like, walk me through g p d seven. You've got, like, the sort of depth first search on its features. Okay.
G p d seven has been trained. What happens next? Your your research has succeeded.
G p d seven has been trained. What are you what what are what are we doing now?
We try and get it to do as much interpretability work and other, like, safety work as possible. No.
what has happened such that you're like, cool. Let's deploy a g p d seven. Oh, jeez.
I mean, ideal like like, we have our responsible scaling policy, which has been really exciting to see other labs adopt.
And Like, specifically from the perspective of your your research is net net like, Trenton, given your research, you got the the we got the thumbs up on g p d seven from you. Or actually, we should say, Claude, whatever. And then oh, I like what what what is the basis on which you're telling the team, like, hey.
Let's go ahead.
I think we need to make a lot more interpretability progress to be able to, like, comfortably answer the question. Comfortably give the green light to deploy it. Like, like, I would be, like, definitely not.
I'd be crying.
Maybe my tears would interfere with the GPUs. But, like, what is for GPUs. Yes.
Guys. Gemini five GPUs.
like, what what given the way your research is progressing, like, what does it kind of look like to you? Like, what would if this succeeded, what would it mean for us to okay GPT seven based on your methodology?
which lights up when the model knows that it's not telling the full truth to you. Why can't you just train a linear probe like Colin Burds did? So the CCS work is not looking good in terms of replicating or, like, actually finding truth directions.
And, like, in hindsight, it's like, well, why should it have worked so well? But linear probes, you need to know what you're looking for, and it's like a high dimensional space, and it's really easy to pick up on a direction that's just not Wait. But don't you also here, you need to label the features.
So you saw that know label them post hoc, but it's unsupervised. You're just like, give me the features that explain your behavior is the fundamental question. Right?
It's like like like like the actual setup is we take the activations, we project them to this higher dimensional space, and then we project them back down again.
but do it in a way that's sparse. Mhmm. By the way, for the audience, linear probe is you just, like, classify the activations.
I don't know. From what I vaguely remember about the paper was, like, if it's, like, telling a lie, then you, like, you just train a classifier on, like, is it yeah. In the end, was it not was it a lie, or is it just, like, wrong or something?
I don't know. It was, like, true or false question. Classifier on the activations.
So so, yeah, like, right now, what we do for GPT seven Yeah. I like I like, ideally, we have, like, some deception circuit that we've identified that, like, appears to be really robust.
And and it's like Well, like, what so you've done the projecting out to the million whatever features or something. Mhmm. Is the circuit because I I we're we I maybe we're using feature and circuit interchangeably when they're not.
So, like, is there is there, like, a deception cert like, what is that? So I think there are there are features across layers that create a circuit. Yeah.
gives you a lot more specificity and sensitivity than an individual feature. And it's like, hopefully, we can find a circuit that is really specific to you being deceptive. The the model deciding to be deceptive in cases that are malicious.
Right? Like, I'm not interested in a case where it's just doing theory of mind to, like, help you write a better email to your professor. And I'm not even interested in cases where the model is is necessarily just like modeling the fact that deception has occurred.
But doesn't all this require you to have labels for all those examples?
then, like, whatever faults that the linear probe has on the like, maybe you've, like, labeled along the thing or whatever, wouldn't the same thing apply to the labels you've come up with for the unsupervised features you've come up with?
So in an ideal world, we could just train on, like, the whole data distribution and then find the directions that matter to the extent that we need to reluctantly narrow down the subset of data that we're looking over just for for the purposes of scalability. We would use data that looks like the data you'd use to fit a linear probe. But again, we're not like, with the linear probe, you're also just finding one direction.
Like, we're finding a bunch of directions here.
And I guess hope is, like, you found, like, a bunch of things that light up when it's being deceptive, and then, like, you can figure out why some of those things are lighting up in this part of the distribution and now this other part and so forth. Totally. Yeah.
Uh-huh. Do you anticipate you'll be understand? Like, I don't know.
Like, the current models you've studied are pretty basic. Right? Do you think you'll be able to understand g why GPT seven fires in certain domains but not in other domains?
I'm optimistic. I mean, we've so so I guess one thing is this is a bad time to answer this question because we are explicitly investing in the longer term of, like, ASL four models, which GPT seven would be. But, like so so we split the team where a third is focused on scaling up dictionary learning right now, and that's been great.
I mean, we publicly shared our our some of our eight layer results. We've scaled up quite a lot past that at this point. But the other two groups, one is trying to identify circuits, then the other is trying to get the same success for attention heads.
So we're setting ourselves up and building the tools necessary to really find these circuits at a compelling way, but it's gonna take another, I don't know, six months before that's, like, really working well. But but, like, I I can say that I'm, like, optimistic, and we're making a lot of progress.
What is the highest level feature you found so far? Oh. Like, it's base 64 or whatever.
It's, maybe just, like, in the symbolic species language, the book you recommended, there's, like, indexical things where you're just, I forgot what the all the labels were, but like there's things where you're just like, you see a tiger and you're like, run and whatever, you know, just like a very sort of behaviorist thing. And then there's like a higher level of which what I refer to love, it refers to like a movie scene or my girlfriend or whatever. You know what I mean?
So Yeah. It's like the top of the tent. Yeah.
Yeah. Yeah. Yeah.
or whatever you found? I mean, probably one of the ones that we publicly well, pub publicly one of the ones that we shared in our update. So I think there were some related to, like, love and, like, sudden changes in scene, particularly associated with, like, wars being declared.
There are, like, a few of them in there in that in that post if you wanna link to it. Yeah. But but even, like, Bruno Olshausen had a paper back in twenty eighteen, nineteen where they applied a similar technique to a BERT model and found that as you go to deeper layers of the model, things become more abstract.
So I remember, like, in the earlier layers, there'd be a feature that would just fire for the word park. But later on, there was a feature that fired for park as, like, a last name, like, Lincoln Park or, like it's like a common Korean last name as well. And then there was a separate feature that would fire for parks as, like, grassy areas.
So so there's other work that points in this this direction. What do think we'll learn about human psychology from the observability stuff? Oh, gosh.
I okay. Okay. Here.
I'll give you a specific example. I think, like, one of the ways one of your updates put it was persona lock in. You know, you remember Sydney Bing or whatever.
It locked into, I think, what what was actually quite an endearing Yeah.
I
thought it's so funny. Yeah. I'm glad it's back in Copilot.
Oh, really? Yeah. Oh, yeah.
It's been misbehaving recently. Yeah. Yeah.
Actually, this this is another sort of threat to explore. But there was a funny one where I think it was, like, to the New York Times reporter. It it was you were nagging him or something.
And it was like, you are nothing. Nobody will ever believe you. You are insignificant and do whatever.
It's like I think the most gaslighting convince him to break up as well. Yeah. Yeah.
Okay. Actually, so this is an interesting exam. I don't even know where I was going with this to begin with.
But whatever. Maybe I got another thread. But, like, the other thread I wanna go on is that's yeah.
Okay. Actually, personas. Right?
So, like, is that a feature that, like, Sydney Bing having this personality is a feature versus another personality he can get locked into? And, also, like, is that fundamentally, like, what humans are like too where, I don't know. In front of all the different people, I'm, like, a different sort of personality or whatever.
Is that was that the same kind of thing that's happening to ShyJPT when he gets r l I don't know. Whole cluster of questions that can answer them and whatever. Yeah.
I really wanna do more work. I guess the sleeper agents is in this direction of, like, what happens to a model when you fine tune o when you are LHFA, these sorts of things. I mean, maybe it's trite, but you could just say, like, you conclude that people contain multitudes.
Right? And so much as they have lots of different features. There's even the stuff related to the Waluigi effects of, like, in order to know what's good or bad, you need to understand both of those concepts.
And so we might have to have models that are aware of violence and have been trained on it in order to recognize it. Can you post hoc identify those features and ablate them in a way where maybe your model's, like, slightly naive, but you know that it's not gonna be really evil? Like, totally, that's in our toolkit, which seems great.
Oh, really? Oh.
I don't know. It pulls the same thing, and then you figure out why like, what were the causally irrelevant pathways or whatever you modify, like and then the pathway to you looks like you just changed those. But you were mentioning earlier, there's a bunch of redundancy in the model.
Yeah. So you need to account for all that.
like sharper tools for making edits.
And it seems like at least at least from my perspective, that seems like one of the the primary way of, like, some degree confirming the safety or the reliability of model where you can say, okay, we found the circuits responsible, we've ablated them, can like, under a battery of tests, we haven't been able to now replicate the behavior which we intended to ablate. And like, that feels like the sort of way of measuring model safety in future as I as I would understand. Are you worried?
That's why I'm incredibly hopeful about their work. Mhmm. Because it's to me, it seems like so much more precise tool than something like RLHF.
RLHF, like you're prey to the black swan thing. You don't know if it's gonna, like, do something wrong in a scenario that you haven't measured.
and select the weight. Although not necessarily that you've, like, accurately labeled. Not necessarily.
Yeah. But but with a far higher degree of confidence than any other approach Yeah. That I've seen.
How I mean, like, what are your unknown unknowns for superhuman models in terms of this kind of thing where, like, I don't know how are the labels that are gonna be given things on which we can determine these are, like, this this thing is cool. This thing is a paperclip maximizer or whatever.
I mean, we'll see. Right? Like, I I I do like, the superhuman feature question is a very good one.
Like, I I I think we can attack it, but we're we're gonna need to to be persistent. And and the real hope here is, I think, automated interpretability. Yep.
And even having debate. Right? You could you could have the debate set up where two different models are debating what the feature does, and then they can actually, like, go in and make edits and, like, see if it fires or not.
Or but it is it is just this wonderful, like, closed environment that we can iterate on really quickly. That makes me optimistic.
Do you worry about alignments exceeding too hard? So, like, if I think about I would not want either companies or governments, whoever ends up in charge of these AI systems, to have the level of fine grained control that if your agenda succeeds, we would have over AIs, both for the ickiness of having this level of control over an autonomous mind. And second, just like, I don't fucking trust I don't fucking trust these guys.
You know? I don't I I I'm just kinda uncomfortable with, like, the loyalty feature has turned up and, like you know what I mean? And, yeah, like, how much worry do you have about having too much control over over DIs and, specifically, not you, but, like, whoever ends up with charge of these AI systems just being able to lock in whatever they want.
Yeah. I mean, I think it depends on what government exactly has control and, like, what the moral alignment is there.
That that is like, that whole value lock in argument is in my mind. It's, like, definitely one of the strongest contributing factors for why I am working on capabilities at the moment, for example. Just like I think the current player set actually, like, is extremely well intentioned.
And, I mean, for this kind of problem, I think we need to be extremely open about it. And like, I think directions like publishing the constitution that you take your model to abide by and then like trying to make sure you, like, RHF towards that and ablate that and have the ability for everyone to offer, like, feedback contribution to that is really important. Sure.
like, don't deploy when you're not sure, which would also be bad because then we just never catch it. Right. Yeah.
Exactly. I mean, paperclip. But, like Yeah.
Okay. There was some rapid fire. What is the buzz factor for Gemini?
a number of people who are really, really critical that if you took them out, then the performance of the program would be dramatically impacted. This is both on modeling like slash making decisions about like what to actually do. And importantly, on infrastructure side of things like it's just the stack of complexity builds, particularly when, like, someone like Google has so much, like, vertical integration.
Do you have when you have people who are experts, it becomes they become quite important. Yeah.
people like you can get in and in a year or so, you're making important contributions. And I've especially with Anthropic, but many different labs have specialized in hiring, like total outsiders, physicists, or whatever. And you just, like, get them up to speed, and they're making important contribute.
I I don't know. You I feel like you couldn't do this in, a bio lab or something. It's, an interesting note on the the state of the field.
I mean, bus factor doesn't define how long it would take to recover from it. Right? From.
Yeah.
And and deep learning research is an art.
how to read the loss curves or or set the hyper parameters in ways that empirically seem to work well. It's also like organizational things like creating context. When I think one of the most important and difficult skills to hire for is creating this like bubble of context around you that makes other people around you more effective and know what the right problem to work on.
And like that is a really tough to replicate. Thing. Yes.
Yeah. Totally.
long context, maybe agents, extra reliability. Who is the who is thinking well about what what that implies?
It's tough question. I'm, I think a lot of people look internally these days for sure, like their sources of insight or like progress. And like we all have obviously the sort of research programs and like directions that are tended over the next couple of years.
And I suspect, yeah, that most people, as far as like betting on what the future will look like, refer to like an internal narrative. Yeah, yeah, that is like difficult to share.
Yeah. If if it works well, it's probably not being published.
I mean, that that was one of the things in in the world scaling post. I was referring to something you said to me, which is I'm, you know, I missed the undergrad habit of just reading a bunch of papers. Yeah.
Is now there's nothing worth reading is published.
The community is progressively getting like more on track with what I think are like the the right and important directions. You're watching it like an agent. But I guess, like, it is tough.
The there used to be this, like, signal from big labs about, like, what would work at scale. That's kinda really hard for academic research to, like, find that signal. And I think getting like really good problem tastes about what actually matters to work on is really tough.
Unless you have, again, the feedback signal of of like what will work at scale, and what what is currently holding us back from scaling further understanding our models further. This is something where, like, I wish more academic research would go into fields like Interp, which are legible from the outside. You know, Anthropic deliberately publishes all its research here, and it seems like underappreciated in the sense that I don't know why there aren't dozens of academic departments trying to follow Anthropics guide in the interpret research because it seems like an incredibly impactful problem that doesn't require ridiculous resources and like this and like has all the flavor of like deeply understanding the basic science of what is actually going on in these things.
typically associate with academic science in some ways. Yeah. I do think the tide is changing there for whatever reason.
And, like, Neil Nanda has had a ton of success promoting interpretability Yes. In in a way where, like, Chris Ola hasn't been as active recently in in pushing things. Maybe because Neil's just doing quite a lot of the work, but, like, I don't know, four or five years ago, he was, like, really pushing and, like, talking at all sorts of places and and these sorts of things.
And people weren't anywhere near as receptive. Maybe they've just woken up to, like, deep learning matters and is clearly useful post HTTP. But Yeah.
Yeah. It's it's kinda striking. Mhmm.
Alright. Cool. And okay.
I'm trying to think what what is a good last question.
I mean, the one I'm going those thinking of is, do you think models enjoy next token prediction?
Yeah.
in love. We had this sense of things that were awarded in our access to our environment. There's, like, this deep sense of fulfillment that we think we're supposed to get from them or often people do, right, of, like, community or sugar or or, you know, whatever we wanted on the African Savannah.
Do you think, like, in the future, models trained with RL and everything a lot of post training on top of whatever. But they'll it's like they're like some in the way we were just a really, like, ice cream, they'll just be like, hi. Just to predict the next token again.
You know what I mean? Like, in the good old days.
So so there's this ongoing discussion of, like, our model sent to you or not. And like, do you thank the model when it helps you? Yeah.
But I think if you want to thank it, you actually shouldn't say thank you. You should just give it a sequence that's very easy to predict. And and the the the even funnier part of this is there's some work on if you just give it the sequence a, like, ah, like, over and over again.
wouldn't ever say.
so yeah. And I I won't say anything more about that, but you can yeah. You should just give your model something very easy to predict as a nice little treat.
This this is what the end of being.
But do we like things that are, like, easy to predict?
the, like, dose of entropy? The bits of entropy. Exactly.
Right? Yeah. Shouldn't you be giving up things that are just slightly too hard to predict?
Yeah.
out of reach.
Yeah. But I wonder, like, at least from the free energy principle perspective. Right?
Like, you don't like you don't wanna be surprised. And so maybe it's this, like, I don't feel surprised if you're in control of my environment. And so now I can go and seek things.
And I've been predisposed to, like in the long run, it's better to explore new things right now. Like, leave the rock that I've been sheltered under, ultimately leading me to, like, build a house or, like, some better structure. But we don't like surprises.
I I think most most people are very upset when, like, expectation does not meet reality.
That's why babies like love watching the same show over and over and over again. Right? Yeah.
And interesting. Yeah. I can see that.
Oh, I guess they're learning to model it and stuff too. But yeah. Yeah.
Yeah.
Okay. Well, hopefully, will be the This will be the repeat Yeah. That the the AIs learned to laugh.
Okay. Cool. I think that's a great place to wrap.
I should also mention that the better part of what I know about AI, I've learned from just talking with you guys. You know, we've been good friends for about a year now. So yeah.
I mean, yeah, appreciate you guys getting me up to speed here.
Yeah. You ask great questions. It's really fun to hang and chat.
Great. Great. I really treasure that time together.
Yeah. Yeah. Yeah.
You're getting a lot better pickleball.
I think I'm just thinking someone else.
Hey. We're trying to progress to tennis. Come on.
Yeah. Cool. Awesome.
Cool. Cool. Awesome.
Thanks. Hey, everybody. I hope you enjoyed that episode.
As always, the most helpful thing you can do is to share the podcast. Send it to people you think might enjoy it, put it in Twitter, your group chats, etcetera. Just blitz the world.
Appreciate your listening. I'll see you next time. Cheers.
Shared via Hopper