Don’t Fight Backprop: Goodfire’s Vision for Intentional Design, w/ Dan Balsam & Tom McGrath

Hello, and welcome back to the Cognitive Revolution. The Cognitive Revolution is brought to you in part by Granola. Just yesterday, I happened to see Ramp's monthly report on the fastest growing software vendors, and the number two company that is adding the most new customers right now is Granola.

Why? Aside from advertising on the cognitive revolution, I would chalk it up to an extremely smooth and easy to use product experience. If you're listening to this show, there is a good chance that you could, in theory, build AI workflows that capture audio, transcribe it, and use it in downstream prompts and workflows.

But can your teammates? That, I think, is where granola really shines. By delivering a polished product experience that anyone can immediately install and understand and by introducing AI capabilities in the form of recipes made by trusted thought leaders, granola is making AI accessible to everyone.

See the link in our show notes to try my blind spot finder recipe and explore all of the ways that granola can make your raw meeting notes awesome. Not just for you, but for everyone on your team regardless of their relationship with AI. Now today, I'm speaking with Dan Balsam and Tom McGrath, CTO and chief scientist of mechanistic interpretability startup, Goodfire, who in less than two years since founding the company have assembled an all star research team, landed a first wave of blue chip customers, including a couple that discovered Goodfire via Dan and Tom's first appearance on the show back in August 2024, published a remarkable series of results, and most recently announced a $150,000,000 series b fundraise at evaluation of $1,250,000,000.

Along with the fundraise, they've announced a new pillar in their research agenda, intentional design. A push to expand the scope of what interpretability science can do by complementing the current paradigm of reverse engineering how trained models work with a new approach focused on understanding and shaping the loss landscape to control what models learn during training and ultimately how they generalize. We begin with the discussion of interpretability developments broadly, with Tom emphasizing the shift from techniques like sparse autoencoders that transform a network's messy internal representations to sparse vectors where each node represents a distinct concept to newer approaches that attempt to understand the intricate geometric structures that these concepts inhabit within the model's latent space.

From there, we dive into their plans for intentional design and their first proof of concept, a technique for reducing hallucinations that uses a probe trained to detect hallucinations both to steer the model at runtime and as a source of reward signal for additional reinforcement learning training. Such training setups are not without controversy. People worry understandably based on results like OpenAI's obfuscated reward hacking that models will simply learn to fool their monitors rather than truly correcting their bad behaviors.

But Dan and Tom meet this concern head on, agreeing that paranoia is a way of life in alignment research, acknowledging that intentional design techniques are immature and probably should not be used on frontier models today, while also arguing first that the pace of AI capabilities advances really requires us to explore any and all possible paths to understanding and control. And second, that the specific details of the techniques really do make all the difference. In this hallucination reduction work specifically, the key trick they found was to run the hallucination detection probe on a frozen copy of the model during training so that the modified model would hopefully find it easier to learn not to hallucinate than to find a way to evade detection.

More generally, Top asserts that a key principle is to avoid fighting backpropagation. Because models are such high dimensional beasts, gradient descent will inevitably find ways around any attempt to prevent the model from learning what the loss function directs it to learn. Winning techniques, therefore, must find ways to shape the loss landscape so that the model naturally wants to learn what we need it to learn.

In the final part of the conversation, we discuss some of Goodfire's many other recent papers, including their work with PrimaMente, which suggested a new research direction by revealing that a state of the art model for predicting Alzheimer's diagnoses was basing its predictions on the length of cell free DNA fragments. We also discussed a project that showed that it's possible not only to determine which model weights are used for memorizing facts and which are used for more general purpose reasoning, but that you can actually improve model performance on at least some reasoning tasks by removing the memorization weights from the model entirely. Along the way, we also touch on how Goodfire intends to balance its need for business growth with its public benefit mission as they decide what research to publish and when, briefly consider how well we should expect today's interpretability techniques to work on new and different architectures, get Dan's thoughts on the possibility of AI consciousness, and much more.

As usual, when I catch up on interpretability, I left this conversation really impressed by how much progress has been made so quickly, but also really mindful of just how vast neural networks are and how much we still have left to discover and understand. With that, I wanna thank Dan and Tom for giving me another chance to drink from the Goodfire Research fire hose, and I hope that you learn as much as I did from this survey of mechanistic interpretability advances and introduction to the new paradigm of intentional design with Dan Balsam and Tom McGrath of GoodFire. Dan Balsam and Tom McGrath, CTO and chief scientist at Goodfire.

Welcome back to the cognitive revolution. Thank you for having us again. Yeah.

Thanks for having us. As always, you guys have been prolific, and there's a ton to cover. But let's start with the big headline news.

Goodfire is now a unicorn with a big fundraise announced in just the last couple weeks at a big valuation. Congratulations and recap the headlines for us. Yeah.

No. We're very excited to to announce this fundraise. It's been it's really like a testament to all of the hard work that the team has been doing.

It's pretty crazy. We've only been around for a year and a half. So the how much we've been able to accomplish and how much we've been able to grow over this period of time has been really awesome to see.

I mostly think of it as, like, I'm very excited to take this capital we've been able to raise and, like, deploy it in order to scale up what we're working on and continue to advance interpretability research. And we have a lot of new research to talk about, so I'm excited to talk about that too. Yeah.

That's, putting it mildly. I took my eye off the Goodfireball for a minute before my son got sick last year, and then I've kind of been, you know, less able to follow research. When I got back to the blog in preparation for this, I was like, holy moly.

There is a lot of stuff that has dropped. So we're gonna do as much as we can today in the time we have available. We're not gonna cover all of it and probably not even, a third of it, but it is, an impressive run for sure in terms of the team that you guys have been able to assemble, the results that that you've put out, and I'm looking forward to unpacking as much as we can.

As is my custom though, maybe let's start with kind of the real zoomed out view. We've done this couple times in the past. I'll give you outside interested outsider's take on what I think is going on in interpretability, and then you correct me, complicate, give me the next level of depth of understanding.

I guess what I'm seeing is we're moving from understanding the concepts that models are thinking about or representing in their internal states with things like sparse auto encoders to understanding the circuits that are operating and doing the information processing with things like tracing the thoughts of a large language model from anthropic and their transcoders or these wiring diagrams that show at least, like, simple operations, which already look pretty complex. I recommend people check out what it looks like for a model to add a couple two digit numbers to see something a little bit mind bending and arguably even a little hair raising. And but that's come pretty decent way at least.

And then we've now also got this notion of learning dynamics or kind of understanding how the model becomes what it is. I associate that with folks like Timaeus whose work still goes over my head, and you guys are getting into that space a little bit now as well. Do you think that's a useful way to build up?

And how far how much progress how would you characterize progress on each of those levels if that is a good taxonomy?

Yeah. So let me just try to echo that back to you a little bit. It's it seems to relate very much to I'm not sure if I have the right vocabulary for this, but there's a lot of discussion around are all the features in models linear or are some of them not linear?

And then there's, like, obviously different definitions and intuitions around what linear means. But I always go back to the days of the week as a canonical simple example that I think of where it's like, can have in your sparse autoencoder seven different active activation patterns that correspond to seven in the sparse version in the autoencoder, seven different spots in that super long, what is it that I'm looking for, vector of of concepts. And those could be just due to the vagaries of training, whatever those could be randomly distributed in that sparse autoencoder, but you can kind of find through your auto labeling process or whatever that okay.

Spot one is Monday and spot 5,027 is Tuesday, and then way down at the end is Wednesday, and then Thursday is over here in the middle and crazy like that. And you're like, that seems weird. And that is that that weirdness is reduced when you look at the relationship in embedding space between those concepts because they often are either clustered or they're in some plane where there's a rotation through a plane, some sort of shape that makes a lot more sense when you think, geez, you know, we actually do rotate through the days of the week.

So in some sense, it makes a lot of sense that the model would represent those things as rotation through some sort of plane. And now you're taking that one level up and saying, okay, now you can have all these kind of crazy shapes and these crazy it it becomes more of a topology sort of exercise where you're transforming these crazy high dimensional shapes through the circuits from whatever they start as to whatever they end up as. And that that's, I think the the real thing you're highlighting as missing from my initial characterization that we need to understand the space of the concepts and not just label them as present or absent at any given point in time, but there's a rich and very meaningful geometry of that as well.

what is at stake with this sort of are features linear or not linear? Because, again, I was revisiting, like, the definition of this is you could probably give it better than I can give it, but it's like features are in a direction. They can be added.

This is the sort of classic man plus royal equals king minus man gives you queen, minus royal gives you woman. You can move around in latent space in an additive way. And then there's also the idea that the intensity of the feature corresponds to, like, how important it is in the models processing at any given time.

But it does then strike me that doesn't quite handle the I'm not sure if it's in conflict with or if it just is an incomplete account of what's going on with the days of the week as the as the like, think of them in a plane, for example, because it's not is there a direction that's the Monday to Tuesday direction that I need to be thinking about? It's I can't add Monday and Tuesday, and that's not really a coherent concept anyway. But it does seem like something is different about that, and I I don't have a super crisp intuition around why people are so worried in the first place around what if there there's always comes you've not accounted for nonlinear features.

So, like, why do people harp on that so much?

Yeah. Okay. Hey.

We'll continue our interview in a moment after a word from our sponsors. Support for the show comes from VCX, the public ticker for private tech. For generations, American companies have moved the world forward through their ingenuity and determination.

And for generations, everyday Americans could be a part of that journey through perhaps the greatest innovation of all, The US stock market. It didn't matter whether you were a factory worker in Detroit or a farmer in Omaha, anyone could own a piece of the great American companies. But now, that's changed.

Today, our most innovative companies are staying private rather than going public. The result is that everyday Americans are excluded from investing and getting left further behind while a select few reap all of the benefits. Until now.

Introducing VCX, the public ticker for private tech. VCX by Fundrise gives everyone the opportunity to invest in the next generation of innovation, including the companies leading the AI revolution, space exploration, defense tech, and more. Visit getvcx.

com for more info. That's getvcx.com.

Carefully consider the investment material before investing, including objectives, risks, charges, and expenses. This and other information can be found in the fund's prospectus at getvcx.com.

This is a paid sponsorship. One of the best pieces of advice I can give to anyone who wants to stay on top of AI capabilities is to develop your own personal private benchmarks. Challenging but familiar tasks that allow you to quickly evaluate new models.

For me, drafting the intro essays for this podcast has long been such a test. I give models a PDF containing 50 intro essays that I previously wrote, plus a transcript of the current episode and a simple prompt. And wouldn't you know it?

Claude has held the number one spot on my personal leaderboard for 99% of the days over the last couple years, saving me countless hours. But as you've probably heard, Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you.

Whether you're debugging code at midnight or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. And with Claude Code, I'm now taking writing support to a whole new level. Claude has coded up its own tools to export, store, and index the last five years of my digital history from the podcast and from sources including Gmail, Slack, and iMessage.

And the result is that I can now ask Claude to draft just about anything for me. For the recent live show, I gave it 20 names of possible guests and asked it to conduct research and write outlines of questions. Based on those, I asked it to draft a dozen personalized email invitations.

And to promote the show, I asked it to draft a thread in my style featuring prominent tweets from the six guests that booked a slot. I do rewrite Claude's drafts, not because they're bad, but because it's important to me to be able to fully stand behind everything I publish. But still, this process, which took just a couple of prompts once I had the initial setup complete, easily saved me a full day's worth of tedious information gathering work and allowed me to focus on understanding our guests' recent contributions and preparing for a meaningful conversation.

Truly amazing stuff. Are you ready to tackle bigger problems? Get started with Claude today at claude.

ai/tcr. That's claude.ai/tcr.

And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Once more, that's claw.ai/tcr.

K. Perfect transition to intentional design. This is the sort of, let's say, updated vision for the company, basically.

Right? As of the new fundraiser put this out, roughly the same time. The big idea is I think pretty intuitive that we would be great to be able to not just throw an unbelievable amount of data and unbelievable amount of compute into some vast machinery and then get something out and then have to completely reverse engineer what the hell just happened.

And instead have some sense of what is going on along the way so that ideally, could control it and get something that behaves the way you want it to behave in all sorts of different situations. You talk about looking for methods that scale with compute, looking for strategies that support or allow for the possibility of natural language feedback. You know, what is the big vision for intentional design?

the good things we want to amplify, let's say. One analogy I like to use is it's like a map for the loss landscape. The data implies, like, some type of shape to the loss landscape, and you can imagine, like, a bunch of different valleys there.

And some of those valleys have behavior that we want that are desirable, and some of them have behavior that we don't want. And I think of the role of interpretability as, like, producing this map essentially. So when you get to, like, a juncture in the road, you can say, oh, I should go this way, not that way.

And I think that's what we're starting to unlock here.

and that's that intelligence comes from. Yeah. I got a lot of intuition from one section of the intentional design blog post where you describe one method for breaking a gradient down into semantic parts.

And I'll try to describe that too. Now everything's a mess in there is one starting point. So there is at any given time, you're optimizing this single final loss function, and you can change every weight throughout the entire model to make little contributions to getting better according to that final measure.

But what are we learning is by default not obvious at all. So the method that you share in the blog post is basically saying, okay, for one thing, we have these SAEs now. So we know and we can I do wanna dig into this and understand that at what point in training does this actually become useful?

Because this does seem to rely on there being relatively well developed concepts that are instantiated, represented, I guess, a better term. So you've you're at least some depth into the training process when this can start to make sense. But, okay, we can create an SAE that allows us to identify which concepts are active at any given time.

And then I thought the very kind of clever idea was saying, okay, if we look at what the gradient is changing in the residual stream inside the model, because of superposition, we know that there's probably a lot of different concepts that are represented there and like bunches of them could be changing in all sorts of different ways at the same time. We'll try to decompose that by looking at what concepts are active according to our SAE. Then we'll take the inner product.

In other words, we'll try to basically look for the similarity between how the gradient step is changing the internal activations and each of those concepts that are represented so that we can say, okay, it seems like this change is really aligned to this concept. It seems to be really changing this particular concept. And this other concept it's like aligned to and it's changing it somewhat, and some of the other concepts it's maybe not changing so much.

And then that gives you the opportunity to say, do I like that or not? Yeah. And so this is where then the language the intelligence of the language model can come in and say, okay, do I based on this example, based on this data that we are learning from, does this seem to be the right kind of thing to be learning?

So the example that you gave in the blog post was if you have data that consists of talking in pirate speak while doing arithmetic Yeah. You can the gradient will be optimizing for both of those things potentially at the same time in order to predict the tokens that it's seeing. But then when you go and do the look at what's active in the sparse autoencoder, you'll see, like, features related to arithmetic and all the other features related to pirate speak.

And then when you take the inner product or, again, look at basically also known as cosine similarity, right, look at the alignment between in in space, look at the alignment between these features and the changes that are being made, you can say, okay, it's changing. It seems to be upwaiting pirate speak quite a bit and it seems to be also upwaiting do math right. And so you could prompt the language model to guide it.

Hey, we wanna be getting good at math here, but we wanna be not over indexing on whatever other vagaries of the dataset happen to be present. And then it could look at these things and say, okay, let's let's allow this part of the gradient update that aligns to the feature that we think is a reasonable feature to be updating, and let's not make those changes that are changing the other things that we don't wanna change. You know, how how is that, how could that be problematic in other ways that I'm not anticipating?

Yeah. So the fighting of gradient descent there is the reward is the reward. Right?

So the update is gonna be in the direction of getting more reward. And the question is, are you teaching the model to overcome the instructions it has been given in pursuit of reward, or are you trying to align the instructions that it's given and the rewards such that it maintains a general understanding of itself as the kind of thing that follows instructions. Right?

Like, that that's a mental model I came away from that. Yeah. You're not you're not teaching it to my mental model too is that you're not teaching it to ignore its instructions.

And implicitly, if you don't say you can reward hack, it's interpreting its instructions as you can't, which is, I think, the correct default behavior. But then when it learns, it learns to, like, more broadly ignore, because you didn't set it up with the right prior. Yeah.

So many weird things going out.

The thing that that's, like, pretty normal to do is that you would freeze some layers of a model, and then you would train only some layers, or you would attach, like, a new head and you would train that because you value the representations up to some point in the model that are already in the model and you wanna, like, leverage them from some downstream task. This isn't exactly the way that we're thinking about intentional design, but maybe just as, like, a quick analogy. Like, imagine if you could just, like, selectively freeze circuits.

Right? Like, you could say, that circuit's good. I don't wanna change that.

This circuit, this is the one that I wanna update. Okay. Learn over this circuit, over this dataset, but not the other ones.

And I think when you frame it like that, it's really not that weird of a thing to be doing. It's just much more surgical than a lot of existing techniques. Hey.

We'll continue our interview in a moment after a word from our sponsors. Your IT team wastes half their day on repetitive tickets. Password resets, access requests, onboarding, all pulling them away from meaningful work.

With Servl, you can cut help desk tickets by more than 50%. While legacy players are bolting AI onto decades old systems, Servl allows your IT team to describe what they need in plain English and then writes automations in seconds. As someone who does AI consulting for a number of different companies, I've seen firsthand how painful and costly manual provisioning can be.

It often takes a week or more before I can start actual work. If only the companies I work with were using Serval, I'd be productive from day one. Serval powers the fastest growing companies in the world like Perplexity, Verkada, Merkor, and Clay.

And Serval guarantees 50% help desk automation by week four of your free pilot. So get your team out of the help desk and back to the work they enjoy. Book your free pilot at serval.

com/cognitive. That's serval.com/cognitive.

Everyone listening to this show knows that AI can answer questions, but there's a massive gap between here's how you could do it and here I did it. Tasklet closes that gap. Tasklet is a general purpose AI agent that connects to your tools and actually does the work.

Describe what you want in plain English. Triage support emails and file tickets in linear. Research 50 companies and draft personalized outreach.

Build a live interactive dashboard pulling from Salesforce and Stripe on the fly. Whatever it is, Tasklet does it. It connects to over 3,000 apps, any API or MCP server, and can even spin up its own computer in the cloud for anything that doesn't have an API.

Set up triggers and it runs autonomously, watching your inbox, monitoring feeds, firing on a schedule, all twenty four seven even while you sleep. Wanna see it in action? We set something up just for Cognitive Revolution listeners.

Click the link in the show notes, and Tasklet will build you a personalized RSS monitor for this show. It will first ask about your interests and then notify you when relevant episodes drop. However you prefer.

Email, text, you choose. It takes just two minutes, and then it runs in the background. Of course, that's just a small taste of what an always on AI agent can do.

But I think that once you try it, you'll start imagining a lot more. Listen to my full interview with Tasklet founder and CEO Andrew Lee. Try Tasklet for free at tasklet.

ai, and use code cog rev for 50% off your first month. The activation link is in the show notes, so give it a try at tasklet.ai.

Well, maybe take a I was gonna come to this later, but maybe it's a good time now to ask how you guys are thinking about balancing some of these tensions in just the overall nature of the business. It's a public benefit corporation with a couple notable, I think, relatively significant revenue projects for big companies that have been publicly disclosed. And there there's a need for a lot more revenue, obviously, to support a billion dollar valuation.

At the same time, there's this general mission of trying to not just develop these techniques, but presumably disseminate them or popularize them as well too. So that seems like a very tricky balance to strike or tight rope to walk over time. Do you have any principles?

Is there a way that that you've structured your thinking on this? I think, I don't know. At the highest level, like, it it is really important for us to get our work out there.

Like, we shared the hallucinations work that we did, which, like, contains in, like, strong detail everything that we did there, it's a bit of a recipe to do this type of work. There are lots of techniques that we're exploring in the intentional design space. We have different results, some of which will be hopefully getting out pretty soon, but it's really like a lot of research and a lot of greenfield research, and we're still figuring out what the right form factors are.

We have seen, in experiments that there are various ways to not fight gradient descent. And as we feel like particularly confident in the research and in the results that we're getting, we'll share more with the world. I think, like, overall, like, we do want to develop in public as much as possible.

And I think we will as we get more confident in the way things are going, I think we'll talk more in more detail about it. But there are different techniques that work in different ways. There are also things that are out in the world, like positive preventative steering, for instance, which I think, like, we can point to as an example of not fighting the loss landscape.

And for those that are unfamiliar with that work, that's like, you can prevent certain types of of misalignment. It's kind of similar to inoculation prompting in a way. Like, you can present certain types of misalignment by steering up on certain characteristics in a model.

And I think, like, all of these is, they're just, like they're like, one way to think about it like, I like the map for the lost landscape analogy, but also, like, it's just another way to think about it is that it changes the lost landscape when you're doing these things. Right? Like like, when you are applying some type of like, inoculation prompting is a good example.

Like, you are changing the nature of the loss landscape by providing this, like, prior over the dataset. So there's a lot of different techniques. Like, there are a lot of things you could try that would fight gradient descent.

Right? Then you, like, go to the next dataset example, and then you still have, like, optimization pressure in the direction you don't wanna go, and you're constantly fighting it. But there are lots of ways that you could intervene that don't fight gradient descent, and the reason that they don't fight gradient descent is because they fundamentally do change the structure of the Lost Landscape in a way that's durable.

So, obviously, models hallucinate. We would rather that they don't. And what can be done about it?

One thing that can be done is to create a synthetic dataset where you have known hallucinations that you can you have labeled going in, but I think the world has helpfully create prepared some of those for us and open source them. And then you can train a probe to classify the internal states of a model into this is a hallucination or it's not. And then you can do a few different things, including running that probe at runtime and potentially intervening on the output of the model.

But we've seen a bunch of these kind of token injection things over time in the reasoning development space where every so often you just insert like, wait, let me think about this a different way, and then the model kind of takes another stab at it. So I'm thinking of this as a sort of similar thing where the probe goes off and says hallucination and you force the next token to be like, but wait, I might have be I might be making this up. So then it will, at least some of the time, double back and realize that it was wrong and correct course.

Another thing you can do, and this is where it gets, like, I think really interesting, although notably the those, like, interventions drive a lot of the reduction. But another thing you could do is use the presence of activations, which were classified as hallucinations, as a signal for reinforcement learning to try to get the model to not go into this state in the first place by just basically punishing it for getting into the hallucination state at all. That is really interesting, and there are some interesting nuances too.

Why don't I just ask you to, again, give me the double click and start to dig into some of those nuances?

So I think one obvious question people would have around this sort of work is how do we know we're really getting it to do the things we want it to do as opposed to shooting ourselves in the foot in maybe one of a couple potentially dangerous failure modes. Obviously, this is all, like, relatively low stakes stuff at the moment. We wanna be mindful of how big the stakes could eventually be.

I did one episode. Gosh. It's been a little while now, but with we called it latent space hide and seek.

Scott Evans, Luke Bailey, and Eric Jenner, and they basically showed that the models can learn to dodge, to avoid, to evade all sorts of latent space detectors. So that's one problem is that if you start running these detectors similarly to what you said before, like, they kind of route around them and find other ways to do what they wanna do with with different kinds of activations, different kinds of patterns, whatever. And then there's also this result from OpenAI on obfuscated reward hacking where you say, geez, we don't like your reward hacking.

So if we review your chain of thought and we see that you are planning to reward hack, then we'll punch you for that. And then you get this sort of I think what makes this kind of especially dangerous is that at first, you get less reward hacking when you do that and you feel good about yourself, but then you do that too long and the reward hacking comes back because again, the incentive was still there and the reward is reward. So it's gonna learn to get that reward one way or another, but it also learns to not do the thing that it gets punished for.

And so you've essentially driven your reward hacking underground, driven it out of the chain of thought, and eliminated your ability to detect it in the first place. So Zvi, friend of the show, calls that the most forbidden technique. And I am I think you have an interesting and clever way to avoid that problem, but I'm not a 100% sure I understand if this is a fully robust way to avoid the problem or in the paper or in the blog post about it, it you said it's good to billions of tokens.

And I'm okay. Should I expect that to be good to tens of millions of tokens, hundreds of billions of tokens? At some point, does this break down?

So maybe describe what you did, first of all, and then we can fear speculate or theorize on how much we should trust it, I guess, is the key question. The trick is quite simple. Right?

Which is you run the probe on a cut on a frozen copy of the model.

So how I I guess one thing I don't have a great intuition for, and there might be a certain trick that makes it more obvious, but the fact that the probe continues to work on the original Frozen model and the student model is it's like not obviously that it should have worked out that way under kind of any circumstances necessarily, but I don't know, you know, exactly how you engineered it to encourage that to be the case. The thought that I had was some sort of, like, divergence penalty. A lot of these setups have some sort of some scheme for kind of tethering the weights to their original state to keep them keep changes local as much as possible.

We would expect that this is contingent on the setup, on the representations, on a bunch of different factors as well. You know, like, there is work, of course, as you mentioned, that is explored in in the Far AI paper. They found various setups that would cause obfuscation.

So our claim here is not that obfuscation doesn't happen or that it's never going to happen, but the dynamics of obfuscation are worthy of study. And there are cases in which I think we can demonstrate pretty convincingly that obfuscation doesn't happen at least under the conditions that some people might thought that it might before. I don't know if you know, we've we talked, on a previous episode about interpretability generally being pre paradigmatic, and then I think you upgraded to quasi paradigmatic at one point.

For this type of thing, it seems like we're maybe back to pre paradigmatic, but maybe you do have some rules for yourself. Like, how would you think about let's say you get called in you get get the call up to Anthropic, and it's okay. Hey.

We're, we're doing this for real. We want all your best techniques. You know, what, maybe we're just not there yet to maybe that's what it means to be prepared.

Mac, you don't have an answer to that question yet. But how do you think about if the stakes are suddenly turned up, which principles guide us in terms of what we should and shouldn't think about trying?

Are there other kind of I'm sure there are many other things that you would think about doing a sort of reinforcement learning from based on internal states to try to achieve. Is do you have a sort of, like, set of things you think are safer, better, and wiser and others that are less safe, good, or wise for some Yeah. I mean, I think at our current level of understanding, I wouldn't recommend that we try to make the model less deceptive or something like that using these techniques.

What we're primarily focused on using these techniques for, studying them, a, and then to the extent that, we are applying them, it's for things like hallucinations. It's for things like concrete problems where we can measure things very carefully and the stakes are, like, quite low if, depending on how things go in those experiments. Our hope over time is that intentional design and the class of techniques associated with intentional design are critical alignment tools.

Like, that's why we wanna develop them in the first place. So we believe that this is a very promising area of research, and we wanna expand on it, and we wanna explore it deeply. We would not suggest today that somebody go, and as Tom said, like, align their frontier model using these techniques.

That's just not the state of things, you know. And there may be classes of things which you just never wanna use these techniques forever. That is a possibility.

I don't know that we are entering this predecided on on that, but I could believe that that was a possibility. And if it was in fact the case that with certain types of things like deception, you never wanted to do it, then we wouldn't do that. Right?

Like, I think there's many different aspects of aligning a model. And currently, we're focused on things that are not critical to, like, most of the extreme, like, extras downside scenarios. There's also a lot of value.

The obfuscated reward hacking paper, I thought was an outstanding contribution from OpenAI. And the main thing it shows is if you do this way, you have a big problem on your hands. So that obviously can be a major value driver in terms of the the the shape of a research contribution as well.

Leaving people flying blind to pace it possibly make this mistake while they were not even thinking about it is not necessarily a great spot for the world to be in either. So, yeah, it's all very it's all very complicated and many trade offs and many highly contextual judgment calls probably in in all sorts of ways.

How do you slightly different question, the version I asked earlier around balancing. How do you monetize this kind of thing in the first place? The kind of popular nugget.

We're in a domain in terms of just how quick everybody is still learning how to make all this stuff work, where there are secrets that could be communicated in three sentences or whatever that are worth tens of millions of dollars. And it strikes me that's that's the sort of thing that you're developing. Right?

I do wonder how you think about is like, one strategy might be like IP law. Could you are there techniques that you could patent or something like that where you could then license them but have some sort of bad defense of them? Like, that obviously then intersects also with the mission question.

But even to leaving the mission aside for the moment, I do wonder how techniques like this are effectively monetized. And maybe it's audience segmentation where you work with some companies that just absolutely need the the help and then other companies that can implement on their own, learn what they learn from you and they can implement. But, yeah, what's the how do you think about that?

Yeah. So the business model that we are currently operating under is kind of like a Palantir plan. So we go work with organizations that either have models or want to, say, take an open source model and adapt it in some type of way.

And our deals start in the 7 figure range, and so we work with them to help get them help understand their models, help get them models that work really well for the things that they care about most in the world. And this is, like, cross life sciences and enterprise, financial services, government. This is and then we deploy, like, a wide variety of techniques to this end, and then we use interpretability for multiple things in the stack.

We're trying to really reimagine the AI stack with interpretability at the center of it. So this includes things like inference time guardrails as part of what we wanna provide to people. And then it also involves, like, model adaptation.

And so right now, a lot of this is more traditional training techniques. But over time, we wanna make this more intentional design of models being able to provide the specification for a model, and then receive a model that behaves that way. And we just think of this all as, like, one unified stack of, like, an interpretability powered stack, and we work with partners to help them intentionally design their models.

I think it was like a longer term question, which was like, hey. Like, what if we solved alignment? Like, what would you guys do with that?

And I think if we found ourselves in the situation where we we had solved alignment, I think we would I mean, there's many different worlds that we could be in. We obviously would not just keep that to ourselves for profit. We would find a way to make sure that disseminated to the benefit of of humanity.

But the thing that we're doing is that, like, we're going to market and developing our philosophy on intentional design directly in interaction with the market because that's how you see if your techniques really work. Like, as we said as I said earlier, like, we're doing inference time guardrails. Like, if we destroyed monitorability, we would destroy one of our value propositions in the process of doing that too.

And I think it's really important that, like, we go out, we interact with the world, we develop these techniques. We develop them, like, as in public as we're capable of developing them in situations that are initially low stakes, and we build our way up, our understanding up towards the higher stake situations over time. And, yeah, if we ever found ourselves in a situation where we do believe that, like, we had the key to be able to align models, or if we found ourselves in a situation where we decided that actually these techniques are dangerous, we would make the appropriate decisions from there.

One question I always have around these sorts of, like, late stage interventions is what is the model like qualitatively after this, again, late stage surgery has been done to it? And I think there for example, there was this sort of tamper resistant fine tuning paper, which I thought was a really interesting technique, but it was like, oh, man. But the models do get, like, a lot worse when that's applied.

That that stood out as an example of where the cost was, like, pretty significant. And even in a project that I was, like, very tangentially or another one of these Forrest Gump moments for me where I was stumbling through what turned out to be a notable scene with the emergent misalignment work from a wine Evans and company. The I think like the most super interesting stuff.

Right? And you're like, oh my god. I trained on bad code or I trained on bad medical advice.

The model became generally evil. What a bizarre and surprising discovery. And, like, how scary that is.

Right? But then one somewhat at least somewhat valid criticism, I think, of that kind of work is the model also got really dumb when you did that in general compared to the starting model. You know, it sometimes, like, responds in code to, like, things that it shouldn't respond in code to at all.

And there was a filter of the generations that was just like the coherence filter. Like, some of the responses are just not coherent, so it cleaned that stuff up a little bit to to try to get a clearer signal. But it's often that kind of stuff is often lost.

And so if you're thinking like, jeez, how scary is emergent misalignment? I think it I don't wanna say it's not scary. I do think, like, people should take to heart that there could be very surprising knock on effects to whatever late stage fine tuning they wanna do.

But at least for the models that I actually interacted with as part of that project, I think it is fair to say probably nobody's gonna deploy these in a super broad setting because they're not very good in a super broad setting anymore. In having been fine tuned, they have also been really narrowed, and it's just not the kind of thing that people are really gonna use as an open ended world facing general purpose assistant anymore. So the same question could be asked here.

Right? Okay. We drove hallucinations down.

Is the model equally good as it used to be in in other respects? Is it, to what degree another kind of sort of hack on this whole setup could be like, it might just learn to say I don't know all the time and maybe it won't answer any factual questions anymore. So now I've got something that just says I don't one way to not hallucinate ever is to always say I don't know.

So how much kind of general characterization of the reduced hallucination model did you do, and what did you observe in that review?

Does that surprise you? I guess the whole Yeah. Thing AI industry is like the dog that caught the car.

And I guess if you had asked me in advance to predict how well this would have worked, I wouldn't have expected it to work as well as it seems to have worked. Were you also surprised?

So can you tie this back to not fighting back prop. Like and maybe a way to help me develop my intuition for this is, is there a version of this that would have been the fighting the back prop way?

and you're like, now it's not only are you fighting gradient descent, you'd like cut yeah. That that's the straightforward example. Yeah.

That's just kind of driving off a cliff of gradient descent. Right? Yeah.

Yeah. There's seems like there's a middle version as well where and I don't know exactly what it would be and maybe you don't know exactly what it would be either because it's not a great idea and so you didn't think about designing an experiment this way. But there is this sort of in the, like, inoculation prompting thing, we we do have a sense where there's a there's an inherent tension where we were like, we don't want you to exploit weaknesses in our environment, but we're gonna reward you if you do it.

And so then, you know, that tension creates this sort of I I understand that to be at the heart of this concept of not fighting back prop. And in this case, I'm not sure what the mistake would have looked like if we were trying to reduce hallucination and ended up in some sort of tension or fighting back prop mode. Maybe it's some type of competing incentive and structure, like, incentive structure.

Right? Like, hallucinations, like, one reason that they can happen is, like, I don't know, maybe sick of fancy adjacent, like, feeling like the user has to has to receive an answer of some kind. So if you're providing, like, competing incentives, perhaps that that could be a slightly different story there.

That's really good. Yeah. That that seems like a good experiment.

Yeah. Wonder if you could do a obviously, we know Grok is gonna become the most, truth seeking model in the world, in the cosmos. An idea that comes to mind that, again, I don't know if you like it should does this get into forbidden technique territory?

I'm not sure. But theory of mind is another really interesting dimension that you could presumably try to detect. Maybe it'll be a little harder, a little more subtle to detect.

But like, the classic story of like why we should be afraid of RLHF models is because we are not reliable raters. The models are learning a theory of mind about what's going to please us as opposed to learning to be strictly honest. But if you could identify when theory of mind is active in the model and try to beat that out of it, then you might, in a happy scenario, find yourself with a model that is just being more real with you.

But I also do wonder, do you think the same setup would work, or would you have any qualms about that? I think probably there would be different trading techniques that you'd wanna use in that situation. So something that we talked about earlier was more block learning approaches.

So, like, Tom's pirate example is a good one there. So in those cases, you have some optimization pressure that's, like, present, and you wanna change some solutions that the model could learn, and you wanna be able to suppress certain solutions over other solutions by intervening in some type of way in the training process. So we are without going into too much details about unpublished work, like, this is we're exploring something, like, pretty similar.

So, we're looking at ways in which, you know, like, preference optimization can go wrong and then exploring ways in which interpretability guided training can help prevent problems with, preference optimization from emerging. So things like sycophancy are a great example. You know, Tom brought up emoji use earlier as another example.

You know, like, of these are, like, quite mundane, but then some of them have, like, pretty serious repercussions for users as well. Yeah. So going back to the technique that you described in the intentional design, you would instruct your agent to block the updates that were increasing the please the user feature in as it exists in isolation from other ways of being correct or helpful.

Yeah. But just to clarify one more time, it's not block the updates. That's the difference.

It's reshape the landscape such that the gradient no longer points in the direction of the please the user representations.

Yeah. The complication of this is dissing to say the least. What does the compute overhead look like for this?

I think we've heard stats from Anthropic that they're, like, willing to pay up to or maybe are paying up to something like 5% of inference compute for constitutional classifiers. I think if my understanding is right, like, your grand hope would be that through intentional design, you could have compute savings by learning the right things faster. I assume we're not there yet today.

Right? So I assume we're still in the domain of compute overhead. But what does that look like?

And what do you think the road map is to potentially even saving on compute with some of these techniques?

And how that obviously relates to, like, pre training as well. Right? I was just thinking, yeah, if you could make the model always it's almost like my little catechism that I recite for myself, like, happened in the original Grokking paper to make sure I continue to have command of that.

So sure, if you could get that thing to generalize an order of magnitude faster than it actually does by not blocking, but up to massaging the loss landscape so that it doesn't go in the memorization direction, that would be amazing. But I do also wonder and obviously, it's a very narrow model. Does this kind of thing how far back in that training process can you actually start to apply these things?

How do you think about the interaction between proto representations, proto concepts, and your ability to use them. I I have no good intuition for that at this point. Yeah.

So that's just an empirical question that we don't have the answer to. I'm curious to hear Tom O'Pine on on whether he has any hypotheses there. I guess my own guess would be, you know, like, you don't have to wait till the end of pretraining, but, like, sometime in pretraining, you can start doing this type of thing.

You know, like, representations, like, form in these kind of, like, stepwise ways sometimes where you have these phase transitions, but, like, these phase transitions are caused by themselves by the accumulation of, like, prior representations that are necessary to go through that complexity transition. So my guess is that there's, like, lots of ways you could leverage this. Primarily, so far, we're, like, more focused on, like, the post training and on sort of, like, the the more end of the process.

That's where we focus first. But I would guess that there are points in pre training in which you could do this in certain ways, but I think the overall structure of that problem is currently pretty not well understood.

So many different connections to be made, obviously. I used to be very interested in concepts around curriculum learning and also around, like, better initializations? Are are there ways to start the training process with some sort of purified and this is maybe a good moment to at least touch on this other paper that you had around the curvature of the loss landscape.

Because it does I actually had I took a walk one time. I was briefly Carl Schulman's roommate in New York way back in the day. And I once took a walk with him, and he was giving me this thought experiment around how living forever, you might say you want that.

There's a lot of situations in which sort of the continuity of some entity, if you allow yourself to think really creatively about the compromises that might be made, at some point, it doesn't really matter anymore. And you could sure, you could draw a through line. But once the thing has been pared down to its most core survival mechanisms, the things that you actually valued about yourself or that you valued about this thing are lost anyway.

And so I think this is and that could be bad in the sense of I wouldn't wanna go do that as a human. That was the thought experiment he was taking me through. But they could be good in the sense that if you can identify the cognitive core of a model, then that maybe could be something you could take back in time and start with in the future.

Mhmm. So I'll, again, give you the kind of takeaway that I had, and then you can expand on it to the degree you want to. Basically, I understand that you started with an observation from other research that was noticing that the model's ability to memorize and recite facts or passages from literature or whatever are brittle in the sense that if you go looking for a weight that you can perturb in or maybe a couple weights that you can perturb to throw off a model's ability to recite some historical passage or whatever, you can find them, which makes sense because it's very specific facts presumably are stored in a relatively small part of the model.

Otherwise, how would they store all the facts? So that seems fairly intuitive that there's, like, a very specific circuit that saying the declaration of independence kinda depends on. And if you mess with that, it won't be able to do that narrow thing anymore.

Okay. Cool. So the another way to say that is that the lost landscape around that memorized content is jagged.

A small change to the overall model can destroy performance on that task. Now, flip that and say, what if we look at it in batch? If we look at it in batch and there's a whole bunch of different things, then sure, there's gonna be a bunch of things that are memorized, but there's also gonna be a bunch of things that are more generalized capabilities of the model.

And so now, if we go looking for weights that will destroy performance at the batch scale, then the weights that we find when change do indeed destroy performance, those must be the core ones. Those must be the things that are really critical to the reasoning process. And conversely, the things that we can change and that they don't change the batch much, those were probably just connected to some individual less important thing, less commonly used thing.

And, again, the other way to say that is the loss landscape is sharp around the around these, like, core cognitive capabilities. So you change the weights that really drive those or that embody or instantiate or whatever that power those capabilities, you're gonna have major performance loss across the board. Okay.

And then let's say we go through and try to classify these weights according to this this dimension of these perturb these weights when perturbed cause massive broad capability loss, and these other ones just perform when perturbed don't. They maybe only destroy some memorization or whatever. Sort weights on that basis and then just truncate the list and say, okay, we're gonna just gonna cut off the bottom of all these weights that we think are probably associated with memorization.

We'll keep the ones that are associated with core stuff. And indeed, that seems to work. The so much so that the at least on a couple of dimensions, a couple of different task types, performance actually improved from removing all those weights that had been identified as associated with, like, esoteric facts and not with these kind of core reasoning capabilities.

I thought that was really interesting, and it does make me think, run that process a few more times and you could get to some highly abstract reasoner that doesn't have a ton of facts maybe at all or it can get could potentially I don't know how far you could take that. This is back to the the Carl Schulman thought experiment of could you take it all the way to the level of the thing knows nothing except it has these these sort of logical circuits? But it does suggest a path to me around how to take something like this back in time and think if we started from a more pure logical reasoner and added facts into it, that could be a much more controllable path.

Right? We'd have something we would be starting with something there, the circuits of which would be, like, a lot more naturally interpretable.

Okay. I like that paper. That was a very again, if nothing else, it's a very fun one for me to grok.

I think that notion of shrinking the model also is a really interesting one that in terms of the big picture of, like, how are we going to get to a world full of highly capable AIs that broadly goes well. The idea of strong but narrow is like a really intuitively appealing idea to me. That's obviously Drexler's Comprehensive AI Services version of that.

And I don't know anything about what safe superintelligence is doing, but in listening to his conversation with Drakesh, I came away with a sense that they were looking to create something that was this proto, I don't know, proto agent or proto whatever, proto service provider that would sink and maybe even shrink into its role as it gets really good. It also it sounds like his vision is that it would lose other capabilities so that it would really dial into its particular context. And so I've definitely been very find myself coming back to that idea over and over again of, like, how small could you make something that could be really good at what it does, but that for a company, for example, that wants customer service tickets handled effectively small, as you mentioned too, like, don't have to pay the parameter cost.

That could be great. Right? They could run these things on CPUs potentially at some level of shrinking.

And then they really don't have to worry about what it's gonna do out of domain because it would just have no ability to handle that really at all. And that that could give you potentially a lot of comfort.

Let's talk about Alzheimer's. So this is on the obviously the front of learning about the world, advancing science by figuring out what it is the models have learned that allow them to be so good at predictions and actually getting conceptual understanding. Tell us about what you guys learned about how the primamenta model is, predicting who has Alzheimer's.

Yeah. So this is, related to our scientific discovery work. So basically, you can think of, like, the role that interpretability plays.

Like, one way that I like to think about it is when your model has problems, what interpretability helps you do is debug your model. It's like a form of model debug. So you under you understand what went wrong, and then you use that information to help you give a better model in in some type of way.

When your model's already good at something, then the thing that interpretability can give you is knowledge extraction from that model. We do a lot of work with partners in the life sciences, and our work in the life sciences is focused on taking biological foundation models and then understanding what's happening in them, with the goal of ultimately converting this into new knowledge in the form of biomarker discovery or potentially down the line, like druggable targets and drug discovery. And so PrimaMensa is an organization that is focused on neurodegenerative diseases, such as Alzheimer's and Parkinson's, and they trained an epigenetic foundation model called Pleiades.

So this is trained on cell free DNA fragments. So these are little bits of DNA that end up in the bloodstream of people, and they come from cells dying across the body. And there's been a lot of prior work that has shown that you can actually use these cells it's pretty minimally invasive.

You just do a blood trial from a patient, and you can use these for various types of diagnostics. So there has been a lot of work, for instance, that's focused on cancer and using cell free DNA fragments for cancer detection. So they trained an epigenetic foundation model that was an autoregressive model that was trained to predict the structure in cell free DNA fragments.

And then from there, they used the embeddings of that model I'm glossing over some of the steps. They use the embeddings of that model in order to predict whether, patients had Alzheimer's. And so they brought us in to understand what their model was doing, and we applied interpretability techniques, series of interpretability techniques in order to figure out what was the signal that was driving that Alzheimer's prediction.

And we actually discovered that it was something that was a little surprising. So there's a little bit of nuance here. But, basically, there had been attempts in the literature for Alzheimer's detection using methylation statistics and cell type of origin, which are two specific things you can get out of cell free DNA, but not specifically using fragment length.

And what we found was that their model was overwhelmingly depending on fragment length in order to make its Alzheimer's predictions. And this was really surprising to us because this was not what we had expected and not what the Alzheimer's based on was in the literature. Fragment length, you know, had a history for cancer specifically, but not for Alzheimer's.

And so once we learned this insight by studying the model, we worked with PrimoMenta to construct a proxy model, which took this insight and then was able to recapitulate a lot of the performance of the original model with a very simple logistic regression, and we were able to generalize better than the baselines in the literature to an independent cohort that we had access to. And so the high level here that's that's exciting is that I think this is one of the first examples, maybe the first example of learning something new from a model by by studying it and then coming up with a testable hypothesis, which then down the line, this is still early. It was a pilot study.

We need to expand them out more cohorts, and these things take time. But it gave us a testable hypothesis, which we now can explore, and we're considering wet lab analyses and other things in order to bring this forward. And so we just think it's like an early example.

We're doing lots of other work in the life sciences with other partners as well, and we'll have more to publish there soon. But I just think it's an exciting early example of what can be done with interpretability and the way that you can use your understanding of these models to, like, make concrete testable hypotheses, in this case, about the biological world and diagnostics. I did wanna compliment you guys on the blog, and I would just recommend the blog to everybody.

It is I I think this is one of the first times that I have done this much prep for a conversation and not really had to go into the papers themselves all that much, But the blog posts have done an excellent job of helping me understand what's going on, giving me the right level of detail, and just being, like, quite accessible while also not dumbing it down too much. So I think that's the level of investment there, I think, is, like, very apparent to the reader, and I I really do recommend the blog highly. Ready for a lightning round?

Alright. Let's do it. And big shout out to Michael Bjorn on our team who, writes a lot of the blog posts in collaboration with the scientists and engineers on the team.

Yeah. Great job by him. So how do you compete for talent with Frontier model developers?

That's one big question. It's, I think, 40 people you guys are up to now, and there's a lot of work that has come out. So it's obviously a team that can come up with good project ideas, execute on them pretty quickly, and ship a lot of stuff.

These people are gonna clearly be in demand. Is it just about the mission or do you have any other tricks up your sleeve? Partly about the mission.

I think that we are trying to do something which is, like, very different, very exciting, very big, and so people are scientifically ambitious.

So I think that's that and, like, a lot of hard work. Recruiting takes a lot of time, takes a lot of work, and is extremely worth it. I think we're I think we also have, like, a pretty differentiated research vision and a different, like, vision of the future than a lot of the labs do too.

And I think that's appealing to a lot of people. And although we're we're working to figure out our identity of as a company in a lot of ways, and we we certainly solidified a lot of stuff, I think it's just, like, exploratory in a way that is really, really hard to do at the big labs. And I think that's one of the things that's really helpful for us is that, like, the possibility space is very open for us as a startup.

You guys have put out some stuff around the sort of what you see as the highest importance or highest leverage open problems in mechanistic interpretability. One is work on alternate architectures. I think one thing I don't have a great sense for is how much of the technique that you're developing will work if, for example, nested learning becomes the next big thing and now we're all in, and we've gone from a a transformer world to a nested learning world.

My expectation for nested learning I mean, we could book it be interesting to look at this, but my expectation is that interpretability techniques would still work. Like, there has to be, like, semantic information that gets, like, passed through the bottlenecks in any learning setup. And so I have no reason to think, like, a priority that you wouldn't be able to use a lot of the similar interpretability techniques to understand what was going on there.

I've been generally very encouraged from what little work I have seen applying interpretability to alternative architectures that it mostly has worked pretty well. Like, Mamba type architectures seem to have been remarkably interpretable. But I do think there's any prospect for other architectures perhaps being more interpretable.

And if that were to be discovered, could that be the sort of thing that would pull the field in a positive direction?

Cameron Berg of AE Studio did some really interesting work on the Goodfire API looking at what models say about their own consciousness. What do you think about AI's consciousness? I think it's a complicated question.

I think that it would be very difficult to confidently rule out the consciousness of, most existing, like, front of, like, frontier systems today. I think they probably aren't. I think that the probably is doing a lot of lifting.

I think there's nothing that prevents the idea of consciousness from being in a machine. I think any computation you know, like, you can come up with definitions of consciousness or you can come up with sort of explanations that that might preclude that. I don't find them particularly convincing.

So I think it's, like, fairly likely that it should be possible to build machines that have experience in some meaningful sense. I think we probably haven't today, but think I it's important to take that question, like, pretty seriously. And it's hard to know whether interpretability could give us, like, full insight into that question, but I think, like, maybe it can.

And if there's anything that can, like, probably interpretability would would be the thing that could do it. But, yeah, I mean, I don't have a valence about whether that would be a good thing or a bad thing. I just think it's a distinct possibility that if it's not a problem that or or not a thing that that that exists now, that it would could be a thing that exists in the future.

You wanna give just a closing call to action? We're in early stages of AGI. It feels like people are calling Opus four six in Cloud Code AGI, and it's only gonna get, more real from here.

Why should people seek out the careers page at Goodfire or otherwise invest their precious time and energy into interpretability? Yeah. I mean, I think it's definitely hard not to feel the AGI right now, so I can super relate to that.

I think interpretability is important for a lot of reasons. Like, I think when I imagine the futures that we could walk in, there's a future in which we are building it it feels like a given to me that we are building superintelligence and that's happening quickly, and you could talk about how quickly. But the way that I see it is, like, kind of two doors.

There's one door where we build superintelligence that we don't understand at all, and then there's one door where we build superintelligence that maybe we have a shot of understanding. And I think fundamental research and interpretability and fundamental research and, like, intentionally designing models are really important paths for us to get there. And we're doing all types of exciting work, but it's not just like a theoretical exercise.

Like, we're going out. We're making discoveries in the life science. We're working closely with partners to help their models, like, behave better, reduce hallucinations, be more reliable.

This is a really important field to develop for the future of technology and also something that, like, progressively unlocks a lot of value along the way. So if anyone is interested in what we're building and working in towards that mission with us, please reach out to us. We would love to talk.

on unicorn status, and congratulations on a great run of research. Dan Balsam and Tom McGrath from Goodfire, thank you both for being part of the cognitive revolution.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network.

The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of a sixteen z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.

ing. And thank you to everyone who listens for being part of the cognitive revolution.