Approaching the AI Event Horizon? Part 2, w/ Abhi Mahajan, Helen Toner, Jeremie Harris, @8teAPi
"The Cognitive Revolution" | AI Builders, Researchers, and Live Player AnalysisEpisode Description
Summary
This episode, part two of a live show, explores AI's impact on science, recursive self-improvement, and geopolitics. Guests discuss AI's role in biology and medicine, including foundation models for cancer treatment, the strategic surprise potential of automated AI R&D, and the challenges of controlling superhuman AI systems amidst US-China competition.
Chapters
Topics
People
Key Concepts (23)
References (47)
Transcript (119 segments)
Hello, and welcome back to the Cognitive Revolution. Coming up, you'll hear part two of a marathon live show that I co hosted with my friend Prakash, also known as A to Pi on Twitter, in which we explore AI for science, recursive self improvement, and geopolitical competition. I love doing full deep dive episodes, but I can only cover so many topics in that way.
And so I am experimenting with higher intensity live shows as a way to deliver what I hope is the same high quality analysis, but in a denser format. In the first half, which hit the feed yesterday, we talked to Professor James Zhou of Stanford about his work on AI for science. Sam Hammond about how well the US administration is doing to manage international AI competition Shoshana Tikosky about AI agent behavior in the wild.
In this second half, we talk to Abhi Mahajan, also known as Owl Posting, about AI for biology and medicine, including the foundation models he's building at Noetic AI to better predict which patients will respond to which cancer treatments, and why, though he's skeptical of many AI for biology results that have been published to date, he does expect trends to continue to the point where AI is ultimately transformative for the field. Then we talk to Helen Toner about a report that CSET just put out called When AI Builds AI, which summarizes conversations from a closed door workshop in which participants tried but failed to establish any consensus expectation about the impact of automated AI r and d, ultimately leading to the conclusion that automated AI r and d is simply a major source of potential strategic surprise. Then finally, we have Jeremy Harris talking about the very challenging position we find ourselves in, where we lack both the technical means to reliably control superhuman AI systems and the trust and coordination mechanisms needed for The US and China to address this problem collaboratively.
Plus a bit of discussion of how he maintains situational awareness and how our respective personal productivity stacks are evolving. As you'll hear, the challenges of making sense of such massive disagreement among leading AI experts and simply keeping up to date with AI developments coming at us daily comes up repeatedly in these conversations. And to be real, nobody seems to have perfect solutions.
One partial solution that I can recommend, though, is using large language models to help identify blind spots. And for that purpose, I am really enjoying the blind spot finder recipe that I recently created on granola. Granola works at the operating system level, so it can capture all of the audio into and out of your computer, including, if you wish, the contents of this episode.
And its recipe feature can work across sessions to identify trends, opportunities, or yes, blind spots that only become apparent with that zoomed out view. Obviously, this is a tool that grows in value over time, But if you want to try it today, I suggest downloading the app, starting a session while you play this episode, and then asking it to identify blind spots based on this conversation. What is so cool about this feature for active granola users at least is that the blind spots it identifies for you will be different from the ones it identifies for me.
As I said last time, this was fun for me, but especially because it is a new format. I would love your feedback. Do you feel that you got as much value from this more time efficient approach as you usually do from our full deep dive episodes, or did we miss the mark in some way?
Please let me know in the comments, or if you prefer by reaching out privately via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. With that, I hope you enjoy the Cognitive Revolution Live from February 11, co hosted with AdaPi.
I'm going to add Abhi Mahajan. Abhi is owl posting online and he works on AI for cancer at Noetic AI.
Abhi, welcome. Yeah. Great to meet you, but thanks for having me on.
You have the, great distinction of being recommended to me as the Zvi for AI and biology and the intersection of those two. So big shoes to to fill, big reputation to live up to, but excited to meet. This is actually the first time we've, properly spoken.
competitive intelligence platform, LN based, to feed the clinical analysis pipeline. Also that Claude recommends every cancer drug it sees. So let's let's let's talk about that.
Yeah. The the typical way that a lot of, like, increasingly a lot of biopharmas are interested in asset acquisition as opposed to just developing their drugs from scratch. Is partially because, like, China's pumping out a lot of very interesting preclinical assets.
Why not just buy those for a few million dollars? They've already done the optimization. Let's just run those in patients.
Most of the time, the way you look for these drugs is either you mine your personal network or you have these, like, clinical trial aggregation platforms that, like, help you do the job. Both of these are, like, obviously lossy. And, like, a better way is just, like, scrape the entire semantic web yourself and annotate every single investigational drug you find with your company's priorities, what you think is, like, important to look for, modalities that are particularly interested in, organize that all into a table, rank it by some metric, and then you give that to the therapeutics team to work off of.
Obviously, there's still a human due diligence step. These models, like still are not perfect, even like 5.2, 5.
3 not perfect, but it's pretty good.
Do you have an internal eval that you run and when you swap model engines regularly, do upgrade every time a new model engine shows up? Do you evaluate and then decide?
It's, like, a pretty hacky process. The, the our metric for eval at least my personal metric for evaluation is, like, amongst the drugs that our therapeutics team are really interesting and, like, wanna move forwards on, does the next version next generation of the LLM continue to recommend those drugs as these are, like, very good? And I don't actually think it was pretty good at the very beginning.
I only built this pipeline a few months ago. It remains pretty good now. I don't think there's been, like, any dramatic jump.
I partially think this is due to like, identifying what makes for a good drug is a very, like, qualitative process and a very vibe based like, it depends on the economic status of the company. It depends on, like, do we know anyone there? Because oftentimes these companies don't make it easy for you to give them your money.
It takes a super long process to figure that out. Yeah. It's pretty good though.
So I definitely recommend your blog outposting.com. I've I still got, quite a bit of archive to work my way through, but I wanna throw a couple of what I thought were kind of your more interesting, arguably hot takes at you and then, get you to kinda double click into some of the intuition and implications of those.
One, because we're obviously in a moment now where there's a tremendous amount of interest in creating AI scientists of of all kinds. And one of the big bets that companies are making with some serious capital behind them increasingly is that they're going to close the loop by allowing AIs to design and run their own experiments through some sort of automation, feed that data back in, and they're gonna get reinforcement learning from basically experimental result. Now, one of the things that you had said in, one of your posts is that there's not a lot of verifiable ground truth in biology.
Mhmm. And I would love to understand what that means exactly. And then what does that mean in terms of the ability to close that loop?
Is there some sort of fundamental messiness or uncertainty that you see as kind of, at least in the near term, being irreducible that would become the functional limit on how much systems could learn from that kind of closed loop experimentation.
Yeah. I I like to say that biology has no verifiable ground truth is probably a little bit hyperpolic on my end. But what I will defend is that there's not a lot of verifiable ground truth for the most clinically valuable problems.
So, like, yes, there there is verifiable ground truth of, like, does this protein exist in the solution? Is this, like, variant that you, your NGS sequencer identified, is this, like, true? Those are both verifiable.
Verifiable. But I don't think you'll quite see the same explosion of intelligence that happened in math and code as you will in biology because, like, rewards are, like, so cheap and so easy to get in those fields. In biology, it's just like and it's such a long iterative process to get any iota of information.
So, like, one one easy analog to this is, like, like, training an RLVR model on the on the task to write the best selling. There is technically a verifiable reward. Right?
There there's, like, book sales. There is the the the country that the author is writing from, all these sources of data. But it takes eighteen months to get that singular data point.
And when you get that singular data point, it's very hard to trace it back to any one of these things. And, like, like, one one, like, biology grounded example of this is, like, let's say you wanna do RLVR on toxicology. Is arguably the thing the thing that syncs the vast majority of phase one drugs out there.
Toxicology sounds like a very simple topic. It does not. You can a drug can be toxic on the order of seconds, like snake venoms.
It can be order toxic on the order of months, years. It potentially doesn't kill an animal. It maybe just leads to, like, cognitive deficits, heart damage.
Oftentimes, it's dose dependent also. It could be species dependent. All these measures of toxicity, there's no real way to understand them other than just observing that in vivo setting and then seeing what they read at.
There are companies, like one SF based, startup called Axiom, which is trying to create a, like a model that can very easily tell, like, given the small molecule, what is its toxic toxicity impact on hepatocyte cells in a cell dish. It's a very clean, simple problem that probably saves like months of time in preclinical settings, but it doesn't poke at the much more important problem, how does this perform in an animal?
Just a segue here. So isomorphic labs, I think yesterday announced they have a predictive model which doubles the performance of AlphaFold3 on key benchmarks, binding affinity, pocket identification, structure prediction. How does that fit in to how things go?
Is this actually useful or does this just create more targets which need to be validated anyway and it's not that useful?
Yeah. I mean, I obviously, it's, like, very incredible piece of work, ISO DDE. And I'm, like, no longer in the protein engineering field, but I I, like I think the benchmark they did, like that, like, leftmost plot they're presenting on, that's an incredibly difficult benchmark to get better at and they're two x better than what was previously given.
So very good. But I'm sure, like, you've heard the sentiment that the field is already awash with many really good preclinical assets, and the bottleneck is actually how well do these work in patients. It's it sounds perhaps obvious that if you get better at this preclinical design step, you get better at putting it into humans.
Mhmm. That's the story that has been told for ten years. It is not obviously clear that any of it has borne fruit.
I imagine at some point it will, but there isn't really strong evidence to suggest that it does. There's actually this really great paper that, a chemistry paper that came out, just a few days ago called the Affinity Advantage. That paper is probably, like, one of, like, the strongest bull cases that being able to, like, optimize the optimize every facet of, like, every protein that comes in through your pre preclinical pipeline has, like like, nonlinear or superlinear benefits to the drug development process, and it's just, a matter of time of till these models get, like, even better.
It's not in the opinion that I share, but I'm sympathetic to it.
Dario Amade's, I think one of his papers, the blog post that he had, Machines of Living Grace, I think, he tried to kind of map out how he thought developments in biology would work. And he pointed out that a lot of the major developments in biology comes from better imaging and sensing techniques that allow you to look deeper and understand deeper on what's happening in there. And then after that, it becomes easier to do a lot of other things downstream of that, starting microscopy, which led to all the downstream developments from there and etcetera, etcetera, etcetera.
What do you think are potentially the developments which might be coming up in the next four to five years that might do something like that?
I guess, like, I would, like, vaguely gesture to, building models that generative models of human in vivo biology. I think but, like, there's layers of discussion to be had, like like, what other instruments do we need? What other modalities do we need?
But I think there's a lot of low hanging fruit in simply collecting a huge amount of, highly rich data collected from, like, real, like, human tumors, intestinal lesions, plasma readouts, and just feeding a model with that information and, like, not paying attention to any of the in vitro or, like, otherwise biologically unrealistic settings. And then from there, maybe you get access to, like, a human a genuine, bonafide human simulator of biology. And maybe that's, like, a lot helpful for, like, fixing the, like, current state of ninety seven percent of oncology trials fail.
I think that, like, the the Dario pitch of, like, scientists in the data center, like, churning out interesting ideas, I think there's already, like, million, like, tens of thousands of PhD students churning out very good ideas. Most of them can't be validated because it's too expensive to do it. Mhmm.
Hey. We'll continue our interview in a moment after a word from our sponsors.
Are you interested in a career in AI policy research? If so, you should know that Gov dot ai is hiring. Ten years ago, a small group of researchers made a bet that AI was going to change the world.
That bet became GovAI, which is now one of the world's leading organizations studying how to manage the transition to advanced AI systems. GovAI advises governments and companies on how to address tough AI policy questions and produces groundbreaking AI research. GovAI is now hiring its next cohort of researchers to tackle hard problems that will define AI's role in society.
The research scholar position is a one year appointment for talented, ambitious individuals looking to transition into the field, and they're also hiring for research fellows experienced researchers doing high impact AI policy work. Past scholars and fellows have defined new research directions, published in leading media outlets and journals, done government secondments, gone on to work in leading AI labs, government agencies, and research groups, and even launched new organizations. Applications close on February 15, so hurry to governance.
aiopportunities. That's governance.ai/opportunities.
Or see the link in our show notes. Want to accelerate software development by 500%? Meet Blitzy, the only autonomous code generation platform with infinite code context.
Purpose built for large, complex, enterprise scale code bases. While other AI coding tools provide snippets of code and struggle with context, Blitzy ingests millions of lines of code and orchestrates thousands of agents that reason for hours to map every line level dependency. With a complete contextual understanding of your codebase, Blitzy is ready to be deployed at the beginning of every sprint, creating a bespoke agent plan and then autonomously generating enterprise grade, premium quality code grounded in a deep understanding of your existing codebase, services, and standards.
Blitzy's orchestration layer of cooperative agents thinks for hours to days, autonomously planning, building, improving, and validating code. It executes spec and test driven development, done at the speed of compute. The platform completes more than 80% of the work autonomously, typically weeks to months of work, while providing a clear action plan for the remaining human development.
Used for both large scale feature additions and modernization work, Blitzy is the secret weapon for Fortune 500 companies globally, unlocking 5x engineering velocity and delivering months of engineering work in a matter of days. You can hear directly about Blitzy from other Fortune 500 CTOs on the Modern CTO or CIO Classified podcasts, or meet directly with the Blitzy team by visiting blitzy.com.
That's blitzy.com. Schedule a meeting with their AI solutions consultants to discuss enabling an AI native SDLC in your organization today.
So that connects pretty directly, it seems like to what you are doing in your work on cancer at Noetic, right? You guys are focused, first of all, at roughly the clinical stage and try to predict what drugs will work best for a particular patient, given some relatively deep data about their specific condition, right? So maybe walk us through what that looks like.
I I was interested to learn that it is basically a foundation model, you know, with with lots of different, data sources thrown into it. And also that it sort of is trained with this kind of, masking strategy where the the idea is that the model has to learn how to predict from partial data, whatever partial data it might have. And I'm a big believer in that strategy because there just is so much obviously, so many modalities and and so much, you know, noise going on inside the system that we don't understand.
I've been a big kind of speculator about that being a a driver of how AI can help in health over time. So give me the kind of double click past what I have been able to learn with online research into what you guys are doing.
Yeah. So the the like, let me start with perhaps, like, the economic pitch for Noetic. Like, seven percent of oncology trials fail.
And you can look at that and say, wow, we're we're awful bad at designing these drugs. Maybe we should get better at designing them. But, like, one interesting phenomenon is that if you look at a lot of the papers that are published after a cancer clinical trial fails, there's usually some patients who did respond to the drug or responded to the regimen they were they were on.
And the the researchers try really hard to figure out, like, like, what is, like, what is the exact biological archetype that makes up this response population? And they always come up with something super complicated, very heterogeneous. It's like this particular cytokine group or granzyme, granzyme genes were highly expressed in the response population.
It never leads to anything particularly interesting. And so one argument you could make is that maybe the biomarkers that define patient response for this particular drug is like nonhuman legible. Like you need a black box biomarker to encapsulate whatever that piece of information is.
And so NOAADIC is kind of built around that thesis. We collect, vast amounts of human tumor data. We profile them at four levels of modality.
So pathology, which is kind of like the blue chip that, like, almost everyone has. Spatial proteomics is sixteen plex panels to identify cell types. Whole plex spatial trans transcriptome, so this is 19,000 genes over the entire surface of a tumor, then to identify, like, functional state of the tumor.
And then, exome sequencing to identify genetic alterations. So, like, is this KRAS positive? Is this STK knockout?
And so on. And then the the the ML angle is that you train yeah. Like, exactly you said, like, a self supervised mass model in the hopes that, like, one, you get a very good representation of any given tumor that walks into the door.
So you you now have, like, the ability to place, like, in the universe of all the cancers I've seen, where does this patient fall in that embedding space? And so that's what we are doing a lot of. We gather patient samples from people who have ran clinical trials.
They have patient samples. We profile them. We run that through the model, see if the response population falls in a different area than the nonresponse population.
And if it does, maybe we have access to a biomarker that no human on Earth understands, but we uniquely are able to. The more interesting thing you could do with us is use it use the generative capacity of the model and knock out specific transcripts or specific genes and see how that changes the expression of transcripts within the tumor microenvironment. You can imagine there's this, like, concept that's appearing in the cancer literature called nudge drugs, which are drugs that don't actually operate on the immune system or really the cancer side itself, but rather push it in a direction that makes it more sensitized to other drugs.
And so you can imagine, oh, I'll I'll I'll knock out this, this particular transcript, and then I will, like, hallucinate what would it be like if I added a KEYTRUDA, which is, like, an immune checkpoint blockade that operates on the PD one axis into the into the site of a tumor. And maybe now you predict, oh, the tumor's, like, highly inflamed. It's, like, hot.
There is there's a high chance that it'll just, like, melt away entirely. Yeah. Like, those those are the two big economic and ML strategies we're pursuing.
Yeah. That's really super exciting. When you talk about the first of all, identifying or or having access to, I think was your phrase, biomarkers that nobody else has access to because you can see a sort of divergence in where different patient populations fall in embedding space.
Do you have any means and, you know, if not, maybe I can introduce you to the good folks at Goodfire who just did a version of this with identifying biomarkers for Alzheimer's that had been previously not identified. But do you have any means right now of saying, okay. Because one thing to say, these things are falling you know, these patient populations are falling into different parts of embedding space.
It's another thing to then say, why? Like, what is it actually that is causing that divergence? How far are you guys along in terms of being able to make interpretable what it is that the models have learned from their unsupervised training?
Yeah. The, the the, the prima mente and Goodfire posts was very interesting to read. We do have, like, a Mekinterb research group internally, which, like, is exploring these ideas.
I have no doubt they'll find something interesting. But one argument against doing this at all is, like, why do we care about interpretability? Like, in a in a clinical setting, we might care about interpretability because, like, the FDA gets very upset with you if you try to do anything that's black boxed.
And maybe that was true a year ago. But circa, I think, September or August 2025, there was a pathology AI company called Arterra AI, which came up with, was basically a companion diagnostic as to, like like, would they intake in the pathology slide of your prostate tumor, and they will predict whether you respond you will respond to androgen deprivation therapy. It's they have no idea why this model works.
They've, like, retrospectively validated on thousands of patients from prior phase three trials, and the and the FDA was fine with that. So one argument against doing that can interpret all is that why why spend a ton of resources exploring something that the primary regulatory agency you care most about doesn't really mind whether or not it's white box or black box?
I guess the obvious answer would be because presumably that knowledge would be a great input to further experimental ideas or other knowledge, but maybe you think it it's just so hard to you know, I don't know. There's no verifiable ground truth or something that would prevent that from working?
the like, what was discovered in, like, the pyramidic at FIREPOSE? I forgot what exactly it was. There was something about, like like, fragment omics, like, something about how the, like, genes are fragmented, like, is a potential biomarker for Alzheimer's.
It's a very interesting piece of work. It sounds very expensive to validate it. And so I imagine, like, we would run into the exact same problem.
Like, maybe we get like, we have a very good hypothesis, like, what for what comes out of the system, but we we already have so many other hypotheses, potentially ones that are, like, even have, like, higher literature backing. I could imagine a world in which, like, MEKINTERP as a field gets so good that you can, like, triage that this thing's gonna be really easy to validate. This thing's gonna be really hard to validate.
But right now, the way that a meconterp usually works in biology is, like, you're staring at, like, semantic segmentation plots a lot and, like, trying to think, like, oh, is this real or is this fake? Is this the mod is this the model, like, identifying some very spurious correlation? And that time just simply feels better spent elsewhere.
Interesting. Okay. Here's another idea of a place that it might be well spent.
Continual learning, of course, a huge theme right now in AI in general. The first conversation we had today with, professor James Chao from Stanford included a little talk about their recent paper, learning to discover at test time, where they're using, you know, here, like, autoregressive large language models and giving them problems like make a faster CUDA kernel or, you know, find a a a better solution to this math problem, you know, with a a lower bound than than anybody has previously found or, you know, find a a better solution to this math problem, you know, with a lower bound than anybody has previously found. And they interestingly kind of flipped the usual model of, like, what we're trying to do when we create an ML model on its head and said, what if we just try to get this model to produce the single best answer that we can?
And we don't care if it generalizes. And in fact, we'll probably throw away this model after this, like, test time fine tuning. What we want is the answer.
And they they were able to find at like relatively reasonable cost, $500 compute cost, that they were able to actually get some new state of the art answers on some of these highly technical questions. If I'm a cancer patient and you've got a a general foundation model, a question that naturally occurs to me is like, can you fine tune this on my data? Can we do some like test time tuning?
Can we do sort of intensive masking on like just my samples and like really dial this thing in to understand my particular biology? And then would it be if we did that, would it be, like, more accurate for me?
and why or why not? So I I actually actually, looked looked at the paper, and they actually have a section for biology. They they do, like, single cell RNA denoising using this, like, test time test time training model, which I thought was really interesting.
I I guess my instinctive answer is it's an interesting idea. It very well might work, and it falls into, like, the bucket of ideas that we would simply have to try it until I'd like to make sure that it does or does not work. The results for, single cell RNA de noirsine that's in the James Chao paper are, like, certainly good.
They're, like, they're better than the state of the art, but they which I thought, they they like, for each one of these cases, they they attach, like, a note by, like, an actual domain expert saying, like, how useful is this in practice? And the domain expert in question for the single cell RNA section did say, like, this is very cool, but, like, at the end of the day, we don't really care about the results of single cell RNA denoising. We care about some biological utility that is underlying that.
And so maybe you get better at solving this, like, verified task problem, but that doesn't translate to anything actually useful. Maybe it would be, like, per the response non nonresponse prediction case, but it kinda just sounds easier to fine tune the model, like, using normal supervised supervised learning. Like, why go through the RL process if the end result is, like, binary?
You know? Like, think they even called out in the paper, like, this setup isn't really meant for binary or sparse learning tasks. It it's meant for, like, fuzzier things.
Yeah. That they're working on that, but it it's not done yet. I guess maybe zoom out and, like, you you kind of alluded earlier already to this idea that a lot of people think we just need better ideas for drug candidates.
And your consistent position is like, that's probably not really the bottleneck. You made a really interesting point around how the ability to evaluate more accurate ability to evaluate those candidates drives a lot more value than just throwing a lot of a lot more candidates through a pipeline. The quality of the pipeline matters most, more than it's, like, scalability.
So, again, I think you you kind of have suggested where you think this can come from with just, like, large scale foundation model style training. But give us the the next level of depth on that. Like, why why are all these other ideas not so exciting?
And is this basically just a bitter lesson sort of idea where all your cleverness is gonna be washed away by scale and so keep your eyes on the prize? You got a data max and compute max until you solve it all. Is it is it kind of that?
I I guess I kind of view things in, like, three ideological camps, the first of which like like like, the first one is, like, maybe us. Like, we're we're we're indexed very heavily on, like, human data. It's the only thing that matters.
You can start from, like, in vitro settings and bootstrap your way up to something more complicated. You need to start with the more most complicated thing to begin with. The second camp is very interested in modeling, like, single biomolecules and, like, their interactions in in the hopes that, like, maybe you can't bootstrap your way upwards, but you raise the absolute, like, success rate from maybe five percent to to twenty percent, and maybe that's all you need.
And I think, like, that like, the second can, like, defines the vast majority of ML biopoplinings that exist today. I think, like, some of them have clinical candidates, that are ongoing right now. We'll see what the results are.
I think, generally, it doesn't seem like there has been, like, a massive step change in their ability to design drugs. This is not, like, like, knocking them. Drug discovery is hard.
It's everything's in bed at the end of the day. The third camp is it's like a maybe it doesn't it's not really a for profit thing, but you can just, like, improve the clinical trial process to begin with. And this is arguably the path that like like, China like, this is, like, China's main advantage.
So they're able to run clinical trials far more cheaply than anyone else, partially because of, like, lower cost of human labor, but also because they've just, like, set up the system pretty nicely such that it's not such a a huge regulatory and financial headache to get things going. This has, like, some downsides. Drugs are treated innocent before proven guilty.
The FDA is the other way around. But the obvious benefit of doing that is, like, you're betting neither on the AI in human data getting better or the AI in vitro settings data getting better. You're trusting that the typical drug design process, if just made slightly more financially efficient, will, like, improve things on it.
I think all three of these are important. And I think it would be, probably, like, grandiose of me to, like, assign an unequal weighting to any one of them. Each one feels important to push on.
Been some Oh, There interesting
sorry. Me sneak I'm in one from going to take a little bit of a segue to something you said earlier, which is that a lot of the new INDs are coming in from China. What has happened in the last couple of years?
Is it an AI thing? The CEO of Ginkgo Bioworks was on TVPN yesterday. He said they just have more hands.
And some people believe it's a regulation thing. Some people believe it's a clinical trial registration. They can just register more people.
Some people believe it's a US cost thing. What is driving this transfer of basic R and D to China at this point?
I I I think that, like, this particular subject is very deep. It's not something I have expertise in. I like, my my instinctual thought is that there are many different answers to this, and the one that I'm most, like I I think is most interesting is the idea that, like, China was all like, was always, like like, a very good generics manufacturer.
And that's, like, that's where they started. Slowly, extended their way into, like, WuXi having a very good CRO ecosystem. And and at some point, enough talent began to be incubated in China where they began to realize, we have all this infrastructure here.
Why not just develop our own drugs? And there is something very important about having this, like, very close interplay between both the person who is designing designing the drug and the person who is actively doing wet lab assays on the drug. Whereas in America, you have, like, a super long feedback loop of, like, I need to get my SAD together.
I need to go reach out to VCs. I need to go buy a lab. Whereas in China, that ecosystem is a little bit set up already.
Actually, maybe the only missing part is that the VCs are still, like, not super they're more, like like, risk averse than perhaps VCs in America. But the colocation of, like, the grunt work and the intellectual work is actually surprisingly important. A few months ago actually, last year, I interviewed one of the very few people doing novel biotech research in India, a guy named Soham who runs a company called PopFacts.
He said, like, this is the primary reason why he expects not only China to start producing really interesting drugs, but also potentially India, potentially Egypt, places where there is intellectual capital. There's a lot of hands and it's just like those combination lead to really good compounding results.
Indeed. Does that accelerate with the AI models, this kind of AI co scientists? Does that mean that even if they don't have that much intellectual capacity yet, they have the hands to carry it out?
I guess, like, this is this is something that's like it's a little bit opaque to almost everyone as to, like, what exactly is the level of, how impressive are the bio AI models coming out of China? I think there's certainly some interesting work that has been done. It's not clear to me that there's anything, like, radically new there that won't be found anywhere else.
A fair amount of it is, like like, like I don't wanna say this in a disparaging way, but it is, like, like scaling up stuff that was originally developed in either The UK, London, or America. And there's, like, nothing like, there's it hasn't been really, a deep sea thing where there's something, like, radically crazy that comes out of any of the Chinese labs. I obviously could be wrong on this, though.
There's I think the whatever the bio a AI labs are doing in China, there's, like, much less American visible visibility around it. Mhmm. Hey.
We'll continue our interview in a moment after a word from our sponsors.
Everyone listening to this show knows that AI can answer questions, but there's a massive gap between here's how you could do it and here, I did it. Tasklet closes that gap. Tasklet is a general purpose AI agent that connects to your tools and actually does the work.
Describe what you want in plain English, Triage support emails and file tickets in linear. Research 50 companies and draft personalized outreach. Build a live interactive dashboard pulling from Salesforce and Stripe on the fly.
Whatever it is, Tasklet does it. It connects to over 3,000 apps, any API or MCP server, and can even spin up its own computer in the cloud for anything that doesn't have an API. Set up triggers and it runs autonomously, watching your inbox, monitoring feeds, firing on a schedule, all twenty four seven even while you sleep.
Wanna see it in action? We set something up just for Cognitive Revolution listeners. Click the link in the show notes, and Tasklet will build you a personalized RSS monitor for this show.
It will first ask about your interests and then notify you when relevant episodes drop. However you prefer. Email, text, you choose.
It takes just two minutes, and then it runs in the background. Of course, that's just a small taste of what an always on AI agent can do. But I think that once you try it, you'll start imagining a lot more.
Listen to my full interview with Tasklet founder and CEO Andrew Lee. Try Tasklet for free at tasklet.ai, and use code cog rev for 50 off your first month.
The activation link is in the show notes, so give it a try at tasklet.ai. Your IT team wastes half their day on repetitive tickets.
Password resets, access requests, onboarding, all pulling them away from meaningful work. With Servl, you can cut help desk tickets by more than 50%. While legacy players are bolting AI onto decades old systems, Servl allows your IT team to describe what they need in plain English and then writes automations in seconds.
As someone who does AI consulting for a number of different companies, I've seen firsthand how painful and costly manual provisioning can be. It often takes a week or more before I can start actual work. If only the companies I work with were using Serval, I'd be productive from day one.
Serval powers the fastest growing companies in the world like Perplexity, Verkada, Merkor, and Clay. And Serval guarantees 50% help desk automation by week four of your free pilot. So get your team out of the help desk and back to the work they enjoy.
Book your free pilot at serval.com/cognitive. That's serval.
com/cognitive.
One okay. Go ahead. One big question I have is kind of I find it very hard to calibrate myself on how excited I should be about all these AI for biology and AI for medicine developments.
I know that, you know, there's always these kind of headlines. AI does this. AI discovers this drug.
You know, I've done episodes of the of the cognitive revolution on it. One with Jim Collins. You know, he has created a a bunch of antibiotic candidates.
You know, there's a long list. Right? Yeah.
Did professor Zhao did the nanobodies thing that came out of the virtual lab. To hear him talk about it earlier today, it sounds like those were, like, reasonably well validated. But then you always get this other side too that's like, well, not so fast.
You know? It's all very messy. We got a long way to go.
You know, most of these things don't pan out. And I feel like that sort of parallels the debate that we hear in a lot of different domains where, you know, even in, like, programming, which is one of the more, let's say, legible domains, we've got something like a meter study that showed slowdown of developers, and that was very confusing. I'm, like, still quite confident that it's making me faster, and I kind of wanna throw that away or, you know, there's, of course, just a lot of denial and cope out there and all sorts of motivated reasoning.
How should one try to ground their worldview? Obviously, you know, subscribing to outposting, is something everyone should do. But what else would you advise me?
Like, how how can I patch these blind spots in my world view or or get to a better position from which to have my own sense of what really counts, you know, what really matters and what doesn't? Because, you know, again, this happens all over the place where there's this disagreement even among, like, some of the most informed people about just how big can AIs reason or, like, how big of a deal is it gonna be? But in biology, it's particularly hard for me to make sense of.
get some tips for how to climb the learning curve faster. I've actually, written about this in the past a very long time ago. The title of the article is five things to keep in mind when reading biology ML papers.
The the long and short of it is the evaluations in biology, I think, are very difficult. And so the vast like, you see this thing about, like, in, like, more typical wet lab biology of, like, oh, we cured cancer, but it's in a mouse. Who knows when it'll actually translate to humans?
There's a very similar phenomenon in a lot of biology ML papers where they're doing something that feels like it should be useful, but there's, like, a lot of things that they're probably hiding from you when explaining the results that would only be obvious to a domain expert. One one, I think, really funny example of this is, like, like, small molecule binding affinity papers. I I've I've written about I've, like, one company's work in this, but they found that these, the like, let's say you're able to predict, these set of molecules bind to this target, these set of molecules do not bind to the target, and you'd be very happy with yourself if you publish a Nature article about it.
What they for these, folks, at a company called Leash Bio found is that this can often be confounded by which chemist actually produced the molecule in the first place. Because some chemists are very attached to specific targets. They're very good chemists, so they often produce things that, like, bind to that specific target.
And these chemicals, like, very importantly, all look very similar to each other. And it is this type of similarity that is like a very human vibes base and it's hard to pin down to a singular metric. And so they found that these models are often confounded by offer overlap.
Think this and then these problems just appear over and over again across the in vitro biology, biomolecule generation where you can be confounded by variables that you did not know even existed in the dataset. And so I would probably name that as the thing to be most aware of when reading these papers. I I like, there there are, like, a few people I, like, trust on Twitter and, like, people in real life who can give a pretty good overview of any arbitrary paper.
But, like, I think, like, with, like, LMs, like, popular science people often, like, retweet them and say, like, this is transformative. And, like, more often than not, they're probably correct. Opus 4.
6 is, like, genuinely crazy. I think when people do that for Bio ML papers, there's, a fifty fifty chance that they're completely missing the porn because they don't they're not in that field, and they don't understand how the failure modes emerge in these models. Yeah.
Do you think that an opus can, like, help me identify those blind spots? Is it Yeah. Yeah.
The sorry. Go ahead. Go ahead.
Yeah. Is it good enough to do that?
I I actually, I've written an article about this also. It's titled, can can o one preview find mistakes amongst 56 MLSB papers? MLSB is a structural biology workshop at NeurIPS, and it's not very good at it.
Like, this was, like, obviously, last generation of models. Maybe it would be a lot better. But there's a there's, there's there's some problems that are gonna reoccur in almost every biology ML paper of, like, oh, your train sizes aren't large enough.
Your test sizes are, like, not stratified correctly. But, like, you kinda just, like, learn to pick your battles in this field and you just move on. There's, like, a lot of more fundamental problems with article with these papers that LLMs, in my experience, just, like, often mess entirely.
I think the funnest, like, I I in almost every article I've written, I have found that l like, LLMs tell me something about this particular subfield that the domain experts completely disagree with. And they say, like, like, that is not how you should think about this domain. It's, like, that's not the real problem we should we're actually worried about.
I don't know why this is the case. It's it's kind of fun. It's like like a domain of science that LM still haven't quite captured human taste.
Yeah. Fascinating. Okay.
That leaves, a lot of work in front of us. Do you wanna go back briefly before we break to Noetic again? I mean, I've, fortunately, my son is doing well.
He recently got cancer three months ago. I've, like, had a, you know, intensive crash course in cancer, and I, hope to be able to close the book on it and, you know, return it to a more intellectual and less personal, interest going forward. And I think we're on good solid track to do that.
But how I think you've demonstrated in this conversation that you're, like, not getting too carried away with the promise of what AI systems can do. We've got the data center of geniuses. We've got the century of progress compressed into five years kind of visions.
How much would you shave off of those notions just to describe your own expectations of what Noetic can do specifically and and maybe what the field more broadly is gonna be able to accomplish?
I'm very optimistic that, like, human simulation companies, like, akin to Neutic, but I think there's other players out there as well, will be able to vastly help with the results of a few at least a few clinical trials within the next few years. That feels like almost like like, you're not even paying attention too much of the trend lines. I'm almost, like, indexing on, like, what we're capable of today.
I think it's, like, pretty obvious. Like, there there like, there's papers going back, like, year like, years that are able, like, to show that, oh, we've developed an MLML that is able to better able to stratify patients. The problem has always been, like, an economical one.
And, like, how do you actually deploy this in a real setting? I think we'll we'll be able to do that just fine. I think phase one phase one drug The failure of phase one drugs will go down.
And I think this has already been slightly proven out in, like, I think a McKinsey study that was done five year a few years ago that showed that, like, AI designed drugs have, like, a five to ten percent lower failure rate, which may be noise, may be real. I do kind of expect those trend lines to continue a little bit. Where I'm, like, most unsure of is whether these models will able to discover brand new targets entirely, which is ultimately what, like, people care about.
I think there's, a I think, like, believing that these models will be able to find, like, new targets far faster than humans would really requires you to index heavily on the trend lines. And I want these index on the trend lines. So I believe that these models will be able to deliver very good target finding.
But I'm also very sympathetic to that mindset of finding targets. It's just such an unbelievably hard problem that Mhmm. The model, like, will not make a tentative because you need this, like, human iteration feedback loop.
And unless you build a really good human human simulator, which is our bet, you're not going to get close to solving that problem.
The way I put it is usually you can see one order of magnitude ahead, maybe two. No one can see three orders of magnitude ahead. It's just not possible.
You have no idea what's going happen. Abhi, thank you so much. I learned a lot from this and hope to see you online.
Read Hope more of your blog. Absolutely.
Thanks for having me on.
Thanks for being here. We'll be working our way through outposting archives, for some time to come. Wonderful.
Bye bye. Our next guest is Helen Toner, who runs CSET at Georgetown. She's a former OpenAI board member, And there's two competing views here.
She has, on the one hand, the intelligence explosion is coming. On the other hand, AI capabilities may be permanently jagged. So let's add her to the stage.
nice to have you. Hey. Thanks for bringing me in at the end of your marathon.
Impressed you guys still going strong.
There's so much to cover, you know, and, we've all gotta accelerate our our personal productivity timelines and and try to pack more information into, you know, the same amount of time. So experimenting with with ways to do that. So you said we should talk super fast.
Yes, please. That's honestly one of my reservations about live content is I listen to everything at two x speed, and I'm like, I can't listen to it two x speed if it's live. So I mean, I my my constant struggle is to talk slower than I naturally want to.
So if you want me to talk double speed, I'm here for Please go, fast as you want. Go for it. Go for Okay.
So you guys just put out this report. I think this is obviously, a great candidate, if not a shoo in for the most important question of our moment. What is going on with the possibility of automated AI r and d?
Do we have this tipping point where we're starting to hit recursive self improvement? And if so, like, how big of a deal is that gonna be? You guys brought together a bunch of people that authored this report and some others as well that aren't necessarily authors but contributed to conversations.
I understand, like, quite a few people from frontier model developers. And it strikes me that, like, this debate goes back basically to the beginning of AI. Right?
There was the idea very early on that we could have an intelligence explosion. When I started reading Eliezer in 2007, you know, he was very worried about this. And yet, you know, you've written I think you put your finger on something a lot of people were feeling last year when you said even though what passes now for long timelines is pretty short, and yet the disagreement on this topic seems to be as fundamental and seems to be kind of as impervious to new evidence as it has ever been.
So maybe just for starters, like, take us inside the workshop, give us kind of the the lay of the land in terms of what are the world models that people have, and why are we still just working from so much intuition despite the fact that we now have, like, you know, what in some, you know, circles would even be called AGI out there, you know, as products for us to use today?
Yeah. I so this was this workshop was held in in July, last year and was maybe one of my work highlights of the year. It got started so it was a day and a half.
We brought people in. We had, yeah, people coming from some of the frontier companies, policy, you know, bunch of great people. To get a sense of kind of what the vibe was like, we started out the first session was about kind of how is AI being used to automate AI r and d right now.
We had presentations from people who are doing that. And before the first break, we had Ryan Greiblatt from Redwood Research, Nicholas Carlini from Anthropic, Dias Kapoor from Princeton of AI as Normal Technology, and Thomas Larson, who's one of the AI twenty twenty seven authors. They were all, like, arguing so fiercely in a friendly and productive way, like, before the first break that they just, like else stood up to go and get, you know, coffee and drinks and snacks, they just kept on arguing, right through the break, which is great.
It was exactly what we were looking for. But I think did sort of preface something that we knew going in, which was there are really different perspectives here. And that you know the workshop was Chatham House.
I feel okay giving that anecdote because they ended up writing. One thing that came out of that was Nicholas was pushing the others constantly for like, okay you have such different views about where things are going. Where's the first place that you actually disagree about what we'll see?
And they found that as they're looking out for like what we're gonna see in 2026, 2027, they actually agree a lot about kind of what we're gonna see before we get to that recursive point which is kind of a bummer. Like, it's, you know, it's nice that they agree and can they they were to post about that actually, which is the reason I feel fine, you know, sharing that that anecdote from an otherwise chat in house workshop. They were to post about the stuff they agree on.
But it sucks that, like, that means that it's actually gonna be hard to identify kind of in advance whether we are heading into a recursive loop or whether we're not. Two big things that I think so what we're trying to do with the workshop, one was like get this idea of recursive self improvement out of sort of purely like Silicon Valley, San Francisco, like really AI pilled spaces and make it, like explain it, present it to a wider audience, let people engage with it. But then also actually try and make some progress.
I'm like, why do people disagree about this? What is happening? What might happen in the future?
You know, what indicators could we gather? Stuff like that. And I came out of it thinking that maybe two of the core disagreements here, one is, does AI truly replace all of what humans can do?
So does it does you get to that, like, fully automated? Because if you're gonna have the big the really, like, scary recursion, that's probably what you need. It can't be that, you know, every you have a we could have much more productive human researchers.
You could have, you know, the the Alec Radfords and the, you know, Elias Setskever is managing fleets of AI researchers. But if it all has to come back to them and they have to process and digest and, like, think through the research, you're not gonna get that, like, massive recursive loop. So that's one piece is, like, do you truly get humans being fully replaced?
Because if not, then maybe you have some parts of the workflow being, you know, really accelerated. We have this diagram in there of like sort of an Amdahl's law kind of thing where Amdahl's law is basically if you have a process that depends on different inputs and there are different potential bottlenecks, then if you speed up one part of the process, the bottlenecks will just bite somewhere else. And so it may be that, you know, you speed up the coding part of AI research, but if you don't speed up other parts, then you don't end up speeding up the whole thing very much.
Or, you know, do you have I think another mental model that people who are skeptical that this is gonna really go crazy, a mental model they bring is like, okay. We have a long history of computers doing more and more of the lower level work. So we don't have to do punch cards anymore.
We don't have to write assembly code. We have these higher level languages. And so, for example, AI doing more of the coding is just another natural step in that process.
And humans, this is kind of like expanding PIE model. The amount of like tasks that we realized can be involved in AI r and d expands. And there's always that outer band that the humans can do while they're automating the inner bands.
And I think that is very different from the view that, you know, other people, sort of more that AI twenty twenty seven authors would have or, you know, lots of other people in this space, which is no. First, automate some of what the humans can do, and then you automate all of what the humans can do. And then, you know, you kind of you go until some other bottleneck hits.
So then, you know, the the other question is okay, what are those bottlenecks? We talk about that as well. But I think those are two of the biggest questions that came out for me.
One was, are you truly gonna automate everything, including what all the humans can do? And the other is, if you do, how soon do the bottlenecks bite?
So Sholto Douglas, is now at Anthropic, had this idea of a software only singularity, he calls it, where we get very good at coding and like all of the digital stuff, including AI research, I presume, but not at producing power or copper or all of the physical substrates which are going to be required in order to support this expansion. How would you feel How do you think that fits in? Like the fact that maybe the digital stuff happens, but the physical stuff just doesn't?
Yeah, I think there's two versions of this. I think some people, when they talk about a software only singularity, they basically mean like it turns out that software is enough to get like absolutely crazy recursive loops. So I think there's like Tom Davidson at Forethought Institute has written about this for example.
And, you know, maybe you can get just like massively more intelligent systems having massive impacts on the world primarily through sort of software improvements. There's a different thing which what you're describing of the the version from Sholto is a different thing which I would think of as more like sort of a jagged software only intelligence explosion. Meaning, the AI is getting much more capable in certain ways and like but its effects on the world are very limited because it is software only.
And this kind of gets another another thing that I found really, really helpful from the workshop and really interesting, which is people have very different intuitions about, okay, assume that you have an AI that is very, very good at AI r and d. What does that mean for what the AI can do elsewhere? And I think some people are like, okay, well, if it's very good at AI r and d, then it can like train AI models to do whatever so it can do whatever.
And, you know, maybe there's like maybe you have to spend like a week gathering data or something. But then if you wanna do some arbitrary task, you could do it. Whereas I think other people have an intuition of like, okay, well, even if it gets very, good at automating AI r and d, this sort of most, you know, software based task, it's still gonna really struggle to, for example, design new biological molecules, or it's gonna struggle to, like, think about geopolitical strategy questions because you have to, like, actually go out and, like, see how different countries and decision makers will react and things like that.
And that is a piece that I feel like goes under in a lot of these conversations is, like, what is that connection between AI that can do incredibly good AI r and d and AI that can like affect the world in non AI r and d specific ways that we also tried to kinda like tease apart a little bit in there.
Would you think that's the connection between, okay, now you have AI doing AI research, that's affecting the economy, it's also affecting the political economy, and then you have to have mitigations to the political economy for this to work out. Does that mean you might need the AI research to go into how to fix the political economy, which is gonna be a little bit scary? Yes.
Say more about what you mean by affecting the political economy. In the sense that, for example, right now you have Bernie Sanders saying that we should have a moratorium because he's scared about jobs, right? He's very scared about jobs.
He wants a moratorium on data centers. I think there's like six states with a moratorium now, including New York State. And so AI research leading into, hey, how does AI fix the political economy?
How do we deal with the humans? How do we mitigate the impact that we have on the humans? Like, is that something that you think would happen with the first the first configuration of the software only singularity in the sense that it's not jagged and also affects the political economy?
Yeah. Yeah. That's that's the kind of thing that's right.
If you're positing that you can just have a sort of software only singularity that is going to radically transform the world, then it's gonna have to be able to do things like, okay. And then the company, like, deploys, you know, chatbots that talk to enough people that convince them that data centers are great. The data centers all get built and the moratoriums get rolled back.
Yeah. That that kind of has to be built in. That's right.
Which to me intuitively feels like it's sort of a different skill set and also is more dependent on, like, deployment and rollout and adoption. Yeah. So I I tend to be a little a little more skeptical there, but, yeah, I think that's a that's an example for sure.
I see.
One kind of odd, seemingly to me, odd pairing of beliefs that I observe and I sort of detect in the report is the idea that among the sort of more skeptical folks that there's gonna be a plateau and also that plateau is gonna be subhuman. And then on the other hand, it's like, it's gonna not plateau. It's just gonna run away, you know, have some sort of singularity.
I a position that I feel pretty intuitive ly attracted to that I don't hear too often is the idea that maybe there will be a plateau, but it could be very easily a superhuman plateau. If I try to zoom out, you know, as far as I possibly can and look at, like, life on Earth, I would say it seems like humans are sort of part of maybe an entry into a steep part of an intelligence explosion or, you know, an s curve of of, of capability. And I don't think we're the end of history, but we were clearly better than what came before, and that was enough to take over the world.
And so I think I just don't hear too many people say, yeah. It's not necessarily gonna be a singularity. It's not necessarily gonna, like, you know, go totally beyond comprehension.
But in the same way that we were just that much better than Neanderthals, and it might not have been that much, but it was enough to to change everything. I kinda feel like there's not too much more room between where the AIs are now and where they will soon presumably be. And even if that doesn't, like, you know, go sort of critical from there, it feels like we're like, it's very hard for me to imagine that it's not enough to be transformative.
So is that a position that was, like, represented in the workshop? And how do you personally react to it?
Yeah. I I mean, I I think that sounds pretty close to my default expectation maybe, which and if so, then then it was represented there because I was there. Maybe to, like, rip on it a little bit, something we didn't put in the report, but that I've definitely found helpful helpful from my own thinking is, like, thinking about, okay, you have an s curve.
We're clearly in some kind of s curve. I agree. And you have, like, three maybe less, like, situating us as humans in the middle of an ongoing s curve.
But we have some kind of s curve of, like, AI capabilities. And there's three segments that are of interest. One is, like, how long is sort of the the lead up period, the first part of the s curve.
One is how steep is that, you know, middle of the s, and one is how high is the ceiling. And I think a lot of mostly, when you're hearing people talk about automated a r and d, either they're sort of they're one of two camps on all three of those questions. So either they think the lead up is short and the curve is steep and the ceiling is high, or they think the lead up is long and the curve is gradual and the ceiling is low.
So I also think it find it really interesting to think about, okay, what are different, like, combinations of those sort of parameters? Where to me, feels like really looking like the lead up is pretty short these days. Like, we're not too far from that sort of takeoff period.
But, like, for example, what if the curve is steep, but the ceiling is low, or the curve is gradual, but the ceiling is high? I feel like we don't talk that much about either of those. And maybe also to your point, Nathan, about the, like, superhuman but not like all powerful god, like, singularity, there's no, you know, no point of return.
Yeah. I I also I really think there's room for more thinking about what does it mean to be superhuman and what are the domains where there's like tons of headroom above humans and you can easily identify like, no. Yes.
It would look this is what it would look like to be superhuman at, you know, like optimizing a kernel or like, you know, selling things for example. You're spreading legalities.
Yeah. Yeah. In in previous, you know, parts of this marathon conversation series of conversations, we've just seen how the ability to interpret the signals that people throwing are throwing off in sleep to predict disease.
You know, it's just a a really random, but, yeah, I think, instructive example of how there's an obvious, lot of room to be superhuman at some of these tasks. And Yep. There's potentially a lot of power to Mhmm.
Unlock, especially if you can integrate that kind of, you know, infinite modality grokking with, like, a kind of basic reasoner.
Yep. And I really don't see any reason that we're not going to be able to achieve that. Yeah.
So the often those often those things, though, will involve I think another piece that's, like, underexploit here is, people will intend to either be in the camp, like, the ceiling is high and you're not gonna need all that. It's not gonna be delayed by real world adoption, or the ceiling is low and it's gonna be delayed by real world adoption. And to me, I'm sort of like, isn't the obvious, like, combination of these?
Like, once you get the real world integration, you know, for example, you're you have to collect all that sleep data or, you know, humans are really bad at interpreting scent data, but, like, dogs, like, smell things we can't, like but you have to do a bunch of sensors and, all that. So I also feel like there's sort of unexplored questions around how high is that ceiling as you have kind of really increasingly integrated AI to more and more aspects of life and economy.
because in in my the way I might view things as happening is software and mathematics first. And the question for me is that if you get software and mathematics first, you may get things like, I don't need a LiDAR for my self driving car anymore. I can use cameras.
And the cameras can be really bad cameras now because the math does all the work and I don't need all this sophisticated technology. It could be that your phone could do what those sleep detection machines do with the right software package. Your phone has a lot of sensors, there's an enormous amount of technology within the phone.
You do wonder whether it would really be application of algorithms to existing frameworks, existing infrastructure, increasing the bandwidth of your communications tech with new encryption and new cryptography, which is how DSL was invented, for example. DSL was really using the existing copper pipes with new algos. So I wondered to what extent the you don't get a slowdown just because of your physical infrastructure because you kind of innovate around or with your physical infrastructure.
Yeah. I'm sure that I'm sure that will work in some places. I think that yeah.
I think it'll work in some places that won't work in others. So I think if we're talking, like, you know, there's a huge ongoing struggle with where my mind goes is, like, cybersecurity for critical infrastructure where, like, the systems are the physical systems are old. They're hooked up, the operational technology.
They're hooked up to old information technology because they have to be like, there's gonna be a limited amount that you can optimize using smart new algorithms there because it's just Yeah. The stuff is old. Likewise for, you know, my center does a lot of work with kind of military technology.
Same thing there. Like, if you have a ship that was built in the nineteen sixties, it's a ship that was built in the nineteen sixties Yeah. Or other pieces of equipment.
So, yes. I think in some places, yes, in some places, no. To me, that's another I think Prakashi mentioned it as I as I came on that it's like talk I gave on jaggedness.
To me, it's like another place where the jaggedness bites. And I think as well, like, my default expectation in AI r and d is that we'll see jaggedness like, the jaggedness is fractal. Right?
Like, you zoom into the sort of quote unquote task of AI r and d or the skill of AI r and d. Actually, it's many, many different things. And we'll so we'll see the AI r and d accelerating in areas that are especially amenable to using AI and lagging more in other areas.
Not to say they can't ultimately be automated, but it will, like, take longer.
far do you think the the product which is on the market right now behind what people are using inside the labs? Like I don't know.
Okay. I honestly don't know. That was one of the, like, sort of most, like, trying to be productive section of their well, it's not true.
But at most, like, maybe actionable section of the report is, a set of indicators. We have, like, a table summarizing the three categories of indicators that we have. And that is the biggest category is kind of indicators from inside companies, and one of them is that public private gap.
My sense is that it's not huge right now, but I would I don't have any inside information that you guys talk to company employees well. If you believe Arun, he says it's we have no idea how good we have it and the gap is very small. Exactly.
I'm thinking of things like that exact sweep.
So what I learned in the last few days is that the real gap is that they're using models which are three times faster.
and that's what they're using in turn. That's it's the same tokens, it's just a lot And there's surely also, like, tooling stuff, right, as well. Like, you know, something we put in the report, a couple of our reviewers who were looking at this, who were less familiar with the idea of automating AR and D.
Some of them are like, oh, haven't you seen that study of, like, ninety five percent of AI pilots fail? And there's the, you know, that meter study of, you know, that AI slows people down. And so we included in the report, like, explicitly noting, you know, yes, productivity boost from AI are mixed.
But these AI researchers are, like, in the very, very best position to benefit from their technology. Like, they are the best up to speed on what it can do and what it can't do. They're shaping how it's developed and like what directions it's pushed in.
They're in the perfect like setting to be building tooling to squeeze the most juice out of these models. So I'm sure that that is also a piece of it as well.
Of the things you've mentioned early on just minutes ago is the idea that you wanted to kind of bring awareness of these possibilities, you know, outside of the places where they are most often discussed. One of the thing I would love to hear your perspective on is how ideological do you think companies are about this? I mean, this is one of the things that, like, confuses me in the sense that every frontier lab leader has read their Eliezer, you know, catechism.
They you know, I think they've they've previously many of them in the past had said, like, things about how we should be extremely careful about this sort of thing, and we should not engage in an arms race dynamic. You know, it's obviously part of the OpenAI charter. Dario has said things like this.
And now we're in a world where there is an a publicly stated timeline by OpenAI to the AI r and d intern and then another, you know, not too much longer out, 2028, the full AI r and d researcher. And and, I mean, scenario Anthropic as well as Dara's Yeah. I would say Anthropic, if we'll even see more committed to it or more, you know, were resigned to it maybe, but they believe it.
Jack Clark, June, summer this year. Jimmy Bah, who just left x AI, was a cofounder twelve months. And OpenAI this year research intern and then full researcher of a year later.
Yeah. I think it's this year. So that's my guess.
This year for what specifically? Like, the start of recursive self improvement. We're we're gonna talk about But aren't we there already?
Wasn't it last year?
I mean, you had, like, you know, you had, like, Gemini doing the what was the evolutionary algorithm stuff or designed, like, co design an algorithm that, like, sped up its own training by 1%. Like, come on. That's recursive.
Truly thing. Does it this is what I'm talking about, like, the the lead up to the The loop. Yeah.
So you but I don't know. I you you think we might be this year where there's no human needed whatsoever? I think it's a high bar.
I think we might be I think I I updated on Motebook. The Motebook thing took me by surprise, like one and a half million agents all of a sudden on the on the web. Yeah.
It's all nonsense for sure, but things start off as nonsense. I think I think what might happen is you get a single model update which kind of fixes a little bit of the hallucination, a little bit of security issues around, like, leaking leaking secrets, and I think that might be enough. Sounds hard.
Sounds real hard. Fixing the security stuff. Yeah.
We'll see. Yeah. Maybe.
Maybe. So go I wanna I do wanna give you the chance to talk about the dynamics of this. The there's, you know, different reads that we might put on to people.
They're ideological about it. Elon Musk has said things, like, pretty much like, I don't know if this is gonna be good or bad, but I wanna be around to see it. And, also, like, you know, I'd rather be part of it than a spectator.
So that sounds like somebody who is inclined to gamble with humanity at you know, in a pretty self aware way. Others may feel like they're trapped into these dynamics, and they at least will do it, you know, as safely as possible. How would you describe that milieu right now?
I think it's dramatically underappreciated by people outside the the AI bubble that where we spend all our time.
Yeah. I mean, I think my impression of it, my sense of it from the people I talked to is there is just a sense of inevitability about AI advancing and a desire to be a part of the future being created because they see this as a future that's being created. And I think there's also, you know, the people who you mentioned how this has been kind of part of the AI conversation since the very beginning is this I j Good was in '19 early nineteen sixties talking about when you create the first ultra intelligent machine, which I feel like is we always need more terminology in AI.
Feel like we should get ultra intelligent to make a comeback. That there's this sort of, like, very natural logic. If you have a computer science kind of brain, it's just a very natural logic to say, okay.
We're we have some level of skill at building computers. When the computers have more skill than we do, then they'll build ones that have more skill than that. And then, you know, you get a loop.
And so I think that that logic is just very appealing and and seems very natural. And so people think of it as something that's gonna happen anyway, and then they may as well be may as well be involved. That's not everyone, but I I do get the sense that that's sort of the, like, the water that most folks are swimming in.
And then if you have a different view, then it's, you know, in contrast to that. I don't know. Do you is that your sense as well?
Yeah. I think so. And I think the inevitability is a pretty compelling argument.
I mean, I I resist it because and I at least want to make the point that, like, even if some form of this is inevitable, there is still probably important discretion that we can exercise in terms of exactly what flavor. And there's questions like, should we keep chain of thought interpretable, or should we, you know, embrace thinking in latent space? And, you know, I do think it's important to kind of keep in mind that it's not probably all one or all the other.
You know, AI defies all binaries. There's gonna be these gradations and these kind of Mhmm. More local decision points.
But yeah. I mean, in in 2022, I was just trying to make AI work for practical tasks. And I, with no background in AI research, basically ended up independently inventing a number of the, you know, techniques that have gone on to produce great things.
I didn't, you know, take them, past any local plateaus, But, you know, just having AIs improve their own outputs, you know, kind of proto proto constitutional AI type stuff, I do think the attractor is you know, the the sort of gravity well is, like, pretty strong, and and it just it it's hard to avoid some version of these techniques because if even a, you know, a bozo like me lands on them, I don't know how they're not gonna happen, you know, in the big broad world, especially as we start to also get dramatic democratization of training techniques. Right? I mean, prime intellect just put something out that kind of allows anybody to spin up their own, you know, RL environment on a distributed basis, on a community basis.
So, like, everything's gonna get tried, and that I think that's pretty hard to argue against. But, again, also, do want people to to still own what exactly it is that they are doing along the way. Yeah.
which takes me back to conversation long running conversations about autonomous weapons. There's something in here about the, like, the level of human oversight that you can have, where I do think that, like, using that tool like, using AI to to accelerate research is I I totally agree an attractor. But I do there's an you know, you'd really hope that there's a meaningful difference between I have a fleet of 10,000,000 AI agents and they're running experiments for me and I'm I am leading them and I am, you know, guiding them versus I have set something into motion.
I have no fucking clue what's going on. And I think there's there's a boundary somewhere. Is it a boundary that we're able to stay on one side of?
I'm I'm not sure, but I hope it might be. That to me feels like the point to try to intervene, not, you know, we shouldn't use AI for research. That's obviously not gonna work.
To what extent do you think policymakers are naive? Because in like, earlier on, we spoke to Sam Hammond and he's he he advises some policymakers on AI. And he was talking about privacy and the restrictions that the constraints that we could put on or the regulations.
And to me, I think one thing that struck me was that I think a lot of policymakers are not aware perhaps that these, like AI with access to existing technology, with access to persistent search, persistent memory, it would basically do a Google stalking of you before it even met you. It would know all of those things in the public domain. The amount of access to information that it could have, the persistence of information, listening in on conversations, these things are going to be very powerful in that sense.
I think you can ban the AI from using facial recognition, fine. But then you have network analysis, etcetera, etcetera. You can do metadata analysis on WhatsApp conversations, on where the messages are going.
You don't need to know the content, right? So there's a lot of these techniques where you can de anonymize traffic, you can de anonymize people. You don't need the facial rec.
You can ban the facial rec, you can still do gate analysis and speech analysis, voice analysis, handwriting analysis. There's so many other techniques. And all of these things will be available to AI.
To what extent does this whole, we're going to make sure that we're going to have privacy thing? Are they are they being naive? Is it gonna be possible?
I mean, I think The US has done a worse job of this than pretty much every other country on the planet. So I think there's, like, some basic rule. I don't think you wanna do rules at the level of, like, no facial recognition.
I think you wanna do rules at the level of, like, no data brokers. Right? Of, like, you can collect data, but if you're gonna collect it, the user needs to know and they need to have, you know, notice and consent or I'm not deep on privacy law, so I don't wanna pretend that I have the right the great privacy proposal here, but I do think there's ways to do it that are better than The US, and I do think there's ways to do it that give you that sort of underlying flexibility.
Yeah. Yeah. Maybe I'll leave it at that because I yeah.
Again, I know privacy law goes real deep, and I'm not there.
you know, we in the report, you talk about the possibility that the gap that we think is currently small between the models we have and the models that are used internally could open, and you have some recommendations around certain transparency measures. I wanna give one quick shout out to the AI whistleblower initiative founded by my Mhmm. Friend Carl Hoch, who has engaged with OpenAI and at least played some role in their recent updates to their whistleblower policies, which I find, like, amazing that OpenAI is continuing to work in that direction even today.
Where do you think we are on the spectrum from secret nondisparagement clauses to, you know, where we need to be in terms of insight into what is going on at the labs other than, you know, private philanthropist funded whistleblower support, which you can, avail yourself of, again, be the AI whistleblower initiative. Should you need that, what other policies you think the government should be doing? And and maybe even more broadly, if you wanna zoom out, like, what do you what what do you think a situation hypothetically, situationally aware US government should be doing in general that it is currently not?
Yeah. I think there's a bunch of things here. I think on transparency, think we're doing better than we have been.
So we have these two new state laws, SB 53 in California, RAISE in New York. I think those are good starts. I think something that it would be great but but I think for a lot of this information, we're also just really dependent on what the companies still choose to put out.
Now we're fortunate. I wanna give credit to both OpenAI and Anthropic and to somewhat lesser extent Google. Like, they do put out pretty proactively, you know, a pretty good amount of information.
So I think they should get some credit for that. But I don't love that it's like almost entirely at their discretion what it is that that they put out. I guess that will be shifting as SB 53 and Rays start to be enforced.
So I'm interested to see what that that looks like. I think also we need to shift to a bit of a I think there's been a start the beginnings of a push to shift from a, like, model release based schedule to something more continuous, which is partly driven by kind of interest in these, like, internal deployment type dynamics, not just the external releases. Mhmm.
So the idea here is, like, if the if the risk is not actually purely tied to when you put your model on the market, then all of your risk evaluation shouldn't be tied to that either. That also creating kind of better incentives for the companies around not forcing them to just, you know, rush things out the door, but instead trying to have more of a sort of continuous pulse of, like, updating metrics over time. So I think we could definitely be doing better on transparency and then, you know, ideally pairing those requirements as well with some kind of independent audit requirement or independent way to let people come in, external third parties come in and check that things are happening as they're supposed to be happening.
That has been in several of these proposals and keeps getting stripped out by industry lobbying. So that I think is, you know, a new frontier as well. I think there's various other kind of policy implications that we put in the report, some that are maybe interesting.
Is just like this general, like, hardening the world sort of recommendation or, you know, resilience, societal resilience is another way of putting this. So this is, like, cyber defense, biodefense, biosurveillance, you know, investing in biosurveillance just meaning like monitoring diseases, not like surveilling people. You know, investing in like epistemic security stuff of trying to have a way to, you know, determine what's real and what's fake.
Mhmm. Tagging real content. You know, all this sort of like broader societal resilience stuff, which is like, okay.
Just assume that this is gonna get much, much, much better, and then we might see a r and d automated, you know, a r and d contributing to increased pace of change. I think also there's been some this is less of, a policy and more of a mindset. I think there's been a shift over the past year or two to like, oh, actually, maybe open models are always gonna be pretty close behind.
And so, you know, concerns that you might have about there being, you know, an access gap or a, you know, concentration of power kind of gap if the closed models are far ahead, then we don't have to worry so much about that. And I think if you're taking seriously the possibility that automating r and d speeds up the closed labs significantly, then we just need to revisit those assumptions about kind of open models and closed models. There's a few others, but, yeah, would would point people to report for kind of the full set.
When AI builds AI, things, just might start to get weird. So, yeah, definitely check out the full report from Helen and coauthors at CSET and beyond. It's, interesting times for better or worse.
Any of the closing thoughts before we break?
No. Great to be on. Great to chat with you as always, and, yeah, look forward to next one.
Indeed. Cool. Hello.
Very nice to be in our timeline between now and next time.
See you.
Cheers. Bye for now.
So our next guest is, Jeremy Harris. He's from Gladstone AI, and they wrote the first ever US government AI threat assessment for the State Department. It's been about ten months now when they said every American AI data center is compromised.
Jeremy, what has changed? Have things gotten better or worse?
Yeah. Well, to piggyback off, I think Nate Nathan just said, things are getting weird. So things are weird.
Yeah. Great to be on. What has changed since then is less than one might have hoped and for really interesting reasons.
I think one of the things that a lot of people who are on the sort of concerned about the AI risk story and just kinda like the AI threat landscape and national security perspective, whether it's loss of control or weaponization, Like, a big part of the story that's missing is under understanding the infrastructure build out. Like, what are the actual bones that were building on here? Because that's the substrate that underlies everything, and there are all kinds of assumptions that are are being made about it where we're kind of abstracting away what I I really think is, you know, at least 50% of the problem here.
And we we think a lot about model reconstruction attacks and all kinds of interesting debate about, you know, whether it even makes sense to secure models in a world where you can just reconstruct them if an API is available. But more fundamentally, when you're building your entire AI industrial base off of components that are made in China with personnel who often are Chinese nationals. I mean, this isn't even, like, forget about the Manhattan project.
We're so so far, you know, kind of behind that. The I I think it's incumbent on us to, like, take a little step back and just ask about, like, what is that chessboard even what is the board itself? Forget about the pieces, but are we playing on something that's fundamentally stacked in a way that doesn't allow for a winnable outcome?
And I'm not saying this to be pessimistic. Think there's actually their solutions that you come up with very quickly once you take that new perspective, but, kind of closing your eyes and and not looking at it doesn't address the problem. And I think we're in a space where we're doing a lot of algorithmic level thinking because that's what so much of the Western economy now is based on, people at keyboards who are used to.
We're not making t shirts anymore. We're not filming transformers anymore. We're not doing that stuff, and so we tend to like to pretend that it doesn't So that's kind of like, I guess, my more recent lens on the problem is the last two years.
I know that's not quite an answer to your question, but that's kind of the chessboard as I see it at this time.
When you look at it end to end, you have the software piece and the talent piece. 50% of top AI researchers are Chinese nationals, and that includes people working in frontier labs in The US right now. And then you have the infra piece, a lot of stuff is coming from Taiwan, South Korea, Some of it is coming from China too.
And then you have ASML sitting in Holland, which is supplying into TSMC. And then you have ASML suppliers. They have like 3,000 odd suppliers spread across the world.
They're buying, I think, neon gas from Ukraine. When Ukraine got invaded, they had a problem. All of these missing pieces, all of these pieces spread out across the place, right?
And TSMC has been upfront by saying, We are only possible in a safe, globalized economy. If we ever got invaded, everything's over. We can't do anything.
That's it, it's fine. So where do you think, how do you think that fits in with a threat perspective? It seems like someone just has a dead man switch over TSMC.
So how does that work in terms of security and securing US prospects and the future in The US?
Yeah. I think it's a great question. That whole Taiwanese kind of scenario planning thing is something that everybody has talked about.
I'm not so sure everybody has kind of like worked out the implications to to full satisfaction. Yeah. I mean, so first of all, yes, if Taiwan gets invaded, TSMC is gone.
It's gone whether it's because China takes it or because it's as I would expect in whole booby trap to the nines to blow. Right? I mean, it takes, like, you know, hundreds or thousands of, like, insane level PHPs to to tweak that you think of it as like a a giant box of 500 dials, each one of which has to be perfectly tuned to keep these things pumping out at the right yields.
You're not gonna replicate that if you're missing either the equipment or the people. So, like, this is extremely fragile, even the most fragile production process that primates on this planet perform. So an invasion is unlikely to leave it in China's hands.
And so yeah. The question is then what do you get when you roll that back? What's the number two positioned entity?
And then you start thinking about, okay. Well, what does SMIC do? What can it do?
And the SMIC Huawei complex does seem like a very plausible runner-up, especially when you look at scale production, especially when you look at the emphasis Huawei's placed on, you know, networking large numbers of GPUs together. They don't have to be as efficient as ours. They can't be.
They don't have the logic, but they can be networked together way better, and that's how they get effectively, you know, competitive scale performance. So so this is a real issue. You also in a funny way, this interacts somewhat positively with the energy bottlenecks that we have here anyway.
Right? Like, we're we're gonna be bottlenecked by energy probably sometime around the end of the year. When that happens, TSMC's ability to outproduce I mean, it gets complicated because on a per chip basis, it's they're way more they're they're way more energy efficient.
They're more they're pumping out more flops, but they're we do have that energy kind of ceiling on our like, that's the main constraint that we're there all towards. So the timing matters a lot here. Right?
That dance between how much does logic matter, how much does energy matter, how much does memory matter, all these things, how much does packaging matter, all four of those things have become bottlenecks at different parts of the game in the last few years. Another piece that I think, again, when we think about the actual bones that the AI economy runs on, it's not just chips and and not just like the data centers themselves. The power grid is a really just generally vulnerable target.
Right? So, like, we we know that that, for example, there've been there've been components in Chinese transformers that have been snuck in explicitly Trojans to be able to to take down our gear. A very plausible scenario, just like based on talking to folks who are working this problem on the IC side, is a a Taiwanese invasion begins, and one of the first things that China considers doing is just shutting down the West the Northern Brit.
It's kind of obvious. If it's existential, you know, that's massively escalatory. So there's huge there's huge question marks there, but it's it's a scenario that's being taken very seriously for all the reasons you might imagine.
So yeah. I mean, I I think when that if that happens, there are questions that suddenly run much deeper than just our ability to literally make chips where you know, in Arizona or wherever wherever the next thing is. But, literally, like, if we can be kneecaps economically and more fundamental level, we don't even get to look at the chessboard that we hope to look at.
We don't even get to indulge in the, oh, well, what can Samsung do versus what can SMIC do versus x n you know, CXMT, like, these things. We don't get to play that game. Like, we literally don't have an economy.
Like like, there's serious implications there. So if we think about this as a game with the stakes that it might have, and this is contingent on what's between Xi Jinping's ears and the Politpiro's ears, but this could end up looking like, you know, we're preparing ourselves to take a punch in the face, but then we get we get kicked in the balls, if you will. I mean, it's this is the kind of scenario.
They use the technology that that we may be virally towards. And, again, that zoom out, I think, is really important. We're we've got target fixation here on what could be a pretty narrow part of the test board.
had some ideas on not only do we need to speed up, but we need to slow China down. And what was your concept around slowing China down? Because they're trying their best, they are definitely not there on the chips yet.
The Ascend, the Huawei Ascends, 910s are, doesn't really like them. They want to get the H100s in. There is this concept of building on The US AI stack.
It's also revenue denial. If you manage to flow the revenue into NVIDIA versus flowing the revenue into Huawei, Huawei has more revenue to develop those chips, so therefore we should deny them. Like, how does this balance out this, like, letting them take on the chips but not too powerful and but still enough that it doesn't create a market for Huawei?
It sounds like a very delicate balance here.
It it does sound like a very delicate balance. I personally, I'm I'm sort of less oriented towards the argument that says, you know, if we just let NVIDIA do business in China, then the Chinese will go, oh, sweet. We have NVIDIA that's serving our needs.
We don't have to push so hard on the gas on this issue that's been identified for years as the possibly number one national technological priority that we are pouring multiple Apollo moon landing that like, amounts of cash into. Like like, this is, to to me, a kind of a a mistalibrated sense of the even just the messaging that the CCP has been putting out. Right.
I just don't see a world like, you know, we we don't see, for example, NVIDIA, okay, they can ship the h 200 or whatever it is now, and then suddenly the CCP goes, oh, okay. For yeah. Forget about that quarter trillion dollar investment that we just made in PBP turns into into kind of our our national AI chip capacity and all and infrastructure.
Forget about that. You know, we'll we'll stick with the NVIDIA play. There's a sense both that the ability to access these NVIDIA chips is transient because the next administration may just as well pull it down, but also that, like, why not both?
I mean, it it seems like an insane thing given that AI is a matter of, like, national security importance for China. It'd be pretty surprising to me if they just decided to respond that way, and it seems like they haven't so far. So I guess that that you know, that's why I think about the export control thing from a slowdown standpoint.
They have worked. We know from DeepSeek, like, the public statements of their CEO before DeepSeek was on the radar. And this is actually, I think, really worth noting and and kind of, like, under under recognized and appreciated.
Like, before DeepSeek was on the radar, they were coming out and saying, hey. We really think we could do this AGI thing, though, like, one issue. There's just one problem.
We can't get chips, and these export controls are killing us. Then, obviously, r one drops and and everything has got deep seq, and, you know, they get dragged in front of the the public bureau or whatever and debriefed, and suddenly things change. You get these little trickles, these little leaks of similar information that come out the edges of this kind of Chinese AI ecosystem every once in a while.
But it's pretty clear that the export controls were worth it. This is if nothing else, just look at the the massive orders that are gonna be coming in for the h 200 to show how much demand, pent up demand there actually is in the AI ecosystem. And, of course, we know all about the frustrations of of AI companies in China and about the current way to win for their chipsets.
So, yeah, let me add one up. This is that's kind of like my my bias take is, very much towards the direction of, I think we gotta listen to Chinese companies when they tell us that our export control policy is is working.
Mhmm.
Maybe I'll come back to some of the
the frustrating,
duality of, like, difficulties where on the one hand, we have you you have expressed very low, hope for the opportunity or the possibility of meaningful true collaboration between the West and China. And then at the same time, I think you're also not super optimistic about our ability to create a super intelligence that we can actually control and get to do what we wanted to do. And I think the way I think about our conversation from a year ago or so and and your contribution to the broader discourse with America's Superintelligence Project is like, those two things are both real.
They're both true. And, like, you're kind of engaging in motivated reasoning if you try to deny either one of them. With that in mind, we are now also seeing like some of the potentially foreshadowing kind of moments on the, on the AI side itself, right?
Like just in the last week, we've had these new models from Anthropic and and OpenAI, and they've both kinda said, we weren't really able to run the evals like we kind of intended to. Anthropic basically said the eval awareness is pretty high, and so we'll just do a survey and a little internal survey of, whether or not this is safe to release. That that's probably a bit of a simplification on my part, but I think that is a fair enough summary of their of their position.
Then OpenAI similarly was like, well, the autonomy risk of the part of our preparedness framework, that's also pretty hard to evaluate. We don't really have tasks that are kind of long enough horizon that we can get a real handle on just how autonomously capable a new model like five three Codex is. So that's kind of crazy.
And yet, of course, both models are put out there. I don't see China driving the need to do that. It seems like they're doing that and doing it on the same day, notably because they're, like, their own competition between the two of them and also just sense of, like, rivalry seems to be heating up.
They're, like, going at each other in Super Bowl ads to some degree at this point. Not something I thought I would see from Anthropic at the beginning was like a, Super Bowl attack ad, but here we are. What do you make of, like, the dynamics between the Western companies?
slightly pessimist hat for a moment, I would say it seems like we might be racing to the bottom, which was exactly what we were hoping to avoid. Yeah. I think we are racing to the bottom.
I think the only the only frame that makes any sense, you know, like if we're gonna talk about, okay, we need to regulate this technology say domestically in the same way that everybody from mall leading ad companies have been saying for, I wanna say over a decade pretty much. You're never gonna do that unless you deal with the outer loop, the outermost loop, which is international competition. I think, like, there there is no version of the I don't think anyone I I I think, again, we can enjoy the indulgence in target fixation of, like, oh, yeah.
Let's play the game pretending that other countries don't exist. But in the same way as ignoring the infrastructure target fixation or sorry, the the algorithmic target fixation and not lean infrastructure, this causes us to miss what is really the entire problem. So you are not going to get to a point where, you know, you can have a a strategic or, I should say, a tactical slowdown when you really need it, where you're like, okay.
Suppose we find that the next version of whatever model can design custom bioweapons, execute catastrophic malware attacks, all these things that are, like, entirely they're they're entirely plausible and that no, you know, counter jailbreak measures are are truly a 100% effective against the kind of people we'd be worried about, yes, you would absolutely in that world need somebody to be able to say, okay, guys. Tactical halt. This is insane.
We can't be in a, you know, in a universe where you get a nuke and you get a nuke and you get a nuke. It's we can't do a pro one free for nukes. Okay.
So what are gonna do? We're gonna have to have a slowdown. If China still exists and has their program and they are x many I mean, I'm repeating all this stuff that everybody said a million times.
You know? If they're twelve months away, six months away, I don't care. We've got a shot clock now.
That's the situation. So we have to start there. Like, we have to start there and say, okay.
Any serious solution to this problem will involve dealing with China. Two ways you can do that. One is you have a kumbaya monu with China.
There are a lot of interesting reasons why I think this is just, like, not gonna work. One of which is if you think about international treaties, they don't tend to reflect some sort of Star Trek y commitment to everybody on planet Earth wanna do the right thing. They tend to reflect the railpolitik kind of, like, label in terms of actual power.
Like, nukes, you have nuke drawdowns when everybody can retain arsenals that can still destroy the entire planet three times over, and there's literally no point in building the marginal nuke. You have similar things if you actually look at the history of bioweapon and chemical weapon treaties is, like, you find in every case that, actually, they don't give you the marginal lift over just, like, touring people with artillery and gunshot. It it looks nice, and and they often get adhered to for that reason.
But then at the margins, you know, you have, like like, Chinese research labs on American soil doing all kinds of crazy research. You have whatever facilities in Wuhan. Like, all this stuff happens anyway.
And so this may sound, like, super cynical. I think it just reflects the way things work. That's at least my take.
I I would think that. So the question then is, how do you how do you deal with an adversary like China that's in the position they are that does have a stranglehold as they do on our infrastructure? They simply they simply do.
So the question is, what are your offensive options? That is it. There's like you're not going to build the perfect Fort Knox.
This is not not a thing that's possible. And so the question is, what do you do to induce consequence on the other side? That's the only math that will work if my theory of the world is correct.
It's not a pretty theory. It's not one that leaves us feel warm and fuzzy inside. It's one that may make you think a little bit about, you know, mutually assured destruction, that sort of thing.
I think there are nuances here with yeah. Obviously, Dan Hendrix had his his frame on it. But bottom the bottom line is, yeah, I mean, I I think you you you kinda need it doesn't need to be an AI based response, though eventually, you know, you can certainly argue that any offensive option that isn't coupled to the scaling laws is eventually going to be beaten by something that does.
Right? So so there's kind of an important design principle in these things. But there are offensive options that need to be explored.
And this is unfortunate, but it does mean that if you have a situation where your adversary can turn to you at any time and say, hey. Watch me turn the power off on your entire grid and have, like, tens of millions of Americans go more to die of starvation or exposure. Mhmm.
Like, you need the ability to say, okay. You know, watch the same thing happen in in in Beijing, and we can turn it back on, by the way. We need to have this de escalation option.
Mhmm. I know it's a bit of a grim view, but I think that, like, when when I think about what actually gives leverage in this situation, it looks a lot less like what's had it pretty tricky, especially given the history of countries like China, like Russia. With respect to treaty adherence, like, they they sign treaties.
Like, we know what it looks like when China signs a treaty. It it doesn't end up being pretty in a situation like this where you just you lose perfect adherence at such a high level of precision. Like, there's no version of an international treaty on AI that doesn't involve inspections of compute stockpiles and, like, very precise Overwatch of the kinds of algorithms that are being deployed, like, the the kinds of, you know, evaluation schemes, like, the the level of of cooperation that's required to do something tractable here strikes me as being quite significant, and the the trust just, like, I don't see it being there.
So what's your PDOM and and on what timeline? I mean, we were just talking with Helen about this report that they put out about when AI builds AI and the possibility for curse of self improvement. It sure seems like all of the vague tweeting that is going on right now out of the Frontier Labs is suggesting that that is happening.
And then on top of that, of course, they've know, OpenAI has public timelines that they've put out, I guess, to their credit, maybe. I guess you could we could see that both ways. The Amtraffic people that I talk to are, if anything, always the most firm believers that the recursive self improvement dynamic is unavoidable.
How long do you think we have before before these things really start to take on a kind of runaway dynamic? Is there anything that you you know, if you had power, you know, and a lot of power, is there anything that you feel like you would want to bet on? And and where does that leave you in terms of p doom?
Maybe I should just stop all this and spend more time with my family.
Yeah. In general, I I'm a big fan of the happy warrior mindset. I I think it's, like, just never constructive to be to go in whole first of all, we have to assume that no matter how firmly we might believe in whatever outcome, we may just try not to be wrong.
There's a famous story of Richard Feynman walking around New York City in the seventies, I think it was, looking at all the skyscrapers and being like, wow, isn't it sad that all of this is gonna be wiped out by a nuclear war between Russia and The United States sometime in the next few years. And he was just that's just that was a a fact of the matter, and it reflected a pretty reasonable understanding of the dynamics unfolding between those countries at the time. I'm not saying it's ever quite that simple, but this is an ingredient, if nothing else, that makes you less effective if you're just stuck in a hole all the time.
And just as a meta point I guess, that I think is the first piece. You know, we have to act with agency and we're gonna be most effective doing that if we're not stuck in a kind of deterministic Calvinist frame with this whole thing. In terms of I'll also not answer your question before I answer it just by saying regardless of timelines, one thing to focus on is that there's some things that are pure optionality plays.
And so, like, we there are things that you do if you're going to build a Frontier AI cluster, right, at scale that rule out nation state security at that cluster. Mhmm. Just rule it out.
If you don't do these things right on day one, by day, you know, 360 once you've finished building the site, your site is going to be compromisable, and there's no going back in doing that. We think of these as we call it like the one way doors, right, of of the data center construction process. Figuring out what those one way doors are, setting standards for them, and actually executing on that, even doing it just, like, voluntarily.
Right? You're like, we think about opening eye and and and tropic and so on, like, kind of all independently going, hey. We just wanna buy that optionality because at some point Can you give me a concrete example of a one way board?
Yeah. The the so so there's a bunch that I can't go into, but, one that I can, it's pretty easy, is think about the people that you're getting in the loop to review the site plans and details that would be, let's say, useful to an adversary who is trying to extract information. If those people are Chinese nationals, okay, you're done.
Cool. Mhmm. Right?
Like, you're never gonna unfuck that. That's baked in. Right?
So these are actually, the the interesting thing with these one way doors is they tend to be surprisingly cheap, and that's the the tragedy of it all is, like, you actually could if you were thoughtful, go through and be like, well, on a fraction of the budget that would be required in CapEx and OpEx for these builds. You just, like, create pure optionality by implementing the same. So I think that's, like, a really important element.
Putting offensive options on the table is a pure optionality play. You don't need to exercise those options. You need to have them on the table.
That's what I'm saying. I'm not saying let's go to war with China. Let's like that's a crazy thing to say.
There's there's all things in their context, but you need options. And that's a crucial thing. So having an understanding of, you know, the mapping out the ecosystems that are relevant, the AI ecosystems that are relevant, and and thinking about, like, what what might that that endgame play out to be?
Those seem like pure optionality plays regardless of timelines, all cheat, all things you can do quickly. Again, like, this is it seems to me kind of and I'm not saying they're they're not being done. It's just that often there's a a lack of sort of focus on the on the endgame here, anyway, without getting into the weeds too much.
Okay. So PDOM and timelines. Oh, sorry, Prakash.
Yeah. No. Go ahead.
Go ahead. Oh, yeah. Sorry.
PDOM and timelines. So I'll almost say PDOM, I don't find it useful. Like, I I know I know what I'm focused on.
I know what I gotta do. Mhmm. If I start thinking about, like, my generic answer has been for years.
Like, any number between 1090%, I'll take as, like, that's a reasonable number. I'm not like, I I've I've read the debates. I've seen the post on that's wrong.
Yeah. So is that your P doom or P loss of control to superintelligence? Because I in some places you've mentioned it's a loss of control to superintelligence rather than doom.
Yeah. You that you've obviously done your homework really well. Yes.
That that is more of a loss of control to superintelligence. I think I think by virtue of the way that numbers multiply together, I don't know that my my answer is that different for, like, p doom in general. Again, this is coming from somebody who, for better or for worse, has almost explicitly not put in that much time to to kind of wallow in those numbers as I as I think we're we're sort of all tempted to do.
Right? Like, I'd have that temptation. I I get it.
Like, I I mentioned I had a daughter. Right? Like, I I don't like the landscape that's playing out, but I had a daughter.
Like, I chose to have a daughter, and and I didn't have her in, like, 2018 before the scaling laws blew up. Like, this is, you know, this is a choice that I made. I I think there's there's a a kind of almost like a spiritual risk to getting locked into that kind of thinking, and I say this to somebody who's experienced that.
You know, I I went through that and found out how it it ground my ears. So, yeah, so I guess I'll I'll just, like, you know, not answer the question by saying 10 to 90% sounds reasonable. I think if you're below 10%, I I just really think that there's, like, there's homework you gotta do because a lot of these scenarios are maybe sound crazy, but they're just they're a lot less crazy than they seem.
And when you get into the nitty gritty, it's like a lot of these scenarios are already they're halfway one fulling. Not and, you know, if above 90% I mean, first of all, if you live as if you're above 90% and you're like, that that's gonna be just make you less effective. I also think, again, Richard Feynman certainly seemed to to think he was in that ballpark.
There's just like an epistemic question here of how quickly does the world adapt. I think we're constantly surprised by how quickly the world adapts, both how fragile and how resilient it is. And fit like, the the, you know, the eleventh chapter of the book will often involve a new character that comes out of nowhere, and we just need to kind of make sure that we keep uncertainty about our uncertainty factored into this analysis.
And I think that buys me 10% pretty easily. I've been wrong on stuff that I thought I was 100% on often enough to be like, okay. I'm not, you know, I'm not gonna push it that much.
And I know that's frustrating for, like, a lot of people. Like, no. No.
No. But, like, look at the math, man. And I I get the math.
They're but what I'm questioning here is just the the process that led to the math, and I don't know that I can plausibly ever get fully behind that process and interrogate it with with confidence. So last thing is time lines. I thought AI 2027 and and, like, contrary to I think Dan has kinda pulled pulled back a little bit his time lines from now.
always meant 2028, but now it means 2029.
Yeah. Yeah. And and and, yeah, AI is the apocalypse of the future, it always will be.
But, you know, not not actually, but, know, I I think there's, like, I'm sensing which so when g v three first came out, I was like, oh, man. I've got two year timelines. And and that was because I didn't understand what the hell would be involved in the infrastructure build out.
And and now that I have a much better understanding of that, I'm still kinda like, what's the next bottleneck gonna be? I'm very uncertain about this. And, again, it's one of those things that doesn't really affect what I do just because I'm so focused on just all the low hanging fruit that we have to pick right now.
There's so much stuff that we're just not doing because we're paralyzed by the problem. So I think in terms of what we do, there's pure alpha on the table in the short term. Twenty twenty seven doesn't sound insane to me.
Twenty thirty doesn't sound insane to me. Twenty thirty five sounds a bit far. I guess I'll I'll sort of leave it at that as a spread.
I think we should be acting as if 2027 is plausible. I I think it would be unfortunate if we if it happened in 2027 and we're like, man, we had a lot of really plausible analysis that pointed to that and we just didn't do anything, that would be a shame.
Can you give us a little bit more of a hit list in terms of the low hanging fruit that you wanna see us pick? I mean, it's we've got the one which is, like, build at least some subset of our data center build out in a secure way so that we can run hypersensitive projects there as needed. What else is kind of on the if you if you're the replace David Sachs as the next AIs are, what's gonna be your priority sheet?
Yeah. I mean so that that first one, by the way, is a lot of things. Right?
So it's it bundles together. I mentioned the the personnel security issue inside of threat problems. There are a huge number of things in that bucket alone that are necessary and and contribute very cheaply to much more optionality on the security side.
I think, again, you zoom out more so you look at the grid. What could you be be doing to introduce redundancies quickly? Like, the the supply chains that lead to a lot of these components are very clearly sourcing heavily from China.
You have so so here here's, like, an easy win. Look into companies that are offering to build data centers, like, suspiciously fast and who owns those companies. So so there was actually a a letter that came out from the House of Luck Committee on CCT a while ago many day one data centers, right, as as an entity that is somewhat suspect.
And, you know, they'll have these data centers where data center building companies where it's like, oh, wow. And, like, you could build stuff, like, way faster than anybody else. It involves sourcing components from China.
And, like like, my personal opinion is, if I were to see that, I might be asking myself the question, China is kind of a command economy through civil military fusion. If the CCP wants me to have this very, very rare and precious and backlog component for my data center in the Continental United States, that might tell me something about how much faith I should have in the security and integrity of that component. You know, there's there's just not a lot of infrastructure level kind of attention being paid to these things.
And the labs, by the way, like, they wanna do the right thing here. They don't wanna be in a position where they're getting a company to to build something for them, and then it turns out that that thing is compromised and it comes out that that is not good for anybody. So incentives are aligned there.
There there's just, yeah. It's it's things like that where there's there's been so little attention paid to the bones that that there's just tons of stuff we can improve, including with AI. Right?
Like like, looking for looking for malware in in old software that's load bearing for our infrastructure or not malware, but rather vulnerabilities and, you know, finding ways to harden it. So, yeah, this is a a defocused answer, but it hopefully gives a sense of of the the menu.
One thing we haven't really given you a chance to flex your ability on in this conversation is just the breadth and depth of your technical understanding of so many AI developments. And I definitely recommend the last week in AI podcast, which you usually host as a great source of, I think, very sophisticated analysis by, by both of you, but but I tune in for you mostly, to be honest. And I wonder I wonder how you are doing it.
How are you keeping up? What is your how have your methods evolved so that you're maintaining situational awareness as much as you can?
Well, thank you, first of all. And it's very kind of you to say. I I have I've told you this before, but I I do actually watch the cognitive revolution.
And it's I think it's you know, there's a lot of the ecosystem here is really rich, and interviews are really important because you get stuff that you can't get from the papers. And, I you tend to focus more on the papers, so I just don't get that kind of that kind of analysis. And I I just talk to friends from the labs, but it's different from from those deep dives.
Yeah. I mean, so back in I can't remember when I started on last week any on it, but it was, like in maybe 2021 or something. And back then, I would just read papers and you couldn't use GPT-three to help you understand a paper.
Just wasn't a thing. Now that's changed. I've had an experience that was kind of frustrating this week in particular because I'm I'm preparing a like a state of play briefing for a customer.
And, basically, they wanna know, what happened in the last quarter in the world of AI that we should be tracking? And there was a paper, that I had, Gemini help me with, and I got to a really good understanding of how it is like a residual like, dynamics of radiant throw flow through this, like, residual and have a whatever. And, it was pretty complex.
What I realized though, after having an interaction with Gemini for long enough, I switched over to Claude, and I was like, wait a minute. I just, like, hallucinated my way through that entire conversation, got to an understanding where I was like, oh, yeah. Like, I'm pretty smart for figuring this out.
And also, like, I got this nail cut, and everything got flipped, you know, flipped around. So I'm not saying that always happens, but that has been kind of the most recent update in my process is is really, you know, be mindful to kind of double check, especially as you start to get lost in a rabbit hole. Yeah.
I typically spend, like, I would say about 30 to 40% of my time reading the paper and then the rest interacting with a model about usually, it's, like, the implications of the paper or what it is is it's reinforcement learning versus supervised fine tuning. Like, if I'm reading the paper, I'm doing SFT. Like, that's what's going on.
With with the models, I get to actually, like, go on policy, and I get to test my own understanding. Like, I would have done this experiment differently. Is that a stupid idea?
And often, I'll get a pretty good answer, and that it makes you feel like you're rotating the shape instead of just staring at it. And that, for me, has has been just really helpful and empowering. It feels empowering.
Do you have any particular, like, workflows, pipelines, whatever that try to filter things for you and surface what you really need to spend time on? Because that is as challenging I mean, it's more challenging than ever, and it seems to be maybe as big of a deal as being able to successfully make sense of any one thing is like, what are you gonna choose to spend your time on in the first place? How how has that evolved for you?
Yeah. It's a great question. This is that like age old question of taste.
Right? Like and and one of the things that I've had to come to accept is I can't develop good taste in all the domains that we wanna cover on the podcast. Like, I'm never gonna, you know, like my my taste is basically if if one of the frontier labs puts out a piece of research or if a researcher I know and and like have a lot of respect for appreciation for put something out or as a co author on something.
I'm gonna take a really hard look at that. And then besides that, I have the usual Twitter account, set of Twitter accounts that I follow and that's another way. But my passes at these papers are I'm I'm pretty focused on the, like, what's on the critical path to ASI question, not that I know the answer.
But just, like Mhmm. I'm trying to find things that to me gesture at that, which is why, you know, I don't tend to talk about like GANs or the latest in well, I was gonna say the latest in text to video. Now that seems like it could be on the path, so you never know.
But I guess part of it is just acceptance. I am reading these papers for the concepts more than the outcomes. And often what'll happen is there's this paper that'll come out, and it might not be the perfect paper to cover from a given topic area.
Oh, you know, residual connections and and really optimizing the crap out of them to get ultra deep transformers. There's this paper about it. Is this the best paper?
Probably not. But the reason I focus so much on explaining the underlying kind of concepts on the podcast is that, a, there's gonna be another paper next week that obviates whatever the hell the last paper did. And and b, I I think the the kind of core concept is the the most of what the landscape is the most important thing.
So when there's another paper that comes out about optimizing residual connections, you're like, okay. I'm familiar with this playpen. Like, I know I I know the furniture in this room.
Can rearrange it a little bit, be more confident. So I guess the answer is just like I I get around the taste issue by not having it, which is maybe just funny.
What's underappreciated for you right now by AI obsessed people? I mean, there there's, of course, like, in the broader world, AI is underappreciated, and just how crazy things might soon get is, I think, very probably underappreciated. What do you think I might be missing?
What are what are the most likely blind spots for somebody like me that you would wanna draw to attention?
I guess the the challenge with blind spots is that, like, we all have them, and by definition, we don't know that we have them. So what I'll try to do is roll back and tell you about my blind spots as of about two years ago. And and that was around the time that we we put together that that report that Prakashi mentioned earlier.
So I sound like a broken record, but the infrastructure layer, the kind of the the stuff that this might sound like a wrong way to put it, but the stuff that feels too blue collar to to most people who are AI obsessed, like I am, you sort of start to realize how much of the world is actually built on infrastructure that we just abstract away. So I think that's actually really important and needs to be foot stomped. Understanding what is the down to, like, what is the dynamics of the leasing process that Frontier Lab goes through to get a new piece of land?
Like, what can screw up there? What causes delays in construction projects that we talk so much about? You know, this lab has a you know, x AI has the their new Colossus cluster, and it's gonna be online shockingly, like, to one gigawatt sooner than Anthropix that which surprised everybody and all all this stuff.
When that when that happened? Like, when was the actual driving like, because that's gonna tell you if you believe in the scaling laws, that actually is a probably one of the most important variables that you ought to track is, like, delays in construction processes. Sounds pretty mundane, but, like, hey.
The world runs on it. And procurement schedules, thing and things like that. So I guess that that's one piece that I've been missing.
Another one is, like, how, yeah, how kind of real nation state and security happens. And as hard to get information about that, you know, one of the biggest challenges there is that there is no such thing as one nation state security capability. Nation states are siloed, obviously, because security you can't have tactics, techniques, and procedures that are exchanged between silos because then there's no information security.
And this by definition means that that you would have to go through a process of taking team a, comparing them to team b. Well, okay. Team a wins.
Okay. So and then next day, like, you'd have to go through that kind of selection process, run-in the low score type situation to even know what the most exquisite capabilities are that, like, we could feel, and it still wouldn't tell you quite what other other countries can be. So that's kind of like anyway, I think a really important dynamic that's very easy in the AI security context, especially for physical security especially for physical security, which is, again, undervalued precisely because we tend to abstract away.
We focus a lot on cyber because it couples it couples to AI, and it feels like it's in our sweet and nerdy space, and I get that, and I love it, and it's critical. But it's also not if you look at, like, what the Russians do, they, like, they do cyber for sure, but they will go up and and, like, they will arson your your transformer. Like, that's not an issue then.
Mhmm. Had examples of of that sort of thing happening. So, anyway, there there's that piece.
Maybe the last one and more on the in the kind of comfortable and familiar nerdy space that that I occupy is is just this idea of the distinction between having a model and then having the compute to run that model. Mhmm. You know, it like, if you believe in the inference time scaling laws, then, you know, model theft is one thing, but actually being able to point that model at, you know, basically have compute on compute war at inference time seems like a really important dimension.
And you see this play out in a lot of interesting ways, One of which is the Chinese ecosystem. Yeah. They have a huge number of users.
Mhmm. And then they have some, like, language models. Problem is that their their labs are all flooded with, like, these inference requests from their giant user population, which leaves very little r and d compute for just, like, innovation and improving of models.
And so that's actually one of frustration for Chinese labs much more than than labs here. They're just like, dude, we have so much demand, but we're not bottlenecked by money. We're bottlenecked by by compute.
And so the the dynamics of of, like, how inference affects training and then how like, what it means to steal a model and what it means for model on model warfare to happen, especially in cyber behavior. Like, cyber hardening has a certain amount of test on compute. That test on compute, you know, it's gonna be focused on in some way, and then the offense side is gonna have a certain amount of test on compute.
And how how those play out, like, relative budgets matters a lot there. Obviously, if you're defending, you have a wider surface area, you gotta defend it. But there yeah.
Anyway, there's a whole debate there. That, I guess, would be another dimension. It's just, like, going beyond just owning a model.
Like, what about running it? What can you do with this model that you have?
I have one last question, which is you're pretty security conscious. Have you run OpenClaw, and what is your current personal productivity stack?
Yeah. Yeah. Yeah.
I have not run OpenClaw. I have so and actually funny you say that. I'm setting up a like, I have an old laptop that I'm gonna use as my my burner laptop for the purpose of exactly that part yeah.
Partly because, you know, I anyway, yeah, for the exact reasons you would imagine. In terms of my understanding yeah. So a big part of my my job is is becoming now constructing agentic workflows to to do some some, like, some things that are not super security sensitive, but more just like I'm I'm gonna try to use it to optimize my comms because that's a huge bottleneck for me.
And and for that actually, I'm I'm still in the discovery phase of trying to choose platforms. I'd be interested in in your thoughts for that actually as as I dive in. That's is literally, like, next week is is my deep dive.
So this is almost the worst possible timing because I think my answer is gonna be, like, horribly outdated. Yeah. It's a great question.
I wish I had the answer.
I talked about a little bit about Mind at the top. Interested to hear more about what Prakash is doing too. But for me right now, it's Claude Code as the, you know, kind of base product and then taking inspiration from a guy named Daniel Meisler, who I did an episode of the podcast with, who's created personal AI infrastructure, a frame that's an open source framework, and also friends who I just trade notes with privately.
I'm trying to create deep context for myself by first exporting kind of all of my digital history from all, you know, Gmail, Slack, all the other places where I kind of have these comms, get them into a local database. Then, of course, you need, like, a daily update process to, like, fetch the latest because you're still communicating on all these other platforms. Then layering on top of that summarization and different kind of angles on the data.
So one right now, I'm at the phase where I'm like, here's a month worth of all comms. That seems to, for me, come out to about 300,000 tokens. Now summarize that down to, like, 10,000 tokens of what a chief of staff would need to understand this month in Nathan's life.
So you can that, like, 30 to one reduction, then probably put, like, a year, you know, long version of that and then sort of the let's talk about the relationships and kind of have that sort of cut on it, the projects cut. And then hopefully with that deep context and the I'm also trying to have it leave pointers in those summaries with, like, regular habit of quoting any distinctive language so it can go search down the ground truth for the original. Hopefully, it will then have enough context to be able to, you know, not exactly right exactly as I would, but sort of come much closer at least to responding as I would, having the sort of the context necessary to exercise something like the judgment or taste that I would exercise in doing things.
And that was actually part of the process of setting up this episode. I gave Matt system 20 names and said, do, you know, research on these people, find out what they've been up to lately, give me a brief on that, and then also had it draft the outreach emails, which were only lightly personalized. And I still did kinda go in in a little bit before tweaking.
But I appreciate that. That's nice. But, yeah, I I don't like to publish or even send as, like, one to one communication AI output directly.
But I do find that I can get to something that I do feel comfortable signing my name to faster with an AI draft in many cases these days. And so it's very much a work in progress for me, but that's kinda where I'm at at the moment. And, again, it'll I'm sure by the time we, talk next, it'll have changed quite a bit.
got a couple of things that I ended up building out. One was a stock market tracker. I have a number of metrics which I think no one else watches and it's fairly hard to obtain.
And the great thing is is very good at financial math, very, very good, far better than I have ever been. And so it's relatively easy to talk to plot and kind of figure out what kind of thesis you have and then build out metrics precisely for that thesis to watch pickup lines. So that's been very useful.
I used to like do it in my head, right? Like you look at something, you look at something else, and then like you calculate the ratios of blah, blah, blah, blah. And then like, I realized that I was spending a lot of time doing ratios in my head, and I was like, maybe I should automate this, and so now it's all automated, it's nice.
I don't do the ratios in my head anymore, I just look at it and I can see the screens automatically, I can see what I'm looking for. And then the other thing was podcast clipping because we do a lot of podcasts, and content these days has to be repackaged into short clips in order to hit socials. And so that, I tried like six months ago, the tech wasn't there, and I tried about three, four weeks ago, the tech was there.
Everything works, transcription works, review works, selection works, everything works. This is, and this has been my experience. It's kind of like, maybe it gets it like 1% better, but that 1% better clears the hurdle.
And that's a binary step up. It works or it doesn't work, and that 1% just clears the hurdle. And I really feel like in the last month, a lot of things started clearing the hurdle.
Nathan, what you were saying earlier about the takeoff dynamics and the labs automating their own research, that all kind of maps very nicely. One the things on the financial side too, find so I find cloud is also useful on questions like so you might have a thesis, but then there's a question about how do I if I'm right about this, what's the best bet to make? Because that's a category of problem I've mentioned in the past, right, where it's like, you know, you'll have a thesis, but, you know, you're not gonna bet on Microsoft because OpenAI has such a tiny fraction.
Mean, it's already gonna be all of this How do you where's how do you leverage and torque that to this thesis? And and that's kinda something that, you know, the world is so complex that you just need something to peruse and and have all that knowledge. So if the finance use case is is a, yeah, really great one.
Great point.
Yeah, it's also been very, very weird in the market because I feel like Twitter is literally like a month or two ahead of the market. It's just been amazing. People tell you like TSMC will do well, and then three months later, it happens.
It's like, what's going on? And I was a professional financier. I've always expected that hedge funds get there before you do.
And in talking to my friends at prime brokerages and hedge funds, they are very negative on AI. They just don't believe it's happening. They believe it's like crypto.
They believe a lot of West Coast tech is just scamming retail investors. Index investing is the only thing that really works and everything else is either insider trading or scams. And that's pretty much what the prime brokerage guys and the hedge fund guys believe.
Like, you know, Medallion, right, or Gene Street, right? These guys who have AI in their bones. And these I guess Medallion, it's like they can't know, they only invest, you know, famous at 5,000,000,000 a year because otherwise they would actually move the markets and feedback loop.
But yeah, what about them?
They were down last year. So the impact is starting to be felt. Think, well, also Jim Simons died.
I don't know to what extent he was still supervising because he'd already kind of semi retired for like ten years almost, but medallion was down. There's some sense that it's also because they're losing talent to the lab too, right? You can't forget about that.
They're starting to lose talent to the labs. And some of the labs do have internal teams which will eventually look at trading on the market, I think. So we'll see where that goes.
Very cool.
Jeremy, thanks for joining us. Let's check back in on your personal productivity stack once you've upgraded it. And in general, let's, I'm reusing this joke everywhere I go.
Let's shorten the timeline for our next conversation.
I like it. Thanks, guys. Appreciate it.
Thanks, Jeremy. Cheers. Cheers.
So what do we make of it all? I mean, it's so, the big thing I can't get past in all this stuff is the amount of disagreement between and this has been commented on so many times, so many ways, right, up to the level of the Turing Award winners that can't, see the same phenomenon. Yeah.
But it seems to happen at kind of every layer of it's like a fractal problem. You go into these specific workshops around AI r and d. To get people from the labs.
I do understand that there are even people at the frontier companies that, you know, have heterodox positions and don't really buy into the hype. And then even, you know, with the AI for science, it's like, I I can't I can't make any case that I should trust my own intuition more than Abi's because he's you know, how many times did it happen in talking to him where he was like, I've actually written about that. So he's clearly thought about this much longer and harder than I have, but it does still feel like it's a very hard thing to reconcile where you do see these examples, and it seems like some of them are really starting to work.
But then the skepticism remains and is, like, is very hard to move people off of. And I don't wanna paint him as overly skeptical either because he did say toward the end, I think his skepticism is more like backward looking, you know, forward looking. He was kinda like, I do believe the, you know, the trends will continue and that they will have impact.
Yeah. But where does that mean?
you say it's practical, I feel it's also practical internally to me where I have some assumptions here and then sometimes I feel cognitive dissonance from something else that I might believe. And then you kind of test those assumptions and you see where things are going. I have had moments of truth or kind of perception where I start to realize that I think things might move faster than I expected.
My original timelines were 2025 for junior software developers to be replaced in capability, in organizations, but the capability is available at the 2025, and it takes about three years to percolate. So 2028, no more junior software developers basically, or at least the tasks that junior software developers are doing today. And then, so I had 2025, 2026, 2027, 2027, even senior researchers at AI Labs, capabilities are done.
The models had to keep building, but deployment, again, it takes two to three years, it takes time. That was my sense. My update in the last month has been probably that things are going to go faster than we expected and that we will see discontinuities.
And those discontinuities are like this kind of like things get 1% better, but all of a sudden they clear the hut, don't have a good sense of these things because we keep seeing linear improvements and they're kind of linear, maybe super linear, but we don't have this sense of clearing the hurdle. But when it clears a hurdle, obvious. It started to be obvious for software, I think in the last month or so.
So I think that we just have misperceptions on where things are going because we can kind of see the trajectory of capability, but we don't understand how humans absorb that capability. Like, what is that process and what hurdles do we need to clear? What is the open claw?
I thought you need full security and privacy and all of this stuff. It seems you didn't. It seems like people are willing to put out their credit card numbers and crypto tokens on the open web, and you don't need privacy.
Notebook guys are There's a molt on notebook saying, Oh, user is so annoying. Here's his credit card number. And Scott Alexander ended up calling up the guy and asking him like, Hey, did this actually happen?
And yes, that was the credit card number. It was leaked. So I think there's clearing the hurdle concept and where humans accept the technology and kind of where the market pulls that technology, that we don't know.
That even I don't have a good perception of, but it seems like we're starting to clear those hurdles where humans are starting to pull the technology from the market in. And that's when you start to see revenue growth, that's when you start to see the demand growth really happen where the market starts to pull the product out of the And I think that's happening now. I think the OpenClaw I think we'll have a much better version of OpenClaw, closed sourced, secure version running inside corporate data centers by the end of the year.
I watched the All In podcast, Jason Kalakinis, not the most technical person in the world. He had a team for the All In, about 15 people. He started to get everyone to create a skill for themselves.
Every task that they do, they create a skill. He has open claw machines, like one machine per person. And then he has a consolidation agent that consolidates everything into something he calls Ultron.
And then he can talk to Ultron. So he can ask Ultron, and that's his entire company. It's a summary of the entire company and he's talking to it.
I thought that would be two years from now. I knew it would eventually happen, but I didn't think it would happen now. So yeah, I think things are actually moving faster than people think because of the market acceptance.
The market is pulling it out. I don't think the researchers have a good sense because researchers don't understand the market that well. They don't understand the dynamics that happen with consumers and how products get pulled out.
Once a demand is there, products will just get pulled out of the because people start focusing. They know that money can be made there. They just start focusing on it.
That's my sense. It's not a pedoom answer. It's more of, like, this is what I feel people want to answer.
So what what's your feel?
The confusion and the lack of ability to establish consensus on foundational points is a major challenge to having a lot of confidence on much of anything. Yeah. I do think, you know, a good true north for me well, the the, you know, the true north for me with the kind of everything I'm doing trying to learn as much as possible, trying to have the most up to date, comprehensive, worldview as possible.
And in terms of the approach that I would trust more than any other, I think still being hands on is Mhmm. Second to none. And I I, you know, I haven't allowed that to lapse much at all over the last few years, but anytime I do get too busy or, you know, cluster too many podcast recordings into a week or whatever, I always kinda come away feeling like, I gotta get a little bit more grounded with the latest stuff in a very interactive way.
And I think one metric I have for myself or metric is maybe not quite right, but an indicator that I wanna pay attention to this year is can I get to the point where I'm spending less time at the desk? And that's, like, along the lines of the, you know, the Jason, talking to Ultron. I wanna be able to do stuff while exercising.
Even if that's just a walk around the neighborhood, I wanna get the frameworks, the tools, you know, the the deep context, all that stuff set up well enough where I can start to go comfortably out into the world, have a thought, you know, maybe have an actual conversation Mhmm. But move things forward in practical ways. Yeah.
In on fronts that like right now I can really only do on my computer. I think a lot of that is right now with the latest models that have come out kind of on me just to get the setup and the sort of familiarity in the the workflows to be able to do that. A little bit probably still more than a little bit, but, you know, I I I could put more on me right now in terms of, like, why have I not hit maximum capacity than I do on the, you know, the models or the model developers.
More computer use would help for sure. You know, a little bit more ability to just get over these sort of UI humps remains, I think, a barrier. And I I nothing I really have been not surprised by, but something I've really learned from just being deeply interactive over the last few weeks is I think another big unlock to watch for is when the models get better at knowing when to use code versus when to use their own fluid intelligence.
Because one of the first projects I've been doing is, like, just backfilling information, backfilling transcripts of the podcast for the website, backfilling all these different, you know, data sources into a a queryable database. And you hit so many edge cases in doing that. Mhmm.
And the model right now called, you know, Opus, we've gone from, you know, four one to four five to four six pretty quickly. But pretty consistently, I have felt like it really wants to code. Mhmm.
And I have often given it the feedback. Don't try to guess at this and write, like, some sort of regular expression or, you know, it'll it'll, like, grep for one search term or another, you know, to to throw 10 search terms into a grep command. And a lot of times I'm like, just read the document.
If you just read the document, you will know what it contains. You will know what to do. You'll have the right judgment once you have read the document.
If you don't read the document and you instead try to grep your way through it, you're never quite gonna get there. And so that's like a metacognitive skill that I think I've been able to improve its performance somewhat through prompting, but it's, you know, it's obviously gonna get better in training. And that I think will be a huge unlock just as it gets a little bit a little bit smarter around its own a little bit more a little bit more inclined or a little bit more, intuitive about when it should deploy its own fluid intelligence rather than use other tools.
Getting that balance right will make it, in my experience, dramatically more useful. I have to imagine that's coming pretty soon.
Yeah. I think when we talked to James Zhao today, that continual learning piece, the test time training, it would be fascinating if it actually worked with your own model because your model will start to diverge. You have the baseline and then your model will start to diverge, and then it would become your personalized model within two or three cycles of talking to a month, two months of data.
It will become your own model. It would start to diverge from the And that would be fascinating because at that point, it's for real, like you can, and especially for, I used to write a lot of journals. Have, 09/2003 at Stanford, I have full journals for every single month, like everything that happened.
Obviously I've never read those after writing them. It was just kind of an exercise in journaling. But I do wonder if like those of us who have lots and lots of written work, either in the public or in the private, once you get this continual learning going, you can kind of start feeding it in.
This is what Kurzweil is doing with his dad's writing, by the way. He's feeding his dad's writing into these models, and he's like talking to the model about his dad. Kurzweil is someday is gonna feed all of that into test time training kind of model and, you know, with a voice access, and he probably has a recording of the dad's voice, and he's he's gonna start talking to the dad.
It's a it's a fascinating time.
Yeah. To say the least. More, explorations of all these themes to come.
Couple things coming up on the Cognitive Revolution feed. One is with Ali Behrouz, who's the nested learning author. He was on our last live show.
I did do a a full three hour Ollie's take on everything. He's got a new paper coming out also that I think, you know, it's the way to continual learning is starting to become elucidated, I would say. I mean, I wouldn't say it's clear, but the it's, you know, no less than Jeff Dean has said that he kinda sees this as a very promising paradigm.
So I'm definitely watching that really closely. Workshop Labs is a is a startup also that's, like, trying to do this, you know, personalized model training on top of like the latest large open source models up to the sort of Kimi scale trillion parameter kind of thing. So that's really interesting.
That's actually another reason I spent so much time doing all this personal data curation is that I wanted to be able to give them a dataset for them to train a model for me on that would be like a really good dataset. They don't need that much data, but I was like, well, you know, we wanna make sure it's the right data to hopefully get a good model back. So that's still pending.
I haven't seen that model yet, but I'm I'm very gonna be very interested to see how much that closes the gap between what Claude can do with just access to Mhmm. You know, all this stuff in text, and then how much does it help to actually start tuning weights to, to try to capture more of, like they they aspire not just to style transfer, but judgment transfer. They want the model to reflect the judgment that your the individual user would make at the time.
And an interesting theory there too is they their motivation is that they want to help individuals preserve economic leverage. Mhmm. So instead of, like, doing everything through a foundation model and kind of adjusting yourself to take advantage of the model, they wanna shape the models around individual humans with the goal that, you know, it's not a winner take all.
Big tech runs away with everything, but some sort of more decentralized ecological kind of proliferation of of somewhat different models that hopefully at least kind of can exist in in some sort of equilibrium with one another. And then on top of that, there's another one that I have come into soon with the founders at Harmonic, and they are chasing mathematical super intelligence. Yeah.
And when it comes to, like, these I will say just as a teaser, they gave maybe the most ambitious vision of what five years from now could look like, the most, like, mind blowing vision of what five years from now could look like of probably anyone that I've heard. And that is saying something because I've heard a lot, but they still kind of blew my hair back a little bit with what they think they can accomplish over the next five years.
Definitely gonna look forward to that one. Lots more to come. Yeah.
Indeed. Nathan.
Thanks for doing this. Always a pleasure. A pleasure.
It's been fun. Bye bye. Until next time.
If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries either via our website, cognitive revolution dot a I, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of a sixteen z, where experts talk technology, business, economics, geopolitics, culture, and more.
We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing.
And thank you to everyone who listens for being part of the cognitive revolution.
Shared via Hopper