🔬ESMFold2: The Bitter Lesson is Coming for Proteins - Alex Rives, BioHub

So ESMC is also approaching programmable biology, but I would say in a very different way. It's approaching it from this kind of world modeling perspective, where the idea is basically you have a predictive model and you're gonna search the world model to find protein molecules that satisfy kind of whatever design criteria that you have. So we've been able to use this to actually now go and design many protein binders.

I think sort of most excitingly, we've been able to use this to actually design antibodies, SCFVs.

head of science at Biohub. Yeah. Would you like to introduce yourself real quick?

Yeah. Yeah. Thank you for having me here.

It's great to be here. I'm head of science at Biohub. I'm a computer scientist, and I work on AI for biology.

And a lot of my work has been on language models for biology. By the time this podcast is released, you will have put out several new exciting interesting models.

you know, since since the summer of twenty eighteen. And so my team, when we were at MetaFair, trained, really the first transformer language model for protein biology. And so I guess, you know, I I've always thought that there would be kind of emergence of biological information as you train a model to predict the next token that evolution creates.

So our team has really explored that idea over a number of different years, and we've really kind of, I think, seen the scaling curve and really seen as we have have increased models by an order of magnitude kind of in each generation that, you know, there's this emergence of new capabilities.

Yeah. I mean, it's a really interesting question, I think, kind of a deep question across AI right now more broadly. And, you know, I I think, you know, what's what's so interesting is AI right now is is such an empirical science, and so we don't have, you know, theory that can always guide us in these things, but we have this really strong empirical evidence of scaling.

The thing that I was motivated by is, you know, if you think about evolution and, you know, you think about the data that we we have around proteins, we have databases that have billions of protein sequences. And those sequences contain patterns. And it had long been known, so this is going back decades kind of before we started working on this with language models, but that there are patterns, the sequences of protein families that come there because of the constraints that evolution is operating under.

So, you can think about like a protein sequence that folds into a three-dimensional structure in space. You can imagine that there are two residues or amino acids that are in this sequence that might be in contact in that folded structure. And so evolution isn't free to choose those independently from each other.

If it makes a choice at one position, it kind of has to make another choice that's going to be compatible at the next position. So going back all the way to the beginning of gene sequencing, when people first began to be able to look at this and kind of look at different related the same protein and related organisms, you could start to see these kind of patterns that are reflecting the fundamental underlying biology. So the idea behind ESM, kind of the thinking behind ESM was, okay, what if you were to apply this principle across all of evolution, across kind of the vast diversity of proteins that have been generated across all of life and basically have a language model kind of predict the amino acids that evolution will choose to place in proteins across all of those biological contexts.

So you can think that there's just this kind of incredible amount of information in that total picture about the underlying biology of proteins. And so that was really the idea that sparked this, as a model is having to predict the next token, and actually we train these models with mass language modeling, so they're predicting kind of tokens that are masked out of various parts of the sequence, that it would have to learn something about those kind of underlying constraints that are shaping which tokens evolution can choose.

Though they were they were called ESM from the start. Okay. They're called ESM.

Yeah. We've sort of various branches of of the different models. Yeah.

Yeah. So so this one I would say is is kind of a a fourth generation model. It's actually a model that we trained a little over a year ago.

Now that we're at Biohub, we're we're open sourcing this this model fully under MIT license for the first time. So we're really excited to do that. But kind of the big thing that is new here is that we've really kind of built a world model of protein biology.

So the foundation of that is ESMC. But, you know, using the representations of ESMC, we've kind of now built a structure prediction model. And this is the next generation ESM fold model.

And then we've also used the techniques of mechanistic interpretability and sparse coding to really start to look deeply into the representation space of the language model and kind of be able to pull out the underlying features that the model actually uses to represent protein biology. So bringing all of this together, we're able to really make predictions for protein structure, predictions about kind of the underlying features that proteins are made out of that allows us to build linkages across evolution. We're able to take this model and invert it to design proteins.

And we've used this to kind of create a comprehensive picture of protein biology. So we put together kind of all the world's largest protein sequence databases. And so that kind of amounts to 6,800,000,000 non redundant proteins.

And then we've we've resolved predicted structures for 1,100,000,000 of those. And and we've also computed features across all of those so that we can make these linkages basically all across evolution and protein biology.

Is that 1.1. 1.

1? Yeah. So what about the others?

Well, so so basically what we did is we took that database clustered it at 70% sequence identity. So it's it's really resolving structures for everything in the sense that for each cluster, we kind of have a cluster center. We're predicting the structure there, and then we can expect that the other proteins are gonna have a similar template structure.

I see. There'll be these small variations, but they have the same fold.

That are that are kind of covering the 6,800,000,000.

and that's the right resolution for this model? Well, we've chosen them so that they really cover that entire space. So I think what I can say about this database is it's really the most comprehensive picture of protein structure and function that's been created.

It's adding, you know, hundreds of millions of structures to our knowledge of kind of protein, the diversity of protein structure. And it's also creating this feature space that allows us to find these linkages between proteins across evolution.

across all the different layers of the ESMC model family. So there's actually three models in that family. There's a 300,000,000 parameter model, a 600,000,000 parameter model, and a 6,000,000,000 parameter model.

And then we've looked really we've done kind of a very deep analysis of the feature space of that 6,000,000,000 parameter model, which is really the state of the art protein language model. And so what we find what's really interesting is there's there's kind of this, this hierarchy of features that emerges. What's really interesting about it is it really kind of corresponds to the reductive picture of biology that has been developed over many decades, a century of biological experiments.

But what's so cool is this is emerging without any prior knowledge. It's been learned by the language model. So the interesting thing about SAEs, right, is they're really just revealing the intrinsic structure of the representation space.

So this model's been trained on protein sequences. It's been trained just to predict the amino acids that evolution will choose. Then somehow this is leading to the emergence of this kind of very ordered feature space that has this hierarchical structure where you can really see everything from the basic biochemical properties and kind of the basic structural building blocks of proteins to these very large kind of functional themes, these kind of abstract concepts that, you know, connect to to how kind of the human picture of of protein function.

I mean, it's a really interesting question. Right? And I think I think I can speculate about it.

I don't think we kind of completely understand this. Right? But let me give a concrete example.

So, you know, the nucleophilic elbow is this kind of, like, core kind of functional motif that people have thought that, you know, maybe this actually has emerged independently in evolution, different times in different protein families. But it has this very clear structural motif that you can kind of see in crystal structure. You know, what we found basically is that the model has a kind of a single feature for this nucleophilic elbow and is activating across these, like, very evolutionarily diverse families, you know, really completely different structural topologies, proteins that probably evolve, like, entirely independently from each other.

But, you know, the model is kind of using this one feature to represent that. So why does it do that? I mean, I I think it's a really interesting question.

I mean, I think one answer is sort of the idea of of compression and the idea that, you know, the model needs to have some kind of underlying latent variables that it develops to help solve this this kind of sequence prediction task. Because the nucleophilic elbow, you know, it's going to be a function of what's so interesting, right, is the choice of any amino acid is kind of like completely entangled with the choice of all the other amino acids in the sequence. So this is a very complex task to try to predict what amino acids should be where in a protein.

But to really do this well, the model would start to have to have these kind of hidden variables that are representing the biology that allow it to look at a protein and say, okay, what amino acids should be there in all these different contexts? So, you know, that's sort of, I think, the intuition. I mean, I would draw the parallel to language modeling.

Right? And so I guess I was I was like very influenced by a paper by Zeilich Harris. It was called Distributional Structure from 1954.

And I think a paper that influenced a lot of people in the language modelling field as well. But I think it has so it focuses on language and it really articulates this idea that the set of contexts in which a word appears are determined by the meaning of that word. And so what Zealy Harris kind of imagined is that, you know, it would be, as you looked at the statistical patterns of, you know, what words appear in what context sets, you would be able to derive the meaning of language.

You know, you would have this kind of statistical structure that would mirror the underlying meaning of language. For me, least, that's one of the most convincing explanations for why, you know, language model that's trained on the text of the Internet is going to learn something about meaning. It's gonna learn something deeper and more fundamental.

And so I think, you know, you can think about the same thing in biology where the contexts in which an amino acid can occur are really determined by the structure, the function of the protein, its biological roles, these, I mean, very complex phenomena, both the intrinsic biology of the protein and its relation to all of the other proteins and the function and evolution. And so but but those are what determine the context sets. And so you would imagine that then those statistical patterns in the use of amino acids, they directly reflect those underlying hidden variables.

And so the model is gonna learn something about those hidden variables.

and what the developments there were. Yeah. Well, I'll I'll admit that I am I am bitter less in film.

I am scaling film. And so I do think that I mean, you know, just just kind of increasing the data and increasing the parameters and having that compression is is going to just lead to more powerful models. But, you know, it is also true, and I think you're you're absolutely right, the structure the underlying kind of structure and distribution of the data is really critical.

And so, you know, some data sets will be far more valuable for kind of learning these these general principles than than others. But I think it goes against a lot of biological intuitions about collecting data, I guess, is what I'd say. Because normally, when you think about what data do you want, you're trying to answer a very specific scientific hypothesis.

You want a very well controlled experiment. You really want multiple replicas. It's something that's very focused, is the way that I would put it.

So I think the change in the way of thinking is to think, okay, what you really want if you want to learn a general representation of proteins is you want to see amino acids in as many evolutionary contexts as possible. That's really what you want. That's really kind of how I think about data.

And I think if you what changed between ESM two, which was kind of the previous generation model, and ESMC, which is this new generation model? Because they're both at the approximately the same scale.

ESM two got a lot of compute, but ESMC got even more compute. But it's not just the compute. The data was was really the critical thing here, actually.

So when we trained ESM-two, we observed two things. The first was that as we increase the number of parameters and compute, we saw improvement. So we had a model kind of at the billion parameter scale, we had a model at the 10,000,000,000 parameter scale.

And, you know, the larger scale model is better than the smaller scale model. But if you kind of look at a plot of parameter scale, you know, sort of a log plot of parameter scale versus capability. And so for capability, you know, we're looking at kind of the representational fidelity.

How well does it capture protein structure? You can see that there's there's kind of diminishing returns in ESM2. ESM2 is trained on UniRef.

And for ESMC, we added metagenomics. So we added billions more sequences to the training data.

I'd say, sort of the gold standard dataset of sequence biology. It's kind of taking sequences from across a wide variety of different sequencing resources. It's clustering them to kind of remove some of this redundancy that you were mentioning.

And so it kind of creates a definitive coverage of protein biology. What has happened in parallel to the classical gene sequencing is this idea of metagenomic sequencing, people go out into, you know, all kinds of different biomes and environments and collect samples from the world, which kind of sequence the natural diversity that's present there. So, you know, proteins from a hydrothermal vent or proteins from a a frigid environment near the South Pole, you know, or or the deep ocean or, you know, soil or the human gut, you know, all all kinds of different environments.

So this is a very different way of collecting data.

We're not assigning them to an organism. We're not assigning them to, like, a larger context. We're just saying, this is probably a protein.

Let's train on it. That is right. Yeah.

And you you don't even get the full genomes. You just get these kind of contigs that often are broken and have even partial protein. So the data is really noisy.

I think so. I I I'm not entirely sure, but there are a lot of proteins. And I I think we've we've barely scratched the surface of measuring Earth's biodiversity.

So there are core proteins that are conserved across all of life. So we we, I think, know that. Right?

But but as you go into these different environments, they're just, you know, new new genes and no new proteins constantly being created by evolution.

microorganisms. Microorganisms.

That's right. Yeah. And just four billion years of of life running experiments in parallel all across the earth and all kinds of different ecological niches.

And we just we see the outcome of all of that.

not even nearly as much diversity as there will be at the microscopic scale because you have this incredible combinatorial effect. Yeah. I mean, there's just, I think, tremendous, tremendous diversity there.

So so kinda going back then to EFIC. No. It's it's great.

Right? And I think it's really I mean, we can also talk about data and building models of the cell and kind of really going from the molecular level to Yeah. You know, to to to higher levels of biological complexity, but but to to to complete the the the description of of ESMC.

So so that that's really the big change was kind of adding these metagenomic sequences. And then, you know, what we saw basically is there are no longer diminishing returns to scale. So that's really saying that ESM-two was kind of data limited rather than compute limited.

For ESMC, there's a really beautiful scaling law that we can plot where we can look at we can train models, you know, to make the larger models. We basically train models at at the smaller scale, and we can we can really look at the best representational fidelity that they can achieve for a given compute budget and just draw a line of extrapolation out that that kind of beautifully predicts what the larger scale models will will be able to achieve in their representational fidelity. So there's there's this really beautiful scaling.

And, you know, the the really the only I mean, there are some changes to to TSMC just to make it a more efficient model for for training. But, you know, I I think the data is really, you know, the really big thing there that's that's driving that. So it still is basically just a standard vanilla churn transformer, a few tricks.

Everyone has a few tricks at this point. Yeah. So that's significant language model and just a lot of data.

So, I mean, this is very much in contrast to something like AlphaFold, right, where you have a lot of inductive bias in built into the model in order to be able to predict protein structure. That's right. And the idea here here is, you know, really, can we just learn the right structure?

You know? Don't don't give any priors. Just allow, you know, allow machine learning to figure out what that structure is.

So you also had your own detour in the priors with ESM three. Right?

that actually we don't need priors anymore? Is that like kind of a key insight or do you still think there's room for priors? I think we need both.

I mean, I think there's just you know, there there's a place for both of them. So, you know, the the goal for ESM three was to really make biology programmable. And so we're trying to think, okay.

Like, what is the programming language? Right? How are you going to be able to allow biologists to prompt a model and design structure and design function and all these things?

And so we really thought it needed the right tracks. But I would say that ESM three was, like, very consistent with the philosophy of ESM because what we did is we basically predicted structures for this vast array of evolutionarily diverse proteins, we're using that as the training data. So the model's now just it's learning from sequence patterns, learning from structural patterns, learning from functional patterns.

of a different model. Yeah. Yeah.

I mean, I I think it's a really an interesting question kind of to what degree can you interconvert these models. I don't think kind of that's fully understood yet, but I I think that's it's a very kind of promising direction to think about Yeah. Doing that and what are the right ways to do that.

So ESMC is all is also approaching programmable biology, but I would say in a very different way. It's approaching it from this kind of world modeling perspective, where the idea is basically you have a predictive model and, you know, you're search the world model to find protein molecules that that satisfy kind of whatever design criteria that you have. So we've been able to use this to actually now go and design many protein binders.

Mhmm. But I think sort of most excitingly, we've been able to use this to actually design antibodies, SCFVs, and we're seeing really, I think, exciting Mhmm. Success rates in in a small number of trials now.

Yeah.

Yeah. Yeah. So an SCFV is basically it's a it's a single chain antibody.

So it's it's a kind of therapeutic modality that basically has so an antibody has a heavy chain and a light chain. Mhmm. And then it basically has a pair of of, you know, one heavy chain, one light chain, another heavy chain, and one light chain that come together to recognize a target.

So there's different variations of these kind of modalities that are used therapeutically. And so what's interesting about the SCFE is it has one heavy chain and one light chain. So it is able to kind of form these very complex binding interfaces where, you know, you can kind of have two different subunits coming together to engage a target.

These are kind of important therapeutic modality. Something like, I think, a quarter of of new drugs or antibodies. So it's really, I think, you know, one one of the one of the critical modalities for for medicine.

And, basically, what we're able to see is that, you know, you can search ESMC and you can actually find antibodies that are reaching the level of affinity there, I should say, or really at the level of affinity that is needed for therapeutic function and activity.

SCFVs and reformat them as antibodies. So I think that would be kind of the quickest approach to do that. We've not tried full IgGs.

I I don't see any reason why that wouldn't work. Actually, something we haven't yet. You know, we've we're basically kind of releasing this now because we feel like it's it's kind of reached a point where, you know, we're we're seeing, I think, a really significant step above kind of what's been possible in the past.

And so we just we wanted to get it out there. But Yeah. You know, I I think there's a lot more progress that's possible.

So we're you know, we have a lot of collaborations to kind of look at some of the other Yeah. Applications here. You know, the thing about it, right, is it it's a general model.

So I I think to me, that's the most exciting thing about it is just, you know, a general model for protein sequence structure and function.

So I think I think that's one of the things that's really cool. That's one of one of the theses that that we had is, you know, antibodies are not gonna benefit from evolutionary information probably in the same way that kind of predicting the structural topology of a of a molecule will. So, you know, I think I think you kind of see that now where the the representation space is containing something that's really interesting about antibodies here.

What were some interesting things that weren't just validating biology, but there's a pattern that was unexpected? Did you find anything like that? It's complicated.

So because we have to we have to now actually go and validate some of these things. Right? So Yeah.

I think what we saw are, like, interesting connections. Right? So, you know, what we can see, for example, is that kind of distantly evolutionarily related gene editing systems clustered together in this space in ways that are consistent with and kind of reflect our knowledge of the origin of those gene editing systems.

So that's really exciting. But but kind of the thing is, right, there's a number of proteins that are in that map that are kind of brought together in different ways where we just we don't know what they are right now. We don't know what they do.

So one hypothesis there is, well, these are kind of novel gene editing systems. I think in this atlas, you know, there's going to be some really interesting basis for scientific discovery there. If you think about kind of how people go out and look for new gene editing systems, for example, they're typically mining the large genetic sequence databases, and they're looking for kind of different sequence patterns or structural patterns that are linked to that.

Actually, the first version of the ESM Atlas was was used by Feng Zhang's group to find a new gene editing system. So I think there's just a lot of biology out there that we don't understand that's waiting to be discovered and kind of being able to connect the dots between proteins so that we can go from what it is that we know today to kind of make those inferences about the unknown. So that's what I'm excited about.

And I think, you know, there are proteins for so many applications that nature has probably invented. You know, you think about the thermostable polymerase, which enables PCR, which came from bacteria living in a thermal hot pool. You know, you you have there may be the solution to to climate change, you know, somewhere in in protein biology.

There are probably all kinds of building blocks for for completely green chemistry infrastructure out there. There's probably new medicines and therapies. You know?

But the the question is, how do you find those? And so I think, you know, kind of being able to connect the dots is is is really one way to to start to be able to open up that space of protein biology to discovery. I'm curious.

beginning to a virtual cell, like, create that map? So I I think something like that would be really valuable. I think, you know, fast so the other thing about ESM TOLD two is a really fast model because it doesn't, you know, require the multiple sequence alignment.

So, you know, you can do inference kind of, you know, directly from the sequence. It takes seconds. You know, you can get an atomic resolution prediction.

So, yeah, that's I think one really interesting application. At at Biohub, I mean, the other thing that we're thinking about is can we actually experimentally resolve this? Mhmm.

And so one of the things that we are we are building is cryo electron tomography, and and we're really building systems that can greatly increase the contrast when you're looking at, you know, at the cell at the atomic level. And so I I think one thing that I I hope to see is actually structurally empirically resolved interactome at some point in the future. And I think there some some pretty big technical hurdles and technologies that have to be developed to to overcome that.

But I think that's something that that's going to be possible. So we we can use computational methods to start to get the proxy of that, and I think, you know, that's gonna be really powerful. But I I think a lot of the future of structured prediction is gonna turn into structured determination, actually.

fundamental principles for the next era of of biology. I think I think it's yeah. It's such an interesting time right now because I think we're really we're at the beginning of a new scientific paradigm.

It's really just the beginning of it. And so what is, you know, what is defining in that paradigm? Right?

And so I think there are there are a few principles. You know, one is scaled data generation. I think that's gonna be really critical.

The second is computational, you know, predictive digital representations of biology. And we can kind of talk about, you you can think of ESM as being, you know, a first generation, AlphaFold as being a first generation of those kinds of approaches. And so you can kind of start to think about what does that look like as we can model more and more biological complexity in that way.

And then you have the principle of feedback. And you have the principle of, you know, we have intelligence now that's scalable and so can be applied to every unit of a biological problem. What would it mean for all of that to come together?

So I think we're gonna we're gonna have increasingly capable and accurate digital representations of molecules, genomes, cells, ultimately physiology, that's where you want to get we're gonna have to have to go up that that complexity scale, the levels of biological complexity that requires traversing a a data barrier. There's, I think, data that does not exist that needs to be generated to achieve that level of predictive fidelity. And then we're going to have reasoning.

And I think, you know, what that will mean is that we can reason over thousands, millions, hundreds of millions of scientific hypotheses in parallel digitally using predictive oracles, which can actually predict the outcome of an experiment. So the scale that we can ask questions and the kinds of questions that we can ask will just fundamentally change through that. Feedback is gonna be critical.

You know, the models are gonna need to there's gonna be sort of a scaling dimension of this, which is which is, you know, building the data to have those accurate representations, and then a feedback dimension where the models can learn from biology, can reason digitally, can reduce that to a small number of experimental hypotheses, examine the outcome of each of those experiments, update their understanding, and build knowledge in that way. So I think that's what's what it's gonna look like. And and we kind of have to build each of those components.

What Biohub is really trying to do is to kind of bring together the experimental and the technology layer that will actually allow us to have these AI models interact with the biology and do experiments. And I think it's, you know, it's Amdahl's Law. You know, we see incredible, incredible advance in areas where we can get feedback computationally, so in enclosed domains.

But, of course, you know, experimental biology is is completely open ended. And so the feedback principle there is is going to be very different. But, you know, something there's gonna be something like RLVR, you know, with with experiments where we can, you know, have models that are that are just really building knowledge and learning from that knowledge and being able to develop more and more accurate representations.

we're building a scientific institution for this new paradigm. And, you know, to do that, you know, it's it's it's an institution that's going to be powered by frontier experimental biology Mhmm. Frontier technology for for measurement, for observation, and it's going to be powered by frontier artificial intelligence.

And this is all open source. Right? It's a philanthropy.

So so our goal is to accelerate science. Our mission is to cure or prevent disease. And so to do that, you know, our belief is that there's a fundamental gap in our understanding, and we need to accelerate science to traverse that gap.

And so we're really thinking about every layer of biological understanding that goes from the most basic, you know, level, like the, you know, the atoms of a protein in a cell all the way to systems of cells in physiology and disease. And how can we create models that can capture that complexity, can allow us to understand that complexity? And I think, you know, if you think, what the cure to disease look like?

Right? It's not a pill, right? It's not a medicine in the conventional sense.

You know, it's going to have to be a system that is capable of modeling and understanding, you know, the underlying physiology of disease in a way that's differentiated for every single human being, for every single different genome. And it's going to have to be able to link events all the way at the molecular scale to the manifestation of disease in in physiology. So it's it's an incredibly complex, incredibly hard problem.

And for us, we're trying to ladder up those layers of complexity, and we're trying to build kind of the foundational tools that scientists can use to answer the fundamental questions there. And so we're creating atomic level imaging. We're creating light sheet microscopy that allows us to observe how all the cells move and develop in a developing organism.

We're creating spatially and temporally resolved maps of inflammation. We're creating cellular programming and immune cell reprogramming to be able to actually design completely programmable therapies. And, you know, we're creating these digital representations at each of these layers so that we can accelerate the science, simulate what's happening, you know, make biological matter and make proteins and cells and genomes programmable.

And I think all of that has to come together. And I think if you have the focus and you build the biology and the computational layers together so that they're tightly integrated, you know, that's how we're gonna make the fastest progress. For the last ten years, I think we've been one of the one of the big champions of open science.

You know, we're we're an organization that does we both fund and we and and we build. And in our funding, we've we've always supported open science. And in our building, you know, we've always done open science.

So that's that's something that's gonna continue. It's it's just really fundamental. We're not a drug development company.

We're not trying to generate therapies. We're trying to trying to build the technology that moves science forward.

a general improvement in our ability to tackle human disease? Yeah. So I I think the next level of complexity that we have to address is is the complexity of the cell.

And, I mean, this is going to be tremendously hard. Billions of proteins.

Well, I I think it's a it's a worthy challenge. Yeah. But but it yeah.

I mean, it requires technology that doesn't exist today. It requires new, you know, modeling approaches and and probably architectures and ideas and machine learning that probably don't don't yet exist. So there's, I think, deep and fundamental problems to solve.

But again, I think, you know, you you take it step by step. And so, you know, we kind of started the molecular layer, and we know that that is really fundamental, and we can begin to link that to, you know, observables in cellular biology.

of proteins powerful and useful is that they generalize. They're able to make predictions for proteins that are entirely unlike the proteins in their training data. You know, they're able to generalize so that you can design, you know, fundamentally new folds, new binding interfaces, new structures.

So there's this degree of what we call generalization or generality. In short, they can predict the outcome of an experiment that we haven't already made, that they haven't already been trained on. So for digital representations to be valuable, they've got to be able to be used to answer a new question.

So I think that's the critical thing. So I don't we're not there with cells. I think with, you know, kind of the current generation of models that are being called virtual cells, they are good representations of the underlying data.

But, you know, they have a very limited ability to predict what will happen when you make a novel intervention in a novel unobserved context. But to be able to answer the fundamental scientific questions about cellular biology, we need a model that can do that. So so kind of, you know, our thinking about this starts starts with that idea.

So what's it gonna take to to get there?

I think that's far beyond reach of of current computational technology. I mean, kind of even simulating the the physics of the folding of a of a single, you know, protein molecule.

So there's kind of this dual view of biology, this dual complementary view of biology. One view is kind of that kind of first principles reduction, you know, where kind of all of biology is explainable in, you know, more basic terms, in basic physical, chemical, biochemical terms. And I think historically, there's a long research line of research that's really sought to understand biological phenomena and simulate biological phenomena in that way.

And I mean, historically, you know, the field had believed that the solution to the protein folding problem or the protein structure prediction problem would come from, you know, this kind of first principle simulation. It really kind of came out of nowhere that, you know, this could be solved using essentially pattern recognition or, you know, this this type of machine learning approach. So I think I think historically, has been productive to understand biology through information theory, through information.

And, you know, you can think about the cell as as a computer, as an information processing machine. It's in informational terms, there are, you know, these very basic principles that link the information encoded in the genome to the genes that are transcribed to the phenotypes of the cell that will result. And so, you know, if we could model and understand the cell at the level of its underlying programs, you know, that sort of gives, I think, the right abstraction.

What do I mean by the right abstraction? Well, I think I mean the abstraction that is possible today because we're in the era of information theory at scale. You know, we're we're in the Claude Shannon, you know, had this idea of of kind of the the ideal predictor of the next character.

And he he had this really beautiful paper where he tried to compute the entropy of the English language and imagine basically, you know, taking an infinite context and then, you know, what is the entropy of the next character? And at that time, I mean, it was unimaginable to I I'd say it create it took a great leap of imagination to imagine that ideal predictor. But today, we're get closer and closer to being able to build that, and we can do that for text.

And so, you know, what would that predictor be for biology? And that's kind of the idea of ESM. It would learn kind of the underlying structure of all biological phenomena.

So if you think about that from the standpoint of the cell, if we can collect enough outputs of cellular biology that we can observe to reveal the underlying programs, patterns, and structure, you know, then we could create kind of the information theoretic description of the cell. And I think that would be sufficient to understanding disease.

of, you know, decades. I mean, for for protein structure, half a century of of of work to experimentally determine the structure of proteins and, you know, and and the the, you know, the effort across, you know, the scientific world to, you know, sequence genomes and metagenomes. And so that's created, you know, this this dataset that, you know, you you can really, you know, train a scale on and really learn these these deeper principles.

A little bit of context. We announced a few weeks ago, the Virtual Biology Initiative. We basically said, you know, we're gonna invest 400,000,000 internally in data creation and development of technology to scale data generation to be able to increase the the number of modalities that we can measure simultaneously.

We're we also announced that we're going to commit a 100,000,000 to catalyzing efforts outside of Biohub to generate data. So, we think that's a fraction of what's actually needed to do this. But the hope basically is that by making this initial commitment kind of giving starting funds to some of the groups that are really thinking about this, you know, working to build different core areas of of the data that's going to be needed, that, you know, that that's gonna be a catalyst that's gonna galvanize other other groups to come in and and contribute to this.

So that's what we we really hope to see. You know, the idea is that, you know, this is this is broad a broad based effort. So it's it's not just us.

So, you know, I can I can say kind of what my perspective is on what data needs to be generated here or what can be generated? But, you know, we also want to approach this really collaboratively with the scientific community. So part of this is also kind of hearing from scientists what they want.

So from my view, you know, there are a few key principles here. The first is speed. Okay.

So, you know, it took decades to build the data for proteins, and we can't wait decades. You know, this is we need to figure out how to do this in a couple of years. And you look at the rate that that general AI is developing, and it's just, you know, the limitation in biology, we're gonna be fundamentally limited by experimental science and data.

And so we really need to, you know, work to address that gap as quickly as possible. So I think that's one key thing is is looking at what are the technologies that we can scale up today to begin to give this picture of the information architecture of the cell. So there's speed.

And then there's also the idea of generalization. So kind of going back to what I was saying before, you know, we want models that can serve as oracles for the biology. They can predict an experiment that you haven't done.

And so how are we going to be able to do that? We're gonna need to look at a multitude of different interventions in a multitude of different contexts. And so it's it's kind of similar to the principle of training a language model on the Internet or training a protein language model across all of evolutionary diversity.

What does that look like for cellular biology? And so we have to scale interventional biology. So that looks like things like perturbation biology, perturb seq measurements that where we can look at combined transcription, imaging, other layers of the cellular information hierarchy.

There are a number of groups. Our teams are working on this. There are number of groups across the scientific world that are working on problems like this that are ready, I think, to to scale.

The second is spatial biology. I think that's gonna be really important. And so that's gonna help us to really understand the cell in context.

And I think, you know, understanding the cell in isolation is is really not what we need. It's not the goal. Right?

The cell is part of an incredibly complex system in the body. And, you know, to be able to understand disease, we have to understand how cells interact, the systems that they form, the circuits that they form. So we need to see that.

So spatial biology, I think, is undergoing rapid progress and, you know, is is an area that's really ready to scale up. That's kind of what can scale now, I think. And Biohub has actually, over the last ten years, really, I think, made kind of pioneering funding commitments in those those areas.

And so we've we've we've funded efforts like the human cell atlas, and we've built tabula sapiens, which looked at built built large cell atlases. And we've built built cell by gene, which is kind of a database of single cell transcriptomics. And so we're, you know, we're really looking to kind of build on that.

And, you know, I don't know how many cells there are in kind of the largest efforts. We're probably around like a billion cells or something like that today. So that we've got to go, you know, multiple orders of magnitude from that.

So, you know, that involves scaling the technologies that we have now, but it also involves, you know, a new the next generation of technology. So we're also funding and supporting efforts in that area. And there, you know, we really wanna look more at cross modality.

You know? Can you simultaneously see the phenotype, observe the transcriptional layer, understand what's happening proteomically, link that to the genome? You know, we'd like to and the epigenetic state, know, we'd like to be able to see all of that.

Yeah. So, I mean, I I think it was basically the same things that I'm saying. Scaling what we have now, kind of being able to expand the number of interventions that we can look at, expand the number of parameters that we can measure.

So really kind of more and more multidimensional measurements and know, drive down the cost and all of that. So better gene sequencing, kind of better ways of encapsulating cells and being able to measure what's happening, not just the transcriptome, but but other layers simultaneously.

versus how much do you spend on actually scaling it? You know, where do you went out there? Yeah.

We we have to do both of those things. Right? Because I think with current technology, you know, we can definitely kinda get get data 10 x to a 100 x where it is today with, like, relatively reasonable investments, you know, but but then to get another 10 x or more there, that's gonna require require a lot more technology development.

But the other the other really big principle is is going to be feedback. And so I think that's gonna be really critical. And I think you can see that as a layer of of technology development that's that's that's gonna need to occur.

And I think there's a lot of great things happening right now, kind of automation, flexible robotics that's gonna accelerate Mhmm. Where where that can go. And experimental design as well.

Yeah.

I mean, it's it's a hard question to answer is there's just so many bottlenecks. I mean, the one that, you know, that I always think about is compute, but I think that's a pretty obvious one. It's the bottleneck for for all of AI in in many ways right now.

But, you know, especially because we're training these large scale models, you know, we're always focused on compute. And I think we're kind of limited both by the data and compute. I think we're in a, you know, we're in a position where I think we have, you know, incredible compute resources for a team working in biology.

But I think, like like all teams working in AI right now, really, the limit is is just how much compute power.

like, be way better? I mean, it would definitely be way better. We also need to scale data.

So both of those things would have to happen in tandem. Have you basically exhausted what's available right now for I don't think so. No.

Okay. No. I don't think so.

Okay. The large datasets out there or you could Well, relatively, I mean, parameters. You know?

So we trained ESM, see up to 6,000,000,000 parameters. Oh, but I'm saying in terms of data available, like, have you exhausted most of what's publicly available in terms of, like No. No.

Not not yet. And, you know, the Atlas that we just built actually has more sequences and structures than ASMC was trained off.

Mhmm. So there's there's definitely probably order a 100,000,000,000 sequences.

there's, I think, a tremendous amount that you can learn from small genetic variations, right? Because these are really revealing of, you know, kind of the very basic determinants of protein structure and function at a very fine level. So I think that, you know, as we think about protein space, having a vast diversity of sequences across a wide range of protein families is really critical for the emergence of this kind of structure prediction capability.

Because I think kind of large diversity is what trains the model to understand to develop a representation of structure. But I actually think that to develop a representation of function, it's these very small variations that are important. And so I I do think that there is probably a lot more.

You know? It's it's like the models haven't yet been trained at that level of kind of just, like, really deep understanding of these very small but critical patterns in in sequence. I mean, single a single mutation is enough to destroy the the function of a protein.

it's an empirical question. Right? Yeah.

It's truly an empirical question. And so we we just we just don't know. You know?

I mean, with ESM two, we weren't sure because there were some diminishing returns with the SMC. You know? Now now there aren't.

Right? So you can kind of look at look at that, extrapolate from from the scaling law there, and, you know, if there is enough data to train that next next model.

Well, we're we just announced or or I should say we are we are going at the time that this this podcast comes out, we will have announced ESMC and this this world model for protein biology. It's gonna be open source. It's it's gonna be MIT licensed, and we want people to use it.

You know? We want we want this to be a tool that can unlock science. We're excited to collaborate.

We have a team that's that that works on that, and we want to hear from people and understand, you know, what what we can build that can help to accelerate their science.