🔬The BioAI Phase Shift - Matthew McPartlon & Neil Patil, Chai Discovery

what you do at Chai? Yeah. No.

Thank you very much for having us. We're super excited to talk about CHI today. I'm Matt McPartland.

I'm one of the cofounders of CHI. My background is in, like, AI biology related stuff during my PhD. I actually started my PhD in, like, theoretical computer science and then transitioned to this later.

Yeah. I I've been doing this stuff now for, like, about eight years, and I kinda came into the field at an interesting time where protein structure prediction was, like, just starting to see signs of life. So this is, like, AlphaFold one days and was in the field during AlphaFold two and, like, got to see a lot of the interesting developments at that time.

So, yeah, I I'd always been pretty interested in, like, applying this stuff in the real world, and Chai was just a perfect opportunity to do that. And I'm Neil Patel. I help lead a platform and product here at Chai.

So a lot of the stuff around infrastructure to train models, serve them, and then the productization piece, you know, the design suite that lets you use the models.

Yeah. So we're like, we've been very fortunate to partner first with Eli Lilly and then with Pfizer, Novartis, and Argenx. Yeah.

I think it's been, like, a really interesting ride, and I think our business model is also very compelling to a lot of people. Like, we really like to we care about the partners succeeding. Like, Chai as a company really depends on how the partners succeed.

companies? Why are why are they compelled to buy from you? The thesis of Chai has always been to, like, be the software and modeling layer, which was, I think, like, very controversial at the time.

Like, everyone, yeah. This this play is definitely Only two years ago, and it's already, like, a completely different world. Yes.

Yeah. It's it's pretty crazy. Like, the like, people tried this play for a while.

And I think, like, the models just really weren't there yet. Mhmm. And even, like, for us, we were taking a risk in the very beginning.

Like, we were kinda banking on the models getting there. And, like, I had seen early signs of life in my work and our CEO, Josh. Like, he he was on the original ESM papers on that team at Meta, and he was seeing, like, pretty early signs of life that, like, you know, there might be scaling laws here.

They like, think I we'll actually be able to start, like, designing things. Structured prediction is getting really good. Like, one one, like, crazy thought is, like, we didn't have a multimer structure prediction model until, like, 2021.

That was five years ago when we could, like, start with deep learning to, like, actually predict the shape of two proteins at once. Like, it was a Fold one was like and Fold two was like this huge breakthrough, but then, like, Fold two Multimer came out, like, a year later. So, like, you really kinda needed that to unlock design in the first place anyway.

Like, we weren't even trying to predict multiple proteins at once. And then really, like, around that time, inverse folding kinda started working, and it was like, oh, protein MTNN. This actually works in the lab.

Like, credit to the Baker Lab for doing all this really excellent lab validation on all of their models. But I think, like, we're starting to see them do interesting things and, like, actually work on, like, real world experiments. And now is probably the time to start betting on this.

I think, like, before then, maybe you could take, like, some experimental data from a campaign on, like, this one target that you had and you care about, And you you might be able to, like, make some progress on that and, like, keep hill climbing in this, like, one very specific case. General models weren't really a thing back then. So I think, like, yeah, we took that bet pretty seriously, and, like, we we decided to just, like, push as hard as possible and to really, like, shoot for generality in our approach.

And then when CHI two came out, our second paper after CHI one, we kinda, like, show the world, like, this is actually possible, and it's possible at scale. We didn't show this for, like, one or two targets. Like, it kinda works.

Like, we were like, let's just go all in. I think Josh likes to say we set a bold company wide challenge to design antibodies to 50 targets. And it actually, like, we saw some signs of life.

We're like, alright. Let's, like, let's do this with real statistics and see if this actually works. It's an interesting story of how we chose these targets.

So we were like, alright. What targets are we gonna choose? We should choose, like, some interesting targets, whatever.

And at that point, we were, like, kinda ramping up with CROs and figuring out, like, what what does our wet lab process look like. And we decided after after trying some stuff with, like, mini proteins, whatever, we're like, here are the interesting targets. This is what we should look at.

And, like, half the time, the targets just, like, kind of didn't work. We were still learning. Whatever.

And we're like, alright. Maybe we should just go with, like, targets that the CROs have actually validated. So let's get the CRO catalog, see what they've already worked, and restrict that to, like, an interesting set.

So from that, we chose 50 targets, designed antibodies against them, got hits to half. And at that point, I think pharma started to realize, like, okay. Are actually signs of life here, and this this might actually work in some of our programs.

where, like, your target this protein that you're trying to bind to might be some, like, disease protein. That's kind of, like, your lock, and then you wanna design this key that fits into it and, like, in in our case, just, like, sticks there. The interesting thing with antibodies is, like, these, like, really flexible general proteins.

Like, in a lot of ways, they're very general. In a lot of ways, they're actually, like, pretty uniform. But at least, like, how they bind to a target is very general.

So, like, you have a lot of optionality in how you design this kind of binding interface. The structure prediction problem for antibodies, like, predict how this antibody actually binds to the target, how it how the key fits into the lock. That's been a notoriously difficult problem.

The nice thing is like, so we we've made a lot of progress on structure prediction. Kind of the field as a whole has come a long way along, like, in in getting structure prediction to where it is. But in the design setting, you can be a lot more selective about the types of designs you wanna make and the types of structures you actually wanna focus on.

And in some cases, it might actually be even easier to design a protein binder that is an antibody than to actually predict how it might bind that target in general. So, like, it's kind of like if you if you have the freedom to choose, you can kind of just pick the easy cases, if that makes sense.

This is coming from from a nonbiologist here, but I think of antibodies, like, they're these kind of, like, y shaped proteins, so, like, kinda looks like a p sign with your fingers. Each each of these fingers is kind of like an arm of the antibody. And, like, it's really actually only the tips of the of your fingers, the tips of the antibody that engage in binding.

So this makes these really, like, nice therapeutic design targets for that particular region reason. The nice part is that, like, the rest, apart from the tips, is, like, actually relatively constant. So this is called, like, the framework region of an antibody.

And then design problem, you're typically just designing, like, the very fingertips. And you can actually choose, for the most part, like, these kind of framework regions that your immune system already recognizes. So antibodies, kinda like these y shaped proteins that your immune system, like, recognizes and knows really well.

It's kind of like your body's it's one of the lines in defense against pathogens and other types of diseases.

So, like or or nature in in certain ways. So, like, traditionally, these these types of, like, drug like molecules were either discovered in, like, these immunization campaigns. So, like, you literally will just, like, infect a mouse of the disease and see what antibodies it makes to try to, like, combat that.

Other ways of doing this is, like, super large yeast display, so on. So you might, like, start with, hey. I really like this framework, and how am I gonna, like, figure out the right loops to design to bind this target?

I'm just gonna try as much as I possibly can and just, like, literally search for a needle in a haystack. And this would be, like, on the order of, like, at least billions of potential molecules that you're screening against this one target. And in that case, you might, like, end up with, you know, one, two, maybe, like, a dozen potential hits to this target.

You actually you don't know much about those hits. All you know is that they kind of, like, stick to the target. You don't know necessarily where or, like, if they're even necessarily drug like.

I think, like, one big separator of chai and, like, a thing that definitely our partners like to see is, like, you can be really intentional with how you wanna do this this design process. You can say, I wanna bind this target in this particular area. You can even go back and look to the designs after.

Like, we validated that our designs. So you can go back and look and say, is this antibody engaging the target in the way that I expect? Do I think this will actually have the therapeutic effect that I'm going after?

Yeah. There there's a there's a nice mix of ideas that went both into the modeling side and especially on the product side for for dealing with selectivity and cross reactivity. So in some cases, you want your molecule to bind one target and avoid another one.

So you might have, like, healthy variants of protein and, like, disease variant of protein. You wanna avoid this this disease variant, or you might have some other similar protein that's, like, not actually harmful in your body that you don't wanna just, like, artificially block. So I think, like, on the modeling side, yeah, we've come up with ways of doing that, but I think it's even more interesting on the product side.

started Chai around two and a half years ago. At first couple months, we're like, alright. We're we're we're gonna work on protein design, and we're working on this.

We're making some progress. We're like, oh, it's pretty interesting. Like, we had some ideas and models.

And then kind of like that was right when AlphaFold three came out, and we were we'd, like, been talking about, like, man, we really need, like, an MSA pipeline.

it might be really useful to see a bunch of very similar protein sequences. And what those protein sequences that are really similar tell you is, like, kind of what positions, like, which amino acids end up being conserved across many variants of this protein. If you see, like, high levels of conservation or, like, kind of high levels of mutation, like correlated mutations, that typically gives you some indication that these amino acids are close in three d space.

So you kinda have this, like, two d view of a protein, which can then be used to help you predict this three d structure. So you're learning from evolution what was conserved because of the things that weren't conserved probably broke the protein and something died or didn't make it. Exactly right.

Yeah. Yeah. It's it's it's pretty remarkable that this works, honestly.

Yeah. One of my favorite, like, bio facts here. Yeah.

So so we were, like, kind of thinking, like, oh, man. It'd be it'd be nice to have, like, a lot of infra and whatever. So the AlphaFold three came out.

We're like, hey. We should we should, like, open source this model. We should just, like, you know, bunker down, build all the infra that we need.

I think, like, this will pay back, like, in the long term for sure of just, like, as a forcing function to, like, be where we are and also just, like, to contribute to the community as a whole. So it's interesting that you chose okay. This we're actually what we're doing here, we're building a model, but what we're really doing is learning how to build the infrastructure.

Is that kinda what you're saying? Yeah. That's exactly right.

And, like, I I had built a lot of, like, similar infrastructure in my PhD, but not at a production level for a company. So, like, at that point, I think we were five people. So there five of us at Chai, we're like, alright.

This is our forcing function. We have, like, a clear goal to work towards. It's, like, very direct.

Let's get this thing going and see how fast we can do it. You guys were at this time sitting in the OpenAI offices. Is We were sitting in the OpenAI offices.

Yeah. In the in the mission. Right.

So the like, what's the backstory on that? It's really interesting. Two of our other cofounders, Josh and Jack, had a relationship with some of the OpenAI people.

Actually, OpenAI co led our seed round. So we were, like, kind of thinking, alright. Should we get an office while we're only five people?

And it turned out, like, that office was mostly vacant. So we we got to sit in on the like, in the OpenA offices for a while. Taiwan.

One one other crazy story there. Let's see if we can actually share this. But this is a hilarious one.

So, like, we we're like, oh, man. We really wanna be the first to put this out. And we're like, okay.

We're we're one week out. We're like, the model is, like, almost done training. We're like, should we should we build a web server?

And then we're like, oh, yeah. Maybe not. And then, like, we ended up spinning up, like, this whole web server so, like, people could use it.

Like, rather than just, like, download the Git repo. It's kind of annoying, especially for biologists. And, like, we actually wanted people to use this.

So, like, let's spin up a web server. Let's get the technical report out, all this stuff. So we we ended up, like we were out for, like, forty eight hours straight.

It's, like, getting the paper over the line, getting the like, all the last things done on the web server. And then Josh was interviewing with, like, Bloomberg TV or something that morning, we've and been out for, like, forty eight hours straight. So Josh, like, runs into a room to do this interview on Bloomberg TV.

And, like, I think it was, like, seven in the morning. Everyone's in the office. Like, we we didn't, like, wanna be seen or whatever.

Like, the interviewer is like, oh, like, interesting company. Doesn't look like there are any employees here. But, yeah, it was it was a really fun time.

I think, like, the early startup days were just just super fun. So, yeah, after that, we kinda set our sights on design. And, really, what we're thinking is, like, we we kinda always had antibodies in mind.

We thought of this as, like, the most tractable problem. The nice thing with proteins is you you have this beautiful sequence representation. There's already a lot of research been done in, like, how do you autoregressively generate sequences?

How do you like, this sequence generation problem is well studied. So we were thinking, like, what what's a nice, like, area to apply sequence generation to in in the biospace? And it's pretty natural to do, like, linear sequences of amino acids.

So we started working on design. A unique thing about chai is, like, we're not, like we're designing antibodies. Like, we're an antibody company.

Like, we don't we don't really, like, pigeonhole ourselves into, like, one therapeutic area. So we, like, try to really tackle this problem very generally. So we were thinking, like, can we design mini proteins?

Can we design antibodies? Can we scaffold regular complexes? So, like, really just take a holistic view on, like, how do you design proteins in general?

And that eventually led to the CHI-two model. So that was our our, like, first flagship design model, and that's where the CHI-two paper and, like, our bold target discovery project came in. So we designed antibodies to 50 targets for that paper, got binders to about half of them with, I think, an on average around a 20% hit rate for binding.

And then afterwards, started working on CHI three. So that's our our latest series of model, but I'll break there.

CHI one has this, like, roughly a tokenizer, a transformer, something that looks like a language model, and then something that kinda looks like an image diffusion model. And they're all just, like, stick stitched together. The tokenizer is, like, not your kinda typical, like, Words of X style tokenizer.

This is like, I have a bunch of atoms in a molecule, and now I wanna, like, pull those into what I would call tokens for my, like, LLM looking trunk. And then that conditions this, like, kinda big diffusion model, which will then emit the image, which is some three d structure. So is it atoms or is it amino acids that are the input?

It's an interesting question as well. So we we have, like, all these different input tracks. So, like, one thing about biology is the data is inherently multimodality in a sense.

You have these this, like, you know, kind of token sequence representation. Each of these tokens has, like, a set of atoms that kind of dangles off. And then you also have, you know, some some properties of the different atoms.

Like, an atom is might have, like, a different charge. It might have a different element type, so, like, periodic table of atoms. And then these kind of all get bunched together into tokens.

Once tokenized, you can kind of process this in very standard ways. But then, ultimately, you have to get back to these, like, three d coordinates. So, like, in order to predict the structure, this is just some three d object, And that object goes through or, like, to emit that object, you go through what looks like an image diffusion model where you kind of go back from from tokens back to the atom representation.

two? That was CHI one. Okay.

CHI. Okay. So CHI one folding model.

Yeah. It's like and, like, all this bio stuff, it sounds, like, kinda scary, like, atoms, tokens, amino acids. Like, at the end of the day, my background personally is, like, theoretical computer science.

That's what I spent, like, all of my earlier years doing, transitioned to this, like, pretty late in my PhD. But I think, like, the background that you need is really similar to the background that you'd need for, like, any other field of machine learning. There are all these domain specific things that you learn about.

But, like, one one analogy or, like, anecdote I like to like to say is people think you can't work on, like, AI bio unless you're a biologist, but it's kinda like you can't work on, like, video models unless you're, like, a director or something. Like, there are all these, like, super domain specific things like, oh, yeah. To understand, like, lighting and a video, things like that.

But the end of the day, these are just machine learning problems, and, like, they're they're all solved the same way. Okay. So then Chi two, there's a jumping capability as well as an architectural change.

Right? Yeah. What we've disclosed about Chi two is, like, it is an all atom diffusion model.

So we we're trying to predict, like, you know, atoms in three d space still, but we're doing it in such a way that, like, the model actually has the ability to, like, design atoms, place them, decide which atoms actually are there. So, like, one way to represent amino an amino acid, like a protein token, is by, like, which atoms are present. So in the CHI two case, we were just predicting, like, alright.

Show the model. Let the model just kind of pick what atoms it wants to keep, and then map that back to what amino acids there are.

kind of like back to to like the image domain. So, like, try one would be like, you know, there is a cat in this image. Like, thanks, try one.

And try two is like I like I'll show you a background. Maybe, like, I'll prompt you with some some, like, image information like, hey. Put a cat in a field.

And Chai Tool actually just, like, give you back an image of a cat in a field, and you're like, that that's a good looking image. Or it's not, you might have some other model which kind of ranks the image. But, fundamentally, it's the generative problem.

Yeah. Okay. So there's taking that analogy a step further, it's maybe more like you showed a background and then it generates there is a cat, and then it generates an image of the cat at the same time.

kinda like the classic way of doing this. Let's talk about both. In structure prediction, like, alright.

I know the sequence. Like, I can, from that, roughly figure out the three d shape. And then there's kind of, like, the inverse folding problem, which is, like, given a three d shape, give me back a sequence that would fold into this.

And now you kinda, like, need to do both things at the same time, but I think, like, similar principles apply. Like, you can kind of have the model, like, think a little bit about what should this structure look like. Then you can have some other part of the model thinking about, like, but now what sequence would maybe support this?

And then, like, a nice thing with diffusion is, like, you can do this pretty slowly and pretty iteratively. So you can give the model a lot of time to think about, alright. If I change the structure like this, how should the sequence change?

And you can kinda just play this back and forth and back and forth. And eventually, it ends up kind of converging on something that's self consistent. It's almost like an EM algorithm.

Yeah. Exactly.

structure design, like, started to become a thing, we're, like, kind of at a loss for metrics. It's like, how do you know that your protein like, you designed some some, like, sequencing structure. Like, how do I know that this is legit or not?

Like, I can tell you it's just, like, anything. It's, like, totally out of domain now. Right?

What was my definition? Yeah. Yeah.

And, like, as a human, you can look at this thing and be like, I I don't know. It checks out. Like, even biologists are like, I have no idea if this thing actually folds.

Like, maybe some of it looks right. Even our biologists are surprised, by the way, with, like, some of our designs that, like, do end up working. What was done at the time is, like, we kind of came up with a bunch of metrics and, like, AlphaFold, it really is what enabled this.

So you'd take the sequence that you predicted. You'd run that through some, like, totally, like, distinct structured prediction method. So this is completely independent of your model.

And you say, if an independent model thinks that this sequence folds to a similar structure, then it has a higher likelihood of being correct than, like, you know, just whatever the prior likelihood would be. So you can take your sequence now, and you can measure, like, how consistent is this structure prediction method with the structure that you actually predicted for that sequence. You can now compare your design to an independent model structure prediction, and that became, like, a really good way of gaining conviction that your your design model was correct.

And people kind of, like, game these benchmarks for a while and kept pushing, pushing, pushing.

What do you do to stop that or to convince yourselves that you're doing something valuable? One of the nice things about structure prediction methods is that usually you have some calibration how kind of how confident the model is in its prediction. It turns out these models, they can give you a pretty well calibrated confidence prediction.

So rather than just say, this is what I think the structure looks like, I'll say, this is what I think the structure looks like and kind of, like, here are the parts that I'm not really certain about. And you can kind of aggregate this down to, like, a single scalar. And, typically, what people do is they'll look at, like, okay.

Like, not only how self consistent am I, but how much does this independent folding model even like the structure that it output? So that was one way of early on, I'd I'd say, to, like, just gain confidence. And then, like, another thing that people often do is they'll look at, the diversity of their generations.

Because, again, you could have a model that's perfectly consistent, gives you great confidence predictions back. Might be the same structure every time, like, same sequence every time. So you also wanna see, like, okay, how diverse are the solutions?

How many of these new problems can I solve in a sense?

you know, sort of get some ground truth on that? It's more that the feedback loop is really slow.

angstrom error, which is one third the width of an atom. Yeah. And we were like, this this, like, can't even be right.

Like, clearly, they just sent us back the wrong design. They just sent us back our design. Yeah.

Exactly. Yeah. Did you check for data leakage?

Yeah.

why CHI three? What's better or what? Yeah.

I think, like, with with CHI three, so, like, honestly, like, there was a CHI two. There's a CHI two and a half. There was a CHI 2.

7. There was Yep. Eventually, a a chi three.

And, like, each time we saw better and better performance. And I think, like, the the main thing with chi three is, like, we look at chi two and, like, we look at the targets we could solve. There was, like, a lot of internal discussion after chi two.

Like, hey. We made, like, successful molecules, binders to half of these 50 targets. What about the other 25?

You know, what can we do to make those better? And then, like, you know, we were at Split. We're like, alright.

Should we, like, study these targets that we missed and, like, figure out exactly, like, are there properties of these that we can look at, or should we just bet on the models? Like, will the models just get there if we put more time into, like, you know, just be bitter, less impilled in that sense and just really bet on the models getting better? And we we definitely took the the latter approach.

Like, we bet on the models getting better, and we just pushed as hard as we could on that front.

Yeah. Is it accuracy? Is that the main thing?

Is it binding affinity? What what do we So I think binding affinity is a big one. Like, you you you can't just bind weekly.

In order for this to be, like, a useful tool, especially for our partners, we need to start producing molecules that are, like, at or very close to therapeutic grade Mhmm. Which means, like, they have to bind really tight. They also have to be developable.

They have to have, like, all of these nice therapeutic properties.

probably be the most impactful in the usefulness of a of a product. Yeah. Right.

Yeah. Absolutely. The nice thing about structure prediction is there is a ground truth that you can compare against.

For design, you don't really have that. You're like, here's some new, like, disease molecule. Give me a binder for that.

And, like, if you wanna know if this thing really binds, you you have to send it off to the lab and and wait a while. For structure prediction, you can be like, alright. The model hasn't seen this sequence before.

It's never seen anything close. Does it actually, like, fold up into the correct shape? And we can just kind of hold that out of the dataset and check.

So I think I've always thought of structured prediction as this really nice speed run kind of benchmark to, like, validate ideas on. Right. Sorry.

I I didn't mean to say I meant, you know, sort of structural models in general. Yeah. But yes.

Exactly.

They're very, like I was saying, pragmatic about this, and they're like, they're willing to be proven wrong. And, like, I actually don't blame them for not trusting the models. Like, I have used these models in, like So many times.

They've, like, rightly so. Like, I I I am pretty skeptical when I, like, see any release. I I always have been.

So, like, you really just, like, need to show them the proof. And, like, they can give you this target that they are interested in or maybe it's more of something they've worked on in the past. They probably don't wanna, like, share IP right out of the gate.

But they can be like, hey. You know, I've had trouble with this particular target in the past. Let's see how how you guys can do on this.

And then once you show them the proof, they, like, almost overwhelmingly are willing to accept that.

Wanna add to that. Like, in in, the CHI-two days, like, we we kinda started with, like, a bunch of engineers and people that have, like, AI bio experience. We didn't have a hardcore lab scientist.

And, like, one of our first hires on that realm was Nathan Rollins, who I think he he started working in the Baker Lab at 14. Graduated from Harvard at, like, 18 and got his PhD by, like, '21 or something like this in the Marks Lab. And he was, like, super skeptical about Chai at first.

And then, you know, the results start to come in. He's like, okay. This is this is kind of interesting.

Like, this could work. And then, like, once the CHI two results came back, he was like, I need to bulletproof this. Like, nobody celebrate yet.

Like, all this. So I think, like, it's it's been really nice to have that level of rigor and to just have people who have really, like they've spent the time in the lab. They've designed proteins themselves.

They've literally, in in the case of, like, Andy, led several therapeutic programs, brought drugs to the clinic themselves.

to solve general lead optimization than it does to solve, like, a discovery. I think, ideally, like, we we really wanna be able to rather than think of this as a bunch of stages, I think part of the reason why we think of it that way is because the initial molecules are usually, like, not good enough to be drugs. Yeah.

And, like, really, like, we're kind of at the inflection point now. We're we're really seeing this internally at CHI where the models are getting pretty close to, like, producing molecules that could eventually or, like, are very close to drugs. So we we try not to make too much of a distinction between, okay, hit discovery, lead optimization, all of the different parts of this kind of preclinical pipeline are are, like you know, the the light the north star is to just really produce drug like molecules straight out of the models.

Of course, this is gonna be hard, and, like, there are gonna be, like, tons of roadblocks, and, like, you need to be able to, like, actually prompt the model to do this.

time horizon? Oh, and also maybe can you define epitope prediction? I'll think of this at, like, some different levels.

So the most basic level is, okay. I have some disease that I wanna target. What proteins are actually responsible there?

Like, actually figuring out biologically what's going on. Like, what should I be targeting in the first place with the drug? Because once you figure that out, it's kind of like a structural biology problem at that point.

You're like, alright. This, like, set of proteins is responsible, and, like, what's going on there? Well, this is interacting with some other protein that it shouldn't be interacting with.

And, conventionally, you just, like, wanna block that interaction or something with anybody. But that's kind of where these proteins interact and, like, the type of interactions that you wanna disrupt. That's typically, like, the epitope.

It's, like, the actual site on the protein that you wanna block. This is a ridiculously hard problem. I I'm with you on this.

This is, like, the harder problem. Like, just the amount of context that you need and, like, the global understanding that you need you need to get in order to, like, actually figure out what's interacting and how. But maybe let's take a few specific cases.

In that case, like, yeah, you could just run a structured prediction model maybe and, like, see where the model thinks this thing will bind. If it's highly confident in that, you might say, okay. Here is, like, the site that we wanna block.

I think in general, still very hard. And even, like, structure prediction, it's getting really good. And, like, a lot of people think AlphaFold two, like, solved structure prediction.

Not really. Like, AlphaFold two got, like, I think, 11%. The multimer version of this got, like, 11% of antibody antigen prediction cases correct.

That means 90% of the time, it's wrong. Yeah.

I I think it's still hard. I think, like, there there are a lot of cases that that are maybe tractable.

Yeah. Like, you're not gonna design your, like, crazy multi specific warheaded super intense formats. These are really things where you kind of have to design these from first principles.

Even just with bispecifics in particular, like, both arms need to now bind different targets. Mhmm. And you've kinda, like, have this multiplicative effect on your binding rate.

So, like, if you have a one in a billion chance of finding a binder in arm one and a one in a billion chance in arm two Yeah. You're not this just isn't gonna work with the traditional Exactly.

Yeah. You're I'm like, go ahead or Different entities. What is that?

Yeah. Divine. I thought you don't wanna talk about this.

Like, the the amount of headaches, like Too late. You already made fun of that. You're kidding, though.

Okay. So, like, just just, like, when you're actually parsing like, first of all, file formats for biologists. Like, I just they just don't care.

There's, like, no standardized there there are standardized file formats. Are they the best? I I don't really know.

But there's, like, also just, like, a lot of information that you wanna pack in. I have the structure. Here are the people who solved it.

This is the method I used to solve it. There's, like, a lot of stuff going on. And then depending on the method that you use to actually figure out what this three d structure is, you might have, like, multiple copies of that structure.

Part of it might not have really been resolved. You're like, it could be here. It could be there.

I'm just gonna give you, like, both options. So, like, the actual just parsing problem on the engineering side of, like, working with this type of data is, like, really difficult. This seems like something that LLMs can excel at, though.

They don't know all the edge cases often. Right? This is more back to just, like, a simplicity approach.

Like, LMs are very good. I'll absolutely give you that. Then you're thinking about, like, do I really want to, like should this function have 20 special cases, or should we be, like, really principled in how we approach this?

Yes. Like, how opinionated should we be in how we do this? We want a strategy that's easy enough for humans to understand and, like, when we're reading through the code base.

We really need to know what's going on here, what are the potential problems. And, like, sometimes that just comes down to looking at examples. But then I think, okay.

Once you've kind of figured out all the infra work and how you get data into the models, there's then, like, scaling the model. There's then scaling the infrastructure around the model to train bigger and bigger versions of this. And that's, like, a lot of work that Neil and the product team actually wins.

Yeah. I mean, that would have been my answer is the infrastructure part.

maybe the compute to memory ratio that you need and things like that. What are some of the, like, sort of cool or interesting optimizations that you've done there? Depending on the type of model.

So, like, we can go back to, like, a chi one type model. In that case, we're following the whole two, three architecture. And there, you're, like rather than doing attention over, like, this, like, normal sequence representation, you're in a sense loosely doing attention over this pair representation.

So you can think of this as, like, a sequence of length l squared rather than, like, typically length l. If you're doing attention over that, the way that you actually batch this up, it ends up being l cubed. Now you're you're in, like, a pretty, pretty heavy compute regime.

So the amount of flops that you're putting into every token stays it's pretty high. The amount of memory that, like, the memory bandwidth overhead of just transferring that from, like, SRAM to whatever, that's a real bottleneck in these architectures. So, like, even something as simple as, like, a layer norm can can take a long time, actually.

Like, that can be a significant amount of the compute that you're using. So I think, like, on our side, we've spent a lot of time just, like, optimizing and engineering taking engineering very seriously so that, like, these operations are, you know, at least better. We're always looking at, like, ahead of new chips performed compared to the older versions.

Sometimes that's even different for training versus inference.

Yeah. Supporting it. I have an analogous tangent on the model side.

Actually, one of the things that's kind of nice about those problems is they're, like, super visible. So, like, at least you know, like, hey. This this crashed.

This failed. For us, we just see, like, loss curve didn't go down or, like, we see weird gradient behavior or whatever. I think a lot of these same principles, like, you know, engineering first, that also applies on the research team.

One thing that I like to say is kind of, like, complexity and being bitter less than pilled, they're, like, fundamentally at odds. For example, I think, like, outfold three, I might get this number wrong, but I think it was, like, 23 submodules. And at that point, that's a really difficult system to optimize and study.

You're like, alright. What happens if I change like, if I tweak this thing in submodule 30 or, like, 21, what what happens to the whole system? And you can always think, hey.

We can make this better by, like, adding module 24. But, like, should you? Or should you think about just, like, removing things and lowering that complexity down?

But I think that's, like, a pretty fundamental thing at CHI is just, like, the engineering culture and just being, like, very simplicity biased. Have you all seen the picture of, like, the SpaceX engines? It's like Raptor one.

except for me and Kevin, really, like, we're the only people with, quote, bio background. Even still, like, we're we're pretty far removed. So I think, like, we we try to, like, look at every problem as a CoreML problem.

We try to think of, like, what's the analog in other spaces. So, like, even for image models, like, CNNs were built to process images. So, like, images should be looked at in patches.

Like, that was the nice inductive bias there. And people are like, well, you can just kinda tokenize this thing, throw it into transform, and it's gonna work. And, like, it did end up working, even, like, on a relatively small dataset.

and, like, that kind of limits what you can do with the architectures. They're not like, not only are they, like, costly in terms of compute, they're just, like, not efficient on modern GPUs either. You have small hidden dimensions, large sequence dimensions.

Like, it's, like, exactly the opposite of what GPUs are designed to process. One takeaway from, like, triangle layers is you're kind of just trading off parameters for compute in that sense. Like, that's, like, one mental model for thinking about this.

I might wanna, like, throw more compute at the problem and just trade that off for parameters because, like, I won't be able to hold as many like, I can't literally store these, you know, large pair representations and still do normal attention. So I think there are fundamental things you can abstract from the ideas like AlphaFold, but you can kind of just, like, tweak these and start building off of them in your own way.

research direction than a lot of the communities going in. Yeah. Yeah.

I think what we built at CHI is, like, it's it's very unique in a lot of ways, but also very tied to, like, what CoreML is is good at. Kind of what I was saying before, like, we try to map every problem into, like, a CoreML problem. We think, you know, how would you approach this if if it were an LLM or something like that?

general because, you know, that's kind of what we're seeing in the research. You can the models are very general that lets the product be very general. I'm thinking back to, like, in my in my CS theory days, my first adviser was like, we're working on some problem, and we we need, like, a polynomial time algorithm for something.

And he he would always tell me, like, never underestimate the power of polynomial time. Like, this is basically, like, you're allowed to choose, like, whatever exponent you want. And my first paper was an n to the twentieth time algorithm for this problem, and I was like, Andy, I did exactly what you said.

He's like, wait a minute. I didn't mean it like that. Yeah.

But I think, like, it kinda like, you can really help yourself. Like, you can free yourself up a lot when you're like, alright. I can kinda do whatever I want and then kinda simplify it later.

And I think that's really, like, a pretty fundamental way of thinking about things that we we leverage a lot at Shy.

makes a huge huge difference for that. That'd be my answer. I mean, you probably have a more model forward answer.

No. Like, I I think, like, like, biology is slow, which is, like, one kind of nice thing. And there's, like, not that much labeled data.

So, like, you could take all the publicly available sequence information out there. That might give you a good base model, but you still need some measurements on that data. That's still pretty time consuming, and then you need to, like, iterate on that.

So I think there are even just data blockers there.

like, you know, we make the models better, the partners succeed more, and just like, you know, that iterates on itself. Mhmm. So, like, I think that's, like, a pretty unique part of Chai is, like, one, just being able to partner with a lot of people, two, getting, like, the feedback on the product.

So, like, you know, knowing that it's very real, this is in, like, like, legit big pharma hands, and, like, they're actually running campaigns on this stuff. So I think it's interesting. We really have to be model forward, model focused.

Like, we need to keep delivering value. So that puts a lot of pressure, like, on the research team, the product team, first of all, to, like, to serve these things. The research teams always shoot for, like, better and better versions.

might not be, like, native to the CHI model, And they can, like, you know, kind of, like, ask the product team in in a sense to just be like, hey. We like you know, our our designs have property x. Can you make sure that they have those?

Yeah. And, like, maybe maybe one other interesting point is, like, if you think of this, like, on a per token basis, I don't know if there's another domain where, like, the downstream value of a token is, like, as valuable as it is for pharma. Like, you know That makes sense.

Thinking about, like, the actual drugs that come out. Like, these can be, like, multibillion dollar assets. So, the case of GLP ones, I think the two GLP one drugs combined are, like, maybe a trillion dollar asset.

Like Yeah. I mean, up until, I think, three months ago, right, GLP one's, like, total revenue was more than all of the AI labs put together. Yeah.

I don't think people realize that. I didn't realize that was crazy. We get the market cap way lower.

It's like crazy how relatively speaking the market cap Yeah.

yeah. I mean, it's maybe the biggest single biggest problem in biopharma is actually just the funding model. There's also have you have you heard of Ahrm's Law?

Yeah. Oh, yeah. Yeah.

Yeah. Exactly. Moore backwards.

Yeah. Moore Moore's Law backwards. So it's like and, like, compute, you know, it's kinda scales.

So you have, like, this nice exponential scaling, log layer scaling of compute, and you have the exact opposite in pharma. So, like, the cost of actually making a drug in pharma is kind of, like, increasing exponentially. So, like, the amount of money put in per drug is growing at kind of, like, an exponential rate, which is it's pretty interesting to see this.

Yeah. Which guarantees at some point, the marginal return on a new drug development will be negative. Exactly.

really, really cool drug targets in the future. That analogy is actually like one the the kind of, like, VC type investor ish model. It's, like, actually how we think a lot about research at CHI as well.

Our research team is is relatively small, I think, definitely compared to, like, a lot of the, like, the isomorphics, deep minds. Like, our research team is, like, you know, in the around 10 people. So, like, we're we're a relatively small team, but we kind of think of it as almost like an investing job where, like, you're investing ideas towards compute.

by fiat, what would that be? Oh, it's that's an interesting question. I think one thing that'd be really nice, like, just I'm, like, always in research land, very hard to turn off.

For me, it's probably just the validation loop of protein design in general. So, like, just being able to say, like, instantly, like, hey. This thing works.

This thing doesn't. There's still a bit of walking around in the dark that you're doing. Just to, like you know, you have you have some ways.

And, like, I think at CHI, we've taken this, like, very seriously, but it's probably along the lines of just, like, validating hypotheses and, like, you know, knowing for certain that things work. Yeah. That's unsolved problem for sure.

Unsolved problem. Yeah. Yeah.

And would be hugely valuable. Hugely valuable. Yeah.

Yeah. I'm gonna take a much more abstract answer to that, which is actually, like, talent obscurity.

the the generalists can often really accelerate the the precision engineering happening in the domain. Yeah. I think for for me, like, the base takeaway is that the field is actually working.

And, like like, not only does it have commercial traction, but, like, the research is, like, actually showing signs of life. Like, it's it's not even just showing signs of life. Like, the signs of life have been shown.

We're actually in a place where, like, the models work. They're delivering value. And, like, there's still tons of really interesting research problems to solve.

So I think there's a lot more low hanging fruit in this field than there would be in other fields. And think the amount of impact that you can have, especially, like, as a researcher, is just, like, unmatched in this in this field. For us, we're all very mission driven.

But even if you're not, like, it's a lot of fun puzzles to solve. Like, there there's, like, this kind of three d geometry angle. There's, like if you like diffusion models, there's, like, a million problems to solve in that regard.

We have this LLM looking trunk in, like, CHI one. There's just so much of, like, core machine learning is touched by these problems. We're still although we've made a ton of progress, there's still a lot to be done.

And I think it's just like one of the most interesting fields to be working in, which like, while also having some of the largest impacts on just, like, humanity.

Twenty two minute walk. Yeah.