Why AlphaFold Didn't Solve Protein Folding — Pushmeet Kohli, Google DeepMind & Sal Candido, Biohub

I was at DeepMind when Rich was with us and working and thinking about this idea of the bitter lesson. The thing that I took from Rich's original lecture that he actually gave at DeepMind was not about the actual notion of whether data is useful or how should we think about machine learning. I think my sort of take was that he was talking about something more conceptual.

And the conceptual thing that he was talking about is sometimes when we are looking at problems, we think about solutions in a very religious way. I'm a modeler or I'm a data generation person. I think that is the better lesson, that if you approach the problem with that mindset, you might not succeed.

The problem comes first, and you should be flexible in terms of your solution space. You should try both things. You should try to understand what is the problem that you're trying to solve, and if it requires modeling effort, do the modeling effort.

If it requires collecting more data, then collect data. But the thing that happened in machine learning in those times is people said, We are machine learning researchers. The dataset is there.

Here's some training data, here's some test data, and we will just optimize the model. That is broken, right? In the sense that if your eventual goal is to solve the problem, then you have to be able to look at both aspects of what goes into the process.

What goes into the process is not just data or modeling, it's also about expertise. So, the example that I have is with AlphaFold. We looked at what was possible with current existing datasets because we did not have the core expertise of now going to, or even the resources, by the way, to say, let's augment the PDB by a significant order of magnitude.

The amount of investment that organizations and scientists across the world had put in to construct that data was invaluable. So, there you have to focus on getting the biggest bang out of your buck by investing it in modeling. But in other areas, if you're approaching, say, cell genomics, then we took the same approach and said, well, what can you do with cell by gene if you're looking at cell genomics?

There, it was very clear that after a bunch of work, that the data was not there yet to be able to build, to go after that grand ambition of building the virtual cell. So, it just brings people together to focus on what are the actual challenges if you want to advance science rather than being religiously following advances on data generation or in modeling.

advice that I give to anyone who is working, who is starting in the area is that think of yourself as a multidisciplinary person. Understand the problem first. Why are you working on this problem?

What are we trying to achieve? And then think about what is it that will be needed, whether it's modeling, whether it will be compute scaling, whether it will be data generation. So, understanding of that problem is extremely important.

And of course, then you have to build your expertise in the house. It's not as if you don't have constraints. Maybe there are constraints.

There is only a specific amount of data that you can generate or there is a specific model size that you can afford to train. Understand those constraints and stop and try to fail fast, and look at what are the approaches that are going to be feasible in the long term in getting you to that intercept, the impact level that you're going after. Yeah.

So, let's approach it from first principles. When you think about the arts and crafts of how do you construct a model to be better, I think it was not an accidental thing. We did a lot of experimentation, but there was a vision behind it that all this scientific intuition that came from biophysics and biochemistry, that those interactions, that amino acid residues are not just doing their own thing, they are being influenced by other residues.

So, let's bake that in. If you have learned something from the scientific community, use it. Use that information and try to encourage the model and give it that unfair advantage that it has.

It does basically make the model much more data efficient because it doesn't have to replicate everything. I also have to mention that curating good data is an art in itself, It's not as if you will say, Well, put more data. If you replicate the same amount of data, you're not going anywhere.

So, it's not just about big data, it's about good data, and actually understanding the coverage of what data is necessary for making progress in the problem is in fact, I would say, a much interesting and much more challenging problem in itself.

Yeah. I think science, how it operates, it's basically by isolating something and then making progress step by step. So when people say the protein folding problem has been solved, at a conceptual level, yes, there have been some advances.

But I think it's also like when you think about that narrative of proteins being the building blocks, proteins are not blocks and they don't act as blocks. I say proteins are the building blocks all the time, but actually I don't believe in it. Proteins are extremely complex.

They are disordered. Their shape might change depending on the context. Like what AlphaFold did, and in fact John Jumper and I basically used to sort of discuss this, like what are we trying to solve?

We don't know what is the actual true ground state that protein stake, and what is the actual distribution of structure that protein stake. What we are trying to do is basically someone got a structure, deposited it in the PDB, and we are trying to replicate and get the same structure. That's what we did.

It just so happens that it's useful. But that does not mean that we have understood all of protein dynamics. So, I think at the top level, it's easier to communicate that we have made progress, but the scientists among us and in this crowd, we know.

I think that's the really important thing for us to understand and have that common ground that a lot of work needs to be done, and we have made progress and there's a lot to celebrate, but let's not stop the funding of protein structure prediction and protein dynamics because we are just getting started.

basically been then. So I'm a machine learning. My background is quite eclectic.

I started as a security researcher, then went into computer vision, Bayesian theory, and finally got into discriminative Machine Learning and Deep Learning and so on, and then AI for coding, and then finally science. So, when you ask me that question, the computer vision researcher in me was super excited about cryo EM micrographs. I was like, what is this sort of PDP data?

I should be working at the source, right? I should be looking at the cryo EM micrographs. I don't want those structures.

They must be missing out all the data. I should just directly operate at cryo EM micrographs. Now, getting models that can scale at that level with the right amount of data and can extract all the dynamics and distributional information that is captured there would be an amazing thing.

I tried it, but it requires more work, right? Still work on it. But you believe that this is a route?

This can give you?

So, I have a different take on this in the sense that I think when we think about models, think some level of understanding is necessary. So, I'll say what level of understanding is necessary. So, AlphaFold actually is not perfect.

It's 90 GTT. AlphaFold two was 90 GTT at that time on that set. But even if it was 95 GDT, but the PLDT score was completely uncalibrated, who would trust it?

It would just magically give good answers, but suddenly tell you here's the answer very, very confident and you will be working on it for the next one year and finding out it was completely wrong. The calibration of the uncertainty measure was extremely important. So, in that sense, we do understand and we actually made a lot of progress in understanding how does AlphaFold2 behave.

Now, that's different from how did it work and find the solution. There I agree with Sal that there's a notion of interpretability. Why did it work?

Why did it give this answer? I think if you think about it, AlphaFold2 was at the highest level interpretable because if you look at and this is its ability to generalize, and we did not discover it before launching. When we launched AlphaBolt II and made weights available, people found out that it was a great disordered protein predictor.

It could figure out which elements of the protein are disordered. So, it just shows that it generalizes. Now, interpretability asks the question, interpretable by whom?

If you're saying interpretable by a human rational system with the cognitive computational limitations of the human brain, then no, AlphaFold two is not interpretable. But if you are asking, is AlphaFold two interpretable to a much larger and more sophisticated model as to how it works? Maybe it is.

We just don't get it, right? But I think as users of AlphaFold two, we do need to understand what is it able to do and what it is not able to do. So, understanding the behavioral characterization and the strength and limitations of the model, and I think that is extremely important.

We can't just be using these models without having that behavioral characterization because otherwise it will rather than being helpful harm us.

is also in the eyes of the beholder, like who is interpreting it? If it is a human scientist who isn't trying to interpret how the model is going about and how will it make this prediction, that's a different question from another much larger LLM. You give some of these large frontier models of the future access to the activation layers and say, Okay, tell me, can you predict what AlphaFold will do?

Maybe they will be able to predict and they will come up with a theory of how actually AlphaFold two was interpreting and producing these results.

I mean, the point is AI results in the clear. AI is being used today already in every part of the drug discovery process. So, from that perspective, it's already there.

But if you are asking me the question, when will we see that 10x acceleration or 100x acceleration in the timelines? Then the other question is, what are we accelerating? Is it doing lead optimization?

Is it the target discovery? Is it the preclinical or the tox work? So, I think over the next few years, which is why this effort that we announced today is extremely important, we need to tackle some of these hard challenges of biology.

Only then will we be able to get these true unlocks of acceleration that dramatically transforms how drug discovery will happen. So, AI will be there in the clinic and will be impacting things that go into the clinic all the time, but the actual larger acceleration, that will be unlocked only with a better understanding of the biological models that this effort is trying to create.