Intelligence on the Edge: Liquid AI's Ramin Hasani on the Search for Device-Native Foundation Models

Thanks for having

Yeah. Absolutely. So Liquid AI, three and a half years ago, we spun out of MIT CSAIL, building on a technology that we have actually have been working on it, like, ten years before, like, basically, like, a decade before when we started the the the whole company.

We've been our objective function, like, at MIT has always been, like, maximizing the amount of intelligence that we can into smallest format of algorithms. So efficiency has been, the cornerstone of our research, and, we have been working on, you know, like, specifically thinking about robotics and systems that were, like, coming into the real world, you know, like the idea of liquid neural networks that was kind of discovered as well in my PhD thesis and, together with my cofounders, the four cofounders that I have. We have been we have been researching ideas about how can we build machine learning solutions that can go on robots and doesn't have, like, millions of billions of parameters so that we can actually host them directly on, let's say, CPUs, NPUs, or let's say, like, you know, like, smaller GPUs that are mounted on top of, physical systems while delivering the reliability of much larger instances of artificial intelligence systems.

Essentially, we try to do, we try to build alternative algorithms in order to get creative in the algorithm space to see how can we build machine learning systems that can generalize beyond the data that they have seen. Because when you go in the real world, the distribution shifts becomes like a real thing. Imagine you deploy a robot in an open world, let's say an autonomous car, a flying drone, a fixed wing vehicle, you know.

So all of these systems are going to go and very, very rapidly you're going to get out of distribution. So you have to actually build systems that are really comfortable around getting it, being able to get out of distribution. You know, humans are extremely good at it and humans do that extremely at an extreme, I mean, natural learning systems, animals also like the same way, right?

So they, they actually follow a really nice trajectory out of distribution as well, So that means like the learned concepts can generalize to data that you have not seen before. So our idea was that I can rebuild systems that can have that kind of properties, better out of distribution generalization. This has been not just our goal.

I think, an entire artificial intelligence field has been like working towards this thing and especially like in the robotics and closed loop environment, you know, like closed loop begin environment where you have an agent acting in an environment, you know, you want to have that kind of properties. So naturally, place that we started looking into was brains. We started looking into animal brains, and we started looking into how One thing that I really liked, I wanted to look at it from a first principle kind of approach, like how neurons exchange information with each other.

We started looking into brain of worms, small animals, and why worms, this specific worm C. Elegans. We started working on the brain of the worm.

The reason behind it is that this was the only animal that in 2015, when I started this type of research as part of my PhD with my co founder Matthias Lechner, we, this was the only animal that we knew the entire nervous system as a whole. And this animal exhibits massive amount of kind of sensory reactive kind of behavior, like amazing levels of control with 300 cells in its nervous system. That's what's fascinating for us because this is much smaller than any neural network that performs control at that time, like on on, let's say, like autonomous systems, you know, and it's warm.

It can do better a better, you know, dexterous movements and stuff like better than, the best robotic systems that we actually have in in in the world, you know. So we thought that, okay, so let's start understanding how neurons exchange information in the brain of the worm. From there on, let's start building nervous system more complex than complex kind of neural circuits so that we can get to the stage where we can, let's say, like build the next animal, build the next, you know, like, like basically follow the path of evolution of nervous systems, you know, and see how, how we can, how we can rise and evolve as, as part of this thing.

So there are equations that describe the neuronal dynamics of, let's say, like two neurons. In the brain of C. Elegans, because the worm is very small, the neurons do not spike.

They are graded kind of neurons. Like they behave like in an electrotonic way, you know? So they're very similar to artificial neural networks that we have because they're also very differentiable, right?

Because you don't have spikes in the activity of neurons. They are very differentiable. That's why it was even nicer for us, more attractive for us because we could apply learning theory to these type of neural networks once constructed, like let's say two neurons, four neurons, eight neurons, like let's say a 100 neurons next to each other, and then you start training them, you can apply back propagation as a differential kind of programming on top of a system that is built by these type of inspirations that we got from nature.

The type of differential equations that were there were also very well behaved. You can make them as complicated as you can, and then when they get more complicated, they mimic the biology much more. Of course, like from a computational footprint, they become more complex to scale, but it's still like you can sacrifice, you can actually tune how much complexity you want to encode inside a neural architecture.

You know, in artificial neural networks, like we try to abstract away like those complex differential equation that describes the neural dynamics between two cells, and we just show them with a sigmoidal function, gated sigmoidal functions, and now we have matrix multiplications that are coming, you're capturing the impact of the inputs that are coming to a system with a transformer based architecture and attention mechanism and all of those things. But these are all simplified computations for us to be able to scale these machine learning solutions. For us from a neuroscience perspective, it was very interesting for us to explore.

You see, if we actually start making, like bringing back those differential equation based competition and a little bit more elaborate form of competition into the behavior of every single neuron and mimic the behavior of like how two neurons exchange information with each other, maybe we can unlock something greater than what we have seen from an artificial neural networks perspective. Early on, the results were fascinating. We saw that with neurons, actually 12 neurons, could parallel park autonomously like a car, like a small car.

With 19 neurons, you could drive a car. With 30 neurons, you can fly autonomously, like navigating kind of a drone. And you can get sensory information and processes information with these a little bit more complex and elaborate version of these neural dynamics, which we called liquid neural networks and liquid for adaptability.

I called it liquid time constant neural networks, that was the LTC paper that we got out. And we coined the name Liquid for the fact that the dynamics of these systems are staying flexible even after training. So the models would be able to react to new type of inputs that they receive and even learn during backpropagation how to be adaptive towards the inputs that you're receiving.

It actually encoded a little bit more degrees of freedom on flexibility of the learning dynamics of a neural network compared to artificial neural networks, compared to other systems that you've seen. That form of dynamics actually allowed us to scale this technology into not just in robotics, but also applying it to predictive AI in financial services, in medical domain, and in many, many different places that we applied in, let's say audio modeling, multimodal video understanding, like we applied this technology over and over, and we saw comments that this technology is actually very, very amazing. Every neuron is much more complex than a normal neural network, but at the same time, you don't need that many points of computation in order to actually get to the results that you want.

We also saw that the out of distribution performance of these models are extremely good, they were suited for robotics applications. We had interactions with the United States Air Force. At that time, Boeing was actually hosting my postdoc at MIT, and, and, and, and there we showed that you can actually fly even jets like with these type of neural networks with like a handful of these neurons, you know, which was very interesting kind of topic to see how far you can push, like let's say out of distribution generalization with a handful of very, very small system, a system that is powered by these liquid neural networks as opposed to artificial neural networks.

As I told you that each node in a liquid neural network is very complex. Like it's a differential equation, you have to solve that and you have to solve a set of different, the more neurons you have, the more complex is a forward pass and a backward pass to a network. Therefore, needed to think about efficiencies.

These systems are highly nonlinear. We try not to sacrifice on nonlinearity because adding nonlinearity to learning systems allows you to build more expressive systems. This is a proof that actually I have in my PhD thesis where I showed like nonlinearity actually directly contributes to, especially on the smaller size of models.

But then the idea here for us was that like, how can you scale these systems? You have non linear systems, but it is extremely difficult to actually scale non linear systems. So you have to solve.

The reason behind scalability is the computational complexity of the models are hitting like even cubic complexity. It's not even quadratic, you're talking about cubic complexity, which is like too much. Then you have usually have, again, you have a set of differential equations, you can roll them out with a numerical solver, you run the numerical solver and you can step by step compute outputs that are desired.

Then the more steps you want to use in order to actually compute, then you would get more and more accurate results at the end. But then the problem becomes scalability again. You cannot really get to infinite precision with these numerical solvers.

One idea that came to us was that what if we just solve the whole system in closed form, you know, like literally like just we have a differential equation system with this liquid neural networks, each of them representing neural networks, neural networks of let's say, two cells that are exchanging information with each other at a certain level of abstraction. We saw that, okay, let's take this system and try to solve it in closed form. Turns out the closed form solution for this type of equation hasn't existed since 1907.

So 1907, there was a scientist called Louis Lapique that actually modeled their membrane potential, like how to model mathematically membrane potential in cells. That format of equation became like a fundamental of channel modeling, like how information propagates through ion channels inside a cell, and then how neurotransmitters are getting propagated to the other one. So 1907, this was Louis Lapides' memory and potential equation that is like an open differential equation.

And then really, I've seen like some scientists called Hodgkin and Huxley, they started working on really biological kind of grounding these type of differential equation and adding a little bit more complexity into the model. They developed a neuroscience model of one neuron, how how actually a neuron reacts to, let's say, a member potential. 1953, they started.

1963, they won a Nobel Prize for the fact that they actually developed a better and a more accurate representation for a neuronal dynamics. Then from there on, so that in every textbook that you read, this type of formats of equations, they do not have a known closed form solution. This format like liquid neural networks were also part of that type of equations.

Really for the first time we actually solved that and this was like 2022, around 2022 we solved the liquid neural network interaction of neurons with each other in closed form for the first time. This became a nature machine intelligence paper published in November 2022. This is called the Closed Form Continuous Time Systems.

This is liquid neural networks in closed form. The closed form, I mean, has massive implications. Why?

Because now I don't need to use any numerical solvers to actually run a liquid neural networks. I can now have not only hundreds of neurons, but now I can have billions of neurons next to each other, and I can actually scale these computations, still keeping the nonlinearity as part of this thing, know. Then in January of actually, February 2023, an article came out of a quantum magazine about, let's say, me and my co founder Matthias.

It was about a profile about like, the implications of having some closed form solution finally on on on on on the dynamics, you know, like, and how important this can be for both machine learning and also, like, for brain science as a whole. And then so my inbox was full of VCs, you know, and Silicon Valley started, like, talk like everybody wanted to throw it like a term sheet of us, like, to really get started and and really scale this technology because this was fundamentally different than a transformer based architecture and attention based architecture. This was grounded in biology, like the type of math that we have around the same times we have alternative models are coming out, state space models, like you've seen faster iterations of state space models, convolutional neural networks that came out like to be scalable.

All of them in a linear form because the scalability of alternative models, you have to linearize them and then you can scale them. We have something here that is also another category, it adds some operators that we learn from biology and from physics as we go to research. We got to the point where we can now become very competitive and building models for solving more and more general purpose tests.

When I say more and more general purpose tasks, I'm talking about modeling signals beyond predictive performance, modeling signals for language, for audio, and for vision in a way that humans understand. That's the complex format that humans understand. So the mission of Liquid AI became building efficient general purpose AI systems at every scale.

We coined the word efficient in our mission and we care about it so much because I think we are from a foundation model company's perspective, we are probably the most efficient published model company on the planet. I'll tell you why about this in a minute. But the reason why, like we set the mission into like efficient machine learning because we start at how much of the considerations that we had to do in order to really build a closed form solution, then taking this closed form solution and scaling it to, let's say, a system that can model language, a system that can model audio models, a system that can model video and vision in general, how much considerations they had to go into these things so that we can in a computationally tractable way scale our normal architectures to the point where we are today.

Today, the base of our technology, I'll talk about the core technology of the company later on, but I will tell you today, the technology became liquid foundation models. Liquid foundation models or elephants are pretty popular right now. We are ranked number five actually in The US from the number of downloads.

From the popularity perspective, we have over 1,000,000 downloads per week on the Hugging Face. That's a lot of downloads of these small models that we are building. They're very popular.

But the top ones from The US side are Google, Meta, Microsoft, and Nvidia, and the fifth one is Liquidia, so from The US side. We got ourselves there by consuming about 1,000 GPUs, having 1,000 GPUs in house. So that's like from a foundation model perspective, like getting to that level of popularity and releasing more than 50 models, instantiation of models that people are using in enterprises.

That's kind of the place where we brought liquid building on top of inspirations that we got from nature and and and the past that I actually like Portrait for you. Hey. We'll continue our interview in a moment after a word from our sponsors.

yeah. Tell me unpack a little bit. Why are we counting neurons versus parameters, and what does that tell us?

That's a great question. So I think early on this, we actually started talking about the number of neurons as the unit of computation, like for in our research, because that was like very interesting. Like you can have such, it's a, you know, like every neuron is a process that can be modeled by a mathematical equation.

Now, you can allocate basically like in a sigmoidal from gates or the more functions you have, the more parameters getting added to that single cell. The cell that now in a liquid model in terms of parameters, you can say multiply it by seven, you know, and that will be the number of parameters often, you know, so that in that differential equation. But that cell itself, because of the equations that it has, it it computes more than just, you know, like a forward pass competition.

It has like internal feedbacks. That's three degree feedback kind of mechanisms inside. All those parameters are actually like one to one analogous, like to any artificial neural network system that you would, you would build like in terms of parameters.

So a rule of thumb for like, when I say number of neurons, you multiply it by seven, you would actually get the number of parameters of the system. But this is for early on version of liquid neural networks. As we started the standardizing parallelization schemes for the type of mathematical functions that we want to explore, and let's say inspirations that we're getting, we just converge to the number of parameters.

Of course, today when we are talking about a liquid foundation model that has, let's say 1,000,000,000 parameters, we are literally meaning 1,000,000,000 parameters in the sense of like the any other kind of GPT that is getting built out there. Does that make sense?

the main challenge is turning sequential computation into parallel computation. So when you have degree, like when you start like going from a single neuron dynamics to multiple neuron dynamics, weight parameters of your system, instead of being scalars or vectors, they become matrices and tensors. So now you're talking about matrix multiplication, you want to turn the scalar computations into tensor computations.

Want to be able to, this is how you parallelize, let's say sequential computers. The problem is that not all the time, if you have nonlinear relationships in an equation, you cannot trivially create a one to one map between a vectorized computation and a tensorized computation. So those nonlinear attributes of liquid neural networks and any other nonlinear recurrent neural network, I mean, itself adds some degrees of complexity into the math of the whole equation.

Let alone if the recurrence has a non linearity on it, it becomes a lot more complex to disentangle vectors or write them with typical linear algebra kind of methods that we know into tensors. You have to be able to turn these computations into tensors to be able to compute in parallel. Compute them once at a time, you see.

So that's kind of the mathematical part. It's not related only to liquid runners. Any mathematical operation that you want to tensorize, if it is nonlinear, you're going to have the troubles that we're talking about.

That's why the state space models that actually came out, these are all around linear dynamics. We're talking about linear dynamical systems. Why those linear dynamical systems are important is just pure fact that we do not have proper ways to scale nonlinear systems.

You can apply nonlinearity to a linear dynamical systems. For example, let's say like a gated or let's say, you can add a sigmoidal function after you perform the dynamical system in a linear way, you compute all the matrices and everything like in parallel, but then you can do a point wise kind of application of a nonlinear operation on top of the entire tensor. That's what you can do.

But what if the relationship between the parameters themselves are governed by some nonlinearity? So you cannot really use typical linear algebra. You got to always approximate that nonlinear system into a linear system, then you would be able to actually paralyze the systems.

Does that make sense? So that's the fundamental bottleneck.

one of the properties of liquid neural networks was the fact that they have multiple feedback mechanisms. They're not just one format of feedback. They have like three layers of feedback between two cells in the synaptic dynamic itself, because we wanted to mimic how it is done in the brains.

It has multiple degrees of feedback. It has then basically nested nonlinearities on top of each other. That nested nonlinearity itself would also add degrees of complexity.

Even the closed form solution version of a liquid neural networks, from a dynamics points of view, you can now instead of 100 neurons, you can have 100 thousands of neurons, maybe 1,000,000 to 10,000,000 neurons, but the nonlinearity, you still have to compute the models sequentially because of this nested nonlinear relationships that actually happens in the circuit. So even the closed form solution would be limited in a scalability in the sense that you cannot paralyze them very effectively. But there are also like people are also like working on these topics a lot, like parallelizing, but not just parallelizing, but the speeding up sequential competitions.

So we're talking about sequential scan, scan is an operator that you can run on top of, let's say, doing matrix multiplication instead of doing it in cubic or let's say quadratic, you can do it in sub quadratic kind of time. So you can actually reduce the complexity of computations even on nonlinear operators without paralyzing them, without linearizing them. There are ongoing research going on, we are doing some of that research ourselves to really find ways to really sequentially run those nonlinear systems because clearly there is an advantage to have like those form of nonlinearities when you do function approximation.

We are talking about also scaling laws becomes also very important here. I believe like the scaling laws define architecture. You know, how does it do that?

When you're talking about transformers being this revolutionary thing, we're talking about maximum scale. The reason why transformer architecture and attention mechanism is such a brilliant architecture is the fact that it is unstructured. There's no structure.

When I'm talking about nested nonlinearities and all this crap that we have like in liquid neural networks, you don't have that in transformers, right? You have basically just matrix multiplication as the core functional kind of things that represent, and it is unstructured. You can literally make, multiply any matrix into any matrix of any size.

So the whole idea here is this, the larger neural networks that you make into infinite size, the larger neural networks you make, the more you want them to become less and less structured. And we've seen like the success of transformers at the trillions of parameters. Now we are talking about tens of trillions of parameters.

Like, I mean, the next generation of models that are going to come, we're talking about trillions of, and you can do that with a transformer architecture. As soon as you start adding a little bit of bias in that architecture at scale, things become completely messed up. So we are talking about right now, liquid neural networks, exactly.

These alternative architectures that we are talking about, they have a scale to heat. There is a regime of parameters that you can just do better. Let's say 100,000,000,000 parameters, up to a trillion parameter.

That's the range we are operating right now. In this range, the smaller is the model architecture, the more you want to specialize them for a certain application to solve, and mathematically they're biased to actually solve a certain type of tasks better than other types of architectures. So I would say biases on algorithms, the more biases you put on there.

By biases, what I mean is this adding a lot more non linearity, adding multiple gating levels on top of a neural network, adding recurrence, multiple different types of recurrence, adding, let's say convolutions, like as a, I mean, convolution itself, if you just keep the convolutional neural networks, they're also pretty unstructured because you can apply convolutions on any size of neural networks. So that question becomes like how much bias and what kind of problems you want to solve so that the bias in the neural architectures becomes kind of a function of, I would say, scale of the neural networks, as well as like the use cases that you want to solve the most. You see, so the spectrum would be the more you go to the larger size, the more unstructured kind of mathematical operators.

Use pure matrix multiplication, use pure convolutions, use pure, let's say, sequential scan, pure, pure operators. You see, it's like you get much more like, forget about gates, forget about like adding like, fancy, like, control, like forget gates and all those things. At scale, this is something that we have seen, like, because we have scaled like neural networks and we have, like, a really good understanding of that architecture of kind of variation at different scales.

You can do that on a CPU, like, on even even simple CPUs or simple GPUs, you know, because they don't have that much of probably, like, one of these liquid neural networks would fit on, I don't know, one to 25 megabytes of file size basically. You can literally put them there and then a CPU would be enough, like a Raspberry Pi would be enough to perform We have shown that on a lot of predictive specialized applications of AI that these systems can be very, very powerful and do stuff. They can be as powerful as the systems, like these clustering variants could be as powerful as the open differential equation version of the neural networks in speech synthesis.

That's one of the applications. In, as I said, predictive sequence, like imagine you have complex formats of sequences coming in, multivariate sequences are coming in and you want to perform at some sort of a prediction on top of them, these neural networks are actually pretty good. Then if you're talking about like more, let's say out of distribution generalization to the extent that you are like within a bounded period, kind of have like an open ended continual learning.

Are not, liquid normotors are not continual learning systems. They are more adaptive formats of computations because of the many gating's, many feedbacks, and many, let's say, input dependent parameters that they have in their mathematical kind of.

So continual learning would be a system that is continuously receiving new data and retunes itself. So it changes also the parameters of the system. Liquid neural networks are the dynamics of the systems are input dependent.

So that means new data comes in, the dynamics of the system would behave to those inputs, but the parameters of the systems like any other neural networks is fixed. It's just because the format of every neuron or every node in a liquid neural networks was like a differential equation, you would have a different dynamics than you. So dynamics, you know, it's another dimension that you add to number of parameters.

It allows you to compress more information, compress kind of more knowledge. If that's why with a smaller instance of the models, there's no free launch or there is no magic here. Like you're doing like also with liquid normative.

It's just that the axis of dynamics has been something that we added to the neural networks for what, for like being more adaptable. So I'll give you a very tangible example here. Imagine you're driving and all of a sudden starts raining, you know?

So depending on the format of the rain, you know, your autonomous driving system that is actually taking control of the car could react to those kinds of driving scenarios. If you have like a traditional neural network that hasn't seen that kind of environment, it might actually get like biased, like really because that rain that actually hits the camera, it's noise, a certain type of noise or certain type of adaptation of the input that is coming. But the reality hasn't changed.

The confounding variables of the whole environment hasn't changed. So that's why liquid neural networks, because they react to the input completely differently, they absorb the input, they apply the low pass filtering on top of the input. They are much more adaptable to those scenarios.

That doesn't mean that they change the parameters of the system to be more adaptable because there are two axes here. One changing the parameters of the system, the other one is changing dynamics of the system. I would consider continual learning to be attributed to where you continuously also change the parameters of the system.

In liquid raw materials, we don't do that.

Absolutely. So there's a system that we developed in house. We call it automated automated foundation model design, you know, AFMD.

You know, like, that's that's a mental mental learning system that explore that puts the hardware in the loop and then tries out, like, many different operators with an evolution strategy, the criteria is an evolution strategy, optimizing for a couple of things, optimizing for memory consumption on that device, optimizing for latency, optimizing for speed, while no sacrifice on quality. When we talk about quality, perplexity is not the measure, it's actually the application, the downstream applications that we care about. It's not just also public benchmarks, we're talking about 100 different benchmarks.

So the problem space becomes, from a meta learning perspective, becomes like a very, very complex kind of problem. Now, I'll tell you that why did we take this approach to design an architecture? Why?

Because we wanted to remove all the human biases early on as we are actually building architectures. One of the things that we realized culturally at companies, this is what I can tell you, even at largest Foundation Model Labs in The US right now, in Tropic and OpenAI, There are a bunch of people that are coming from the science, call them the avengers of the architectures or avengers of the post training or let's say pre training. These groups of people, there's usually a very small set of people that are calling the shots on like, Oh, you know what?

You're going to tweak this portion of this architecture so that it performs better. Why? Because in my personal experiences, it has started working better.

If you're really truly thinking, and this is something that is broken in all the Foundation Model apps, You cannot say that somebody has a fix to this, but now the recursive self improvement kind of process is actually fixing for that because now people are just finally realizing you got to give it to the algorithms. You have to be bitter lessened, be have bitter lessened pill. You got to, you, you, you got to be giving it to a systematic way to actually find out what is the true architecture for the, for the problems that you want to solve.

You can build like a general purpose computer. The insights that I shared with you in, in, in, in the format of the scaling loss of neural networks is coming out of our massive exploration of the space of architectures. The fact that in the smaller category of models, some biases on the architecture help.

In the larger instances of the models, you don't need to bias the systems. You can actually go pure convolutions, pure transformers, and you can go pure, let's say, recurrences that are very simplified. You don't need to add any form of you know, specialized kind of treatment like gating there, you know, gated Delta nets, you know, and then there's the mambas and there's the jambas and there's like, there's a lot of different variations of these architectures that are coming out.

What you want to do though, you want to be completely unbiased. Even us ourselves, you know, day one, like when we actually started liquid, we had one of the inventors of SSMs actually as our founding scientist, Jimmy Smith, you know, he invented S5, you know, like the simple, he basically brought the parallel scan idea to the SSM world, you know, Then we had, Stefano Massaroli and Michael Polly, they were actually designing like the, you know, like the hyena hierarchy, you know, there was like a convolutional kind of past that is going on. All of these are linear systems, you know, like within the realm of linear systems.

And then we came out of like these nonlinear, like hyper control theory kind of enabled, foundation models, which is liquid neural networks. A lot of big heads in the room and you to call the shots around. Again, the same problem in all of the labs, like you got to figure out.

We thought that, let's fundamentally solve this problem. Let's really systematically first principle, let's put all the operators of interest, whatever we think that could be an operator that can give rise to a general purpose computer, like a system that can, let's say a liquid neural network on its own is a general purpose computer. You can actually scale it if you can, theoretically, can get it.

So you put all those kinds of equations, all those kinds of things in a unified kind of theory. With linear input varying operators, all the space of operators can actually be encapsulated inside this linear input varying or let's say in general liquid input varying operators because input dependence was something that we have been talking about this for the last ten years. That's extremely important and we see that right now in transformers is also extremely important to have.

That input dependence is actually coming naturally with the attention architecture as well. But in a very, very unstructured way, it's not as structured as let's say, what is done in RNNs, in liquid neural networks, SSMs, in all of those things, you don't see these gated variables like to be very structured in transforms. Now, we actually have various variants of a convolution operators, various variants of a recurrent operator, then the attention themselves.

There's group query attention, there was the original transform, there's like many different things. Then on some of these variations of dynamical systems, and also various types of convolution, I said convolution before. But then then what we brought, we brought also the the liquid computational blocks.

Like for example, the gated, the double gated conferries of gating, what like a certain type of biases on onto like different dynamical systems or different operators. We added them and probably this space becomes something around fifty-one 100 different operators. Then you want to build hybrid models so that you can reduce basically what's the goal here?

The goal here is that maximizing efficiency of computation without loss of accuracy. That's the goal that we set. The objective function of the search space is also this, let's run.

Let's all of our compute actually thrown at this problem and let's try to see how the system is going to design this. We started doing scaling laws on this. We went as a small as a neural network as let's say 10,000,000 parameters to running the scaling laws up to like 72,000,000,000 parameter models like in these hybrid kind of structures.

So we have done a massive early days of liquid, 2023 and 2024 has been always proving out on a certain type of processor, what's the most efficient type of architecture that can come out. Turned out, when we put all of our biases away, all the gating stuff that we are putting around operators, they got simplified. So, like in Mamba style architectures, like you have a bunch of gating mechanisms in, let's say, gated Delta Nets, gated Delta Nets, have like a bunch of operators in there.

In linear attention, there are like gated variants. Everybody's like tweaking a certain parameter in the network like by hand because in their own experiments, actually they observe something. Turns out all of this has to go away if you want to get to the most efficient format of architecture.

It became the double gated convolution that actually came out of this massive search space like AFMD, this original system that we designed. This was one of the candidate architectures that came out that turns out to be very good on general purpose computer that are called CPUs. Because CPUs don't have, like they have a special infrastructure.

On all the CPUs coming out of AMD, Qualcomm, let's say Intel, you know, everyone that is building processor ARM, all the ARM processors. We try to get to the place where tested different operators. What's the, that generic kind of structure and, and, and let's say, computational graph that gives you to the most simplified, no, you know, no added, hand tuned, like there's no hand tuned features to anywhere, you know, it's just literally coming out of that, the test, the semantic test that we've done, And this became like the, the, the de facto kind of architecture elephant to a structure that we actually announced.

But while exploring these things, like we've observed like many different candidates also popping up, you know, like there's like too many things. And then we tried to like really figure out, for example, for let's say an NPU and neural processing units that is inside an AI PC powered by AMD or powered by Qualcomm or powered by Intel, which of these variants would be a better neural architecture that gives those hardware providers and silicon portals a boost on the amount of efficiencies that they are unlocking or let's say speed of computation, latency that they get, memory footprint that they get while having a computational graph that doesn't sacrifice any format of quality. So this was like the whole thing, removing all the human bias with a systematic approach or two architectures, even our own biases, you know, and the only thing that actually remained in elephant two on CPU competition is this double gated.

As I told you, we have nested computation in liquid neural networks originally. This nested format of competition was something that actually became very interesting to be part of these, a very simplified kind of neural networks that we built. The fact that we have unstructured 1D convolutions as the layers of choice, seventy-eighty percent of our networks are structured by these double gated convolutions that we have.

They're extremely simplified and they replace attention, they reduce the computational complexity by a lot, they reduce the memory footprint by a lot, they maximize the speed of computation by a lot and scale as well. At the same time on the quality, I mean, you have seen like some of these models are actually like extremely kind of competitive to to to to the transformer based alternatives. I think you asked a bunch of other follow-up questions as well, but I would pause here like for any follow ups here.

Yeah. Let's get into use cases in a minute.

That's a liquid that's a liquid structure. You know, that's when I say, like, this input dependent kind of thing that is actually coming into Mamba. So right before Mamba actually got out, you know, like, actually not right before, but a year and a half before Mamba come come out, we released a paper called liquid s four.

You know, this paper was if you just read the abstract of that paper, you know, like, we are actually for the first time, we are introducing this idea of input dependent SSMs, basically, input dependent SSM. The idea there was that let's bring the fundamental, let's say building blocks that we found has a lot to do with representation learning with capacity of learning, this input dependence element to really bring that format of gating to the neural architectures, to SSMs, you know, and then bringing that to convolutions as well with the other kind of, systems that we are designing today, you know, liquid foundation models as well. That format of gating, actually adds a lot of, you should just think about it, naturally it makes a lot of sense.

If the neural networks, it's not about the forward pass of the neural networks that it is adaptive, because once when you're input dependent, have input dependence the learning theory perspective, all the magic happens in the backward pass. If we think about when you're computing the gradients backwards, that input dependence operator itself, it's going to represent itself. So when you're learning from that data that you're seeing, you're also learning some sort of dynamics, input dependent dynamics there.

That second axis that I was talking about, it's not just the number of parameters of the neural networks, it's also the dynamics that the neural network actually learns. You have some representation of that dynamics being learned in the backward pass. Now the complexity of that gate is going to make a huge change.

Also it turns out that gate is extremely important also in language modeling, especially when you have sequence models like RNNs, let's say classical control, time RNNs, let's say discretized RNNs like LSTMs, then you have liquid neural networks, you can not very non linear versions of these linear versions of recurrent neural networks is an SSM. When you add this type of gating, can improve the language capabilities as well. It seems like for, let's say, discrete sequences, this gating also helps a lot.

It's not just for the continuous time sequences, but also discrete sequence because I would consider language to be like a discrete sequence of information. So you would have that sequential dynamics that is added by this gating. So it is a very right frame of mind to think about that feature.

For us, we think that this input dependence is one of the fundamental discoveries that we've done, that we learned also in biology, it happens. It comes out of physics. If you just put the math together, you're going to realize like, Oh, what is special about this dynamical system is the fact that it has this nonlinear input dependence.

That's kind of a structure. By biases, I mean, the complexity of that gating, even existence of that gating is a bias that you're adding to your system. You know, Now, is it needed or not at scale?

That's a big question mark. Like you're going to see when neural networks are a 100,000,000,000,000 parameters is matches. Multiplication is enough, like to really like perform like general purpose computer, like get to AGIs of the world or not.

That's a big question mark. But also, you need to also put this discussion about architecture and the obsession about architecture in perspective. Another perspective, you have to put these things, the learning theories themselves.

Right now we are talking about different learning schemes that are coming. For example, you can train a model with Next Token prediction. You can train them in the word modeling context.

You can train them in a sequential, in a long term horizon and long term history. There's so many ways you can construct also the objective function of the learning algorithm itself. That would also contribute to a lot to the learning process.

Architecture, I'll tell you that we found the purpose and the most important application of architecture or kind of research has been inefficiency. Really getting to efficient formats of competition without loss of quality. That's an extremely important thing because we're talking about the resource allocation problem right now.

As the models are becoming bigger and the demand of AI is exponentially going higher, you want to have more efficient versions of the systems to actually run the systems at scale. Otherwise, how can we actually provide access to AI to all? We are seeing early instances of this thing that the larger labs are wiping out the compute off of the planet.

Why? Because they have to train and they have to then host these models for all of us. Efficiency becomes a fundamental property of these model architectures that we're doing their research on.

I would say if you want to get into architectures that enable next generation of networks, sorry, next generation of intelligent systems in a way that they can become like human brain that actually performs computation with 20 Watts of power and it's already an AGI that does that between, that's like the massive exploration that goes beyond architecture. It goes into memory research, it goes into learning algorithms feature, it goes into data feature. You know, it goes into prior research, you know, like looking at it from a statistical learning perspective, that becomes kind of a, and then learning theory itself, like the limitations of learning theory itself, it also gets imposed on the today's learning systems because the definition of learning theory itself is actually broke like at scale.

Like what we have right now, like everything is ID, you know, like we are, we are building averaging machines, you know, you've seen the quality of writing and the sycophancy that actually comes out of AI systems because of the fact that we haven't gotten like this. Maybe like to some extent, like multi agents has been becoming the solution, and test time scaling became solution for the caveats that you're seeing from just learning like systems with autoregressive modeling as a pre training method. But I think there will be innovations needed, not just on architecture, but the whole thing as a whole.

You have data, you have models, and you have learning algorithms. Altogether, they can design basically the future that that that ultimate kind of a holy grail basically in this space.

It is one component of it. That's what I would say. Like, looking at it from a data problem, data representation from, let's say, an access would be architecture and the other access would be just the learning algorithms themselves.

Then we get into that recursive self improvement, is the continual learning characteristics of the new system. The fancy name the continual learning curve research that has been happening like for more than probably four decades now. Mean, and that's how I would actually characterize the whole space, you know.

So you cannot just look at architecture in isolation as, like, the fundamental thing that actually changes everything. Right? Yeah.

I mean, you're touching on the right things, and this is basically, it's like the regime you're operating and the goal of your system. Are you trying to build super Intel? Are you trying to build like the most powerful version of the AI system?

You need the most unbiased version of an algorithm. Now attention is an extremely rich, unbiased format of algorithms. Even if the competition complexity of attention is n to the power two, maybe we really need n to the power two to really get to that kind of level, you know, and maybe even we need like more complex architectures.

We've always like tried to actually reduce the complexity of architectures for the sheer purpose of the fact that we are resource constrained as humanity as a whole, we are resource constrained right now. I would say the discoveries that we have right now is just shows that there is a gradient on architecture that you can follow as you scale models. The gradient that you're following is the fact that for smaller models and specialized models, you can put as many biases like the scaling mechanisms that you're bringing in, you can play around with as many operators of interest, like in, in, in, in your competition graph, you know, and it is going to work and it is going to give you some sort of a boost if you're really maximizing for like linear, let's say linear time complexity, like you want to implement linear attention systems, you know, like just the fastest kind of, if the speed is like so important and you're actually wanting to even sacrifice a little bit quality, you can bring in like linearity and the whole system could be linear.

You don't even need some of those hybrids. One hybrid is just boost that accuracy because as we see there, as I said, the O and two, like basically the complexity that the competition complexity at a certain level, at a certain scale, it is needed for us to really like get to those performances that you want to. The larger in the network becomes, the more unstructured you can make it.

That's kind of the learning from that that whole kind of algorithmic approach that we started designing in all architectures. Yeah.

Great question. So for example, if you go to, let's say, biology, and you want to model sequential data in biology, we're talking about DNA data in biology. DNA data from a vocabulary perspective, they're very limited.

They're not language that is maximum amount of vocabulary and stuff. In DNA language, they're very simplified, but the lengths of those sequences you have to actually process are, let's say, for a human being or maybe for a bacteria I heard, it's somewhere between 100,000,000,000 basically elements in a sequence. For that long context things that you want to perform when you do not have that much of a large vocabulary, you don't need attention.

You can actually run on these data. You can run pure convolutions, pure SSMs, pure liquid neural networks, like recurrent neural networks and parallelized version of these recurrences and linear attention for extremely long context. You cannot do it any other way.

The reason behind it is because context has become so large that that quadratic cost of attention just kicks in. So let's say on biological data, you would want to have like some sort of a structure there. Then, there are places like video modeling and you know, like in those kinds of places, let's say you might want to have like different architectures and even different learning algorithms.

You have seen like the success of diffusion for example, there, right? Like you've seen like diffusion is also like a, I would say like format of a prior that you put on a certain architecture, but it's still like you, there's debate between like diffusion and learning algorithms or diffusion is actually part of the architecture. You can actually make that connection and you can make that distinction as well.

You have seen like on video and seen understanding probably an elements of diffusion would be needed. Some people still believe that with auto regressive kind of modeling, you could actually get there anyways, but we will see if that stays true. Then when you're talking about audio signal as well, for example, if you're talking about audio alone and language is not part of the whole thing, it's just pure voice to voice, or let's say noise to clear signal to signal.

These are places where recurrent neural networks is still very powerful. They're extremely powerful, especially in the low data regime. I would say models that have a lot more biases in them, like in the smaller regime, when you do not have that much data, this is the place where you can bring a lot of value.

The current neural networks and biases in architecture can help you in the low data regime to really fill out that gap with the feedback mechanisms that they have. The more complex architectures, the more you would be able to handle. Like the closer the architecture is to the dynamics of the datasets that you're trying to solve, the better of a learning system you're building.

In fact, the sake physics modeling, like when you're talking about, there was like a class of models like physics informed neural networks. These are again, another architecture that we're talking about here. They would be really good for physical simulations.

This is the other side of the spectrum that I'm talking about, the properties of data, the lengths of the context links and, you know, like, like all the other considerations that I said would change the architecture by a lot, you know, and then there are the, again, you want to scale these to the largest kind of regime, make it again unbiased. Again, a transformer would be able to add scale to beat this, but at a smaller scale, transformers would not be able to beat any of the the other formats of dynamical systems that we describe.

Yeah. A 100%. Think about places where you have to bring in latency to like, computation has to happen, like, in microseconds.

So you cannot really afford to have, like, larger computational complexity. You want to have the most simple type of system that actually handles that. You know what?

There you adaptive can systems. You can have many different formats of systems. And there, this is like one place.

The other thing is simulation in a whole, as I mentioned, like in physics, have like physical data that is coming, like you want to build a digital twin of a chemical reaction that happens at factory. So in those kinds of places, you would go towards like an extremely biased and even maybe like just differential equation based models like liquid raw networks today. They're getting applied to many, many different applications.

I see that because the original repository from years ago is still like open source. People are still building predictive machine learning models on sequential data or physical data or sensor data that it makes sense because the systems are continuous time and it makes sense to have like a continuous time dynamical system to apply to modeling those kinds of behavior. Then you can like today with these larger instances, like clouds of the world, could also like direct your agents to actually try out a bunch of auto research format.

You can just direct them to say, Hey, know what? Go pick the networks, the best neural networks would be the best fit for these type of dataset. Then here is the space of possibilities explore, based on the biases that you have.

Machine learning people are now mostly orchestrating these automated agent pipelines as you're building. So that's what's happening. In terms of Liquid ourselves, we are trying to contribute to the open source.

Are open sourcing some of the instances that are coming out of our, let's say, search spaces, like let's say LFM2, LFM3 is going to be like, again, you know, the objective is that the next generation should always beat the previous generation on that criteria that we care about, you know, without sacrificing quality, you know, like that's extremely important for us. And then the range of possible, like we decide, for example, how large of a neural network we want to make based on these neural networks that we have. And then this becomes kind of the open source version of our models.

Then we work very closely also with Silicon companies. You know, we work with AMD's of the world, Qualcomm's of the world. And, what we do with them, we try to understand the silicon roadmap that they have, the hardware roadmap that they have.

And based on that roadmap, we try to, let's kind of inform also like what they have to do even in the next generation of their ASICs, because when we have this understanding of the algorithmic aspects of intelligence and the variety of things that they could be supporting in order to dramatically reduce the cost or satisfy the constraints of the use cases that they want to enable, these are the considerations that you have to do. Or even building like to the point that we can build also like specific foundation model graphs for them. That's the projects that we do with semiconductor companies.

In terms of like on the commercial kind of side of things, we also take models, we apply them in many different, I call it the category that Liquid AI is very enabled. We call it device foundation models, you know, anything, all the processors that are outside of data centers, we try to apply our technology to those places. This is outside of data centers.

Inside of data centers, apply to constrained use cases. Use cases like low latency, like ultra low latency applications of AI. Extremely long sequences and wanting very, very small memory footprint of your AI system.

You want to have one cost efficient implementation of these found that you have. You know, we contribute to memory, speed, latency, without sacrificing quality and cost. These are elements that define use cases where liquid can come in.

In terms of access to our technology, any new generation of our architecture that comes out of our efforts of, let's say architecture search or new modalities that we check. For example, with Shopify, you're exploring many different range of applications across recommendations, search, product catalog, understanding multimodal systems very well. That's something that we do.

Our models are in production like we Shopify right now and they're improving the quality of the click through rates and internal criteria that Shopify actually cares about. Our models outside of data centers, they go inside cars. Some of the use cases that we enable is basically in car intelligence.

For example, we power that. Recently we have a contract we signed. It's actually a historical one with Mercedes Benz, where our models is going to be powering the audio and also the visual elements inside the car.

Whenever you want to talk to your car, basically the new voice would come out of a liquid foundation model, you know, and we control that with a model that is like giving you the quality of the best models that you have seen so far. What like, let's say like the audio models that you've seen, like all around, but at the same time, it is 600 megabytes. It can actually fit inside like the smallest processor that is inside that car, you know?

So like, and that, that that's kind of that's, that changes the game, you know, like, you're enabling local AI at scale, like on places that matters, you know, on cars on, let's say on mobile phones, because you can do that there's a lot of devices, like you're talking about billions of devices in the world, that mobile mobile devices can can can be extremely useful. And you've seen that that that market itself, by the way, just the mobile kind of business is $500,000,000,000 market is absolutely insane. And, it is as big as kind of the data center kind of market.

So you can imagine like there's, there's a parallel here to be made, like, you know, the efficient markets, you know, and also constrained intelligence market. It's something that Liquid AI is going after Laptop markets, let's say wearables, robotics, manufacturing, IoT systems. Anywhere that we have a processor inside a system, Liquid can bring in intelligence on top of that.

So the aspirational goal that I have is that like we'd really build an intelligence layer on top of the diverse formats of hardware that is out there and and processors that are available in the world. You know, that that would be something that I would want to have.

And we need to do it. You know? Like, we need to we need to take advantage of it.

I mean, because we don't have enough energy to host it. I feel like like we we are realizing it right now, like how difficult it is to really power this format of intelligence that we have at scale. We really have to work smart.

We cannot just do the simplest format or maybe more more constrained use cases. Like, let's say, if you have like a one shot, let's say, predictive task or maybe data extraction task and stuff, you don't want to call the fanciest type of intelligence, like the the mid class level kind of model to perform data extraction for you. You can have a variety of different intelligence systems that are doing many, many different things.

Then for, let's say, the most sophisticated problems in the world, you can go to the most sophisticated AI systems that are existing in the Cloud. It's inevitable that the processor world outside of data centers should get enabled as we speak right now. I think there is actually some talks coming out.

There's a lag in Silicon Valley, investors that are actually, like, just, oh, okay. Now that makes sense. You know?

There is that lag that is getting gap getting answered right now. And I think I think we are in in exciting times also for efficient AI in general.

Yeah. Well, I mean, that's one of the You see, there's different layers of abstraction. We're testing out also the AIs to see how good they can get on the kernel design side of things.

For the things that they have seen in the open source, like the kernels that are available in the open source, this, let's say the GPU structures that they have seen like out there, but not the NPUs, like the most, hidden type of NPUs that, that, that their, their IPs actually not disclosed. Those require like, let's say another architecture search because we don't have those knowledge. But I would say kernel design is at a level that we can actually start automating.

So, you can definitely have loops where you can juice out the best post hoc optimization because some of the things that we were like, whatever we talked about so far has been optimization that we do before design of a foundation model. Like, all sort of considerations before we start treating your foundation model. Right?

The approach that Liquid takes before getting into, let's say, like pretraining your model, we run this massive architecture search before getting a little bit of hardware in the loop. But then right after you design the architecture, then there's like all these sort of like quantization of Edward training, changing the bits of the system, like going into the kernel level and trying to even juice out even more off of the system, you know. All those postdoc stuff is also something that is like, you can automate automatically kind of orchestrate it with a kernel engineer that does that and performs those optimization, and we do that.

That as a post hoc optimization step. That being said, we have the capability in house to define operators at a kernel level as well. So, for example, you could say, matrix multiplication have like on this front of kernels, they are like probably 100 way.

100 ways that you can actually structure matrix multiplication with caching, with not caching, like how distribution between CPU workload and then GPU workload, how do you actually do the entire forward pass as you perform. All of these optimizations could also in principle become part of that massive search space as well. These are kind of the questions that some of the cell phone partners are asking us, how can we even before design a foundation model, figure out the competition graph first, and while having like the guarantees that when the training is finished, because that's like a costly process, you know, multiple millions of dollars has to go into full training of a neural network, you know, even the neural network is small.

You want to have like all sort of like inference optimization before the post hoc, like, quantization of where kind of optimizations. Right? But, yeah, in principle, we can definitely launch something like that as part of our search.

I feel like in every iteration of technology, software technology, like the level of abstraction that the hardware providers should be building for, it's coming up. For example, we've been talking about CUDA modes for a long time now. There is no CUDA mode anymore.

Why? Because if you look at the AMD's, there are agents at the kernel level at that level, now they can automate that stack. The stack that foundation model companies should, sorry, hardware companies should start working on is the intelligence layer.

Like what the NVIDIA is doing with their Nematron project. If you see how this project is actually quite successful and probably a couple of billion dollars actually went into design of these Neutron projects. If you really look at what does it do for NVIDIA, is just basically building an intelligence layer on top of NVIDIA's compute so that you have an easiest barrier of entry as an enterprise to buy NVIDIA solutions.

So the sales of the solution is always around those things. I'm just talking about the enterprise business of these hardware providers. If they want to sell more hardware, they're going to up their stack from that current level optimizations, those are things and they get into the intelligence layer.

Now, does that mean like they have to become a foundation model company? To some extent, yes. And they have to be able to train that intelligence layer themselves.

And NVIDIA is like a success example there. How these things actually like, paying dividends for them, like building the Nematron project. If you look at the other foundation model companies like the other hardware companies, they have not done this yet.

They have started strictly optimizing their models for the open source architectures that are out there. That's also extremely valuable. You have to do that.

That's like a given. But again, if you want to create a differentiated value and you want to be successful, I would say you got to be able to bring this back to the intelligence layer. And that intelligence layer should naturally fit on top of your hardware.

If your hardware have limitations, let's say in terms of like to your competition, software can always help to give you that juice. Like for example, if you're talking about maximum token speed on an AI PC between an Intel computer, between, let's say, an AMD computer and let's say, a Qualcomm computer, they might have their own trade offs. The company that is going to have an edge is a company that actually owns that efficient intelligence layer on top of their hardware so that they can say, at the end of the day, you're going to run tokens on top of this system.

It doesn't matter what's the bandwidth of my CPU. If it's a little bit lower than you or higher than you or the amount of memory that I'm actually having in there is lower or higher, with the intelligence layer and software optimization techniques, I can actually get myself to the place where I'm actually the winner here. For example, all of my laptops or all of my PC solutions that I'm actually building as a hardware provider, they are naturally coming with an intelligence layer on top of that.

NVIDIA is entering this game also heavily as a competitor in the CPU space now, as you're seeing devices like Google actually entered the Android Nexus that they have. They're replacing Chromebooks with Androids, With Android laptops. This is what they're putting out there.

I think it's called aluminum OS or something like that. I don't actually know what exactly it is. Then Meta is going to come with the devices themselves as well.

All the processor builders, I have a feeling they need to get a little bit closer to that intelligence layer and try to leverage software and efficiency considerations into account when they are going for planning, like planning ahead, where things are going. Why that intelligence layer is important? Because right now, agentic AI is something that everybody else wants.

All that applications you want to do on Silicon, you want to do that on a base of intelligence that is already provided to you. You want to build harnesses, you want to build like all sort of the harnesses are basically applications you can build for solving a certain problem in, let's say, in traditional sense of an application. Market can get enabled on top of chips if you have a very nice stack that enables intelligence on top of what you have.

The other thing that I would say is that, Silicon is always, if you're going outside of data center, there's a massive diversity of Silicon, but then the resources are constrained no matter what. You don't have that much power. So you got to be very careful at what sort of applications are the most important uses of your hardware medium and platforms.

Then those use cases, if you already have an intelligence layer, you can identify when is the time to sunset basically a set of, let's say devices. For example, I strongly believe, I think glasses could be like a format of competition for hardware. Later on, like devices, I think glasses would be like very, very interesting kind of medium and I think somebody is going to get it right eventually, it might actually be a replacement for, let's say, for laptops.

Then but is it? We really need figure out what is it. But if you really want to define a market and define where the field is going from a hardware perspective, I would say you got to up your software game by a lot and really get hands on on and investing a little bit more more energy and and and capital into into into the r and d side of things on the Foundry Mono side, you know, because that's the new base of software, you know, like, inevitably.

Or why do you want to change the model? You know, like, it it it it bring it brings it to that choice kind of question. You know?

You're given basically a default. Now you want to switch this thing by all means, but why do you want to switch it? If this model that the intelligence layer that is in there, it's not fixed, it's kind of an adaptable kind of system.

It is, a self improving system, with call like to your datasets and stuff, you can actually have like a platform that does, let's say full fine tuning of that system. It is enabled. And there is not just one model that you can load into your system.

There's not just one Cloud model or one on device model that you're going to use. You got to be able to orchestrate between, let's say, many different instantiation of this model to be able to build application. We're talking about a model class that would actually like sit on top of, let's say hardware on a laptop, for example.

I would say you need to have that, but I mean, if the default is just giving you that efficiency and that it's ready to go, that's like choice. That's something that I think NVIDIA is trying to propose in enterprises when they go and sell NVIDIA NEM projects, when they go out there and sell these things, it makes a lot of sense because you already have a project, you already have a multimodal model that is loaded on top of your, let's say, the PC that you want, and why should I switch? Because they already have done all sorts of optimizations for me, and it is running extremely fast.

Why do I need to change it? And the model itself is tunable, I can use their Megatron framework to actually tune the model. Now, if you don't want to do, then you want to just choose another model to host it.

As I said, this has to be given. Your hardware should be optimizing for the entire open source ecosystem and, know, like models that are available. But at the same time, it would give you an advantage, like, you know, like to yourself and to your customers.

Yeah. Well, I mean, that's a that's a that's a great question. Obviously, like, the the local co works is, like, as it stands today, like, it's it's obviously just to open minds like, Hey, you know what?

This type of applications you can enable. The class of what this format of local agents are going to enable is basically a local computer. You have an orchestrator.

If something is so complicated, it should be able to send it to the Cloud, fetch the answers for you if it's not sensitive. You have, let's say, models that are PII models. It should be able to use the PII models to really filter out all the personally identified information, send it to the Cloud for you.

You don't need to even see these models. These models should be on the background. The model that you have to tune is that orchestrator that'd be able to route between many different services or even smaller specialized models that are doing stuff and also some of the Cloud models that are out there, right?

That router is the computer, like that's like the local computer. That's like a definition of an, but when you open your laptop, it should be just that, like you literally have like just that. And then you start like working with all the services you want to have is just the same way that you communicate with your, let's say assistant, like it should be like in the format of an assistant that does all sorts of those jobs.

It takes a while for people to get not weirded out by the user interface, just being that and not seeing all the file formats and what do you have to do? But you got to get used to it. Now, soft code is even as for the ID.

Now, is the 24B off the shelf is going to be there on the quality that does all of those stuff for you? No, it's not. None of the local models today are there.

On the local models today, you got to fine tune them. You got to get them to be specialized for the stuff that you want to do with the proper explanations for your Cloud Agent. You basically give it these things, but is Cloud being able to go out there and actually build this model for you?

Let's say, do the fine tuning and getting it to that place? No, because today, even Fable level models, they would not be able to undertake, first of all, you wouldn't have access to that because Entropic is actually having access to the auto tune and stuff like automated models, models generating models kind of platform. There are companies and also ourselves, we are building platforms that enables you to do fine tuning of this whole thing that would cost you between tens of dollars to let's say like low thousands of dollars to actually get you to the Cloud quality for your and beat all the checks production quality model.

It's not going to cost you like tens of thousands of dollars. It's going to be between tens of dollars to actually those thousands of dollars. That's the scale that we are thinking about because again, efficiency here matters a lot because we want this fine tuning to happen on the compute that you already have, or if you don't have it, you would actually put it in a secure and proper data center that you would actually host it from the providers basically.

You would be able to get to that quality of the models. There are platforms that hopefully as we go forward in the next few months, you're going to announce some platforms that you can hook it just directly in your channel, you don't have to do anything, you just say, Hey, you know what, go call this platform. Now for fine tuning this thing, go call this platform.

This would give you a production grade foundation model that you can deploy it now for yourself. Either fine tune on that data or depending on the use case, depending on what you want to do, it doesn't even have to see that data. It can even synthetically generate data on its own and actually train the model to be like just the perfect, reliable tool caller and understands when is the shortcomings of it and it can actually go away and deliver it to some other place.

You know, when when when you think about intelligence, like, way the way I I look at it right now is that, like, transformer based networks and also, like, our type of, let's say, current architectures that are landscape of architectures that are available, they gave us with scale, they gave us in context learning capability. The thing that actually emerged from Next Token you see, the the word is important, emerged from Next Token prediction. Intelligence for me is an emergent property.

If you to miniaturize kind of intelligence, bring it to the physical world, I don't believe with the current set of algorithms, we would be able to get close to the, let's say, intelligence per watt that human brain is actually providing. You're not going get there. Why?

Because I believe human brain over the We also have to consider the amount of energy that went into design of humans as a whole of biological evolution. It's actually a very, very long kind of process. A lot of the pre training process that people say, Oh, you're going to see the entire world.

Humans do not need to see the entire internet to be able to reason about something. No, but humans have gone through years of evolution. So I would actually attribute it a lot to the evolutionary aspect of things.

But again, I would say human intelligence came with multiple mechanisms of in context learning. When our current AI systems do in context learning, they learned a vague representation of one algorithm, which is least square. It's basically least square.

So what they figured out is basically gradient descent in a mushy way, like for some use cases, with some examples, you would be able to make the system understand and give you the next example. These are beautiful properties. This is what I would call emergent property of the systems.

You set the algorithm to be next token prediction, you got a vague version of a gradient descent in context learning capabilities. For humans, you don't have only emergent gradients. You can learn by examples, but you can also do reinforcement learning.

You can run simulations in your head. You can do all sorts of algorithms. You can do Bayesian statistics in your brain.

You can, you, you see what I'm saying? So you have like a diverse set of algorithms emerged from the way that humans actually got designed and got intelligence. For me, I believe if you start optimizing for those, you try to force your way into the system to become a reinforcement learner by learning from trajectories, that's not the right way to actually get to emergent property of intelligence.

I believe the format of intelligence that is going to be miniaturized, being in a smallest amount of time, maximum amount of intelligence, like the human brain, we need to come up with these identifiers, like these confounding kind of algorithms, which is next token prediction is one of those. What else do we have to do at the beginning of the design of these systems so that reinforcement learning, curiosity driven like intelligence drive with limited amount of energy from that final system that actually comes out of it? So I would say that emergent property is something that really there's a lot of research that has to go in that direction, I would say.

And multiple kind of variations of way of learning that would enable the next generation of artificial intelligence, I think.

No. I think we covered a good lot of information. Think we went into some things that I usually talk about.

Now in my current role as the CEO of the company, I get to talk a lot more about the business opportunities and the market aspects of the intelligence and stuff. But it's still at the core, we are scientists and we are really pushing the boundaries. We are actively thinking about it.

Day to day, you cannot imagine, even right now, if you're talking, I'm already in the backend, I'm actually training some stuff. And I really don't want to get away from that kind of mentality because I think everybody can actually be able to now build. And I think that the times couldn't be better.

The agents are pretty amazing at giving you this opportunity of really doing frontier form research. If you're curious about something, it's just that we need to find people that are a lot more curious and improve the curiosity in people as well to really get, instead of just fear of everything. So I'm a techno optimist, probably you've observed that.

I usually talk about technology reading in a very, very positive light. One thing that I always say is that we need to really get to the level where people are as curious as the scientists were. What gets us science?

And the purpose of science has always been satisfying our own curiosity, and also understanding the world around us, right? So that's kind of the purpose of science. And I feel like that with the current superpowers that are given to us, like with these AI systems, I think everybody could actually start really contributing to understanding the world around us and also generating more value and also just satisfying our own curiosity.

But also it requires a certain degree of rebasing our biases towards like, oh, what is work, what is not, and what is automation? What is an army of agents working for me? What's that definition even look like?

And then changing culture and changing way of thinking about this thing, not just for individuals, but also for enterprises where we are dealing with as well as Liquid AI. And yeah, so I think that change is something that I'm looking forward to, and I think this is something that would happen. The pace of it is extremely fast, but I think we gotta do it one way or another.

Thank you.