Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO

We we had a bunch of art installations, which is kind of fun seeing, like, our products on pedestals next to, like, Rodin.

I actually first met Eric, who's the CEO, through an investor. And back then, Eric was already thinking about building a new kind of runtime. And he he got there thinking through why are workflow orchestration products so hard to use?

It's because you have to run them on Kubernetes. Kubernetes is hard to manage.

Yeah. I mean, initially what we were thinking about was if we build a better runtime, it's a very useful primitive in itself. There's a lot of things that get solved by serverless functions.

Like, can do ETL stuff, you can do job queues, you can do all this, like, bursty processing, which it turns out every company had needs for. But then we also were thinking about this as, like, this is a primitive that we can build a whole collection of products on, which are very verticalized. So perhaps data engineering would have been the first one, but we were thinking about inference.

Back then, was more classical inference, like computer vision stuff and running XGBoost and whatnot. But we we added GPUs to the product a year before ChatGPT came out. Nice.

We just didn't think it would be that big of a deal. Yeah. Just like add a 100.

Was there any like early key problem that really sparked off why you built it? Yeah. Primarily, it's just none of the tooling that was out there was built for, one, a really great developer experience.

And also there's a general trend of a lot of the workloads that we were seeing were very this is I wish there's a better word for it, but compute heavy. Like, they need one, like, need a lot more resources, so you need to burst up and down a lot, versus like Kubernetes designed for like slow scaling and more for like web server use cases. And also, there's just a lot more specialization in, kinds of environments these workloads run-in.

Like, we kind of sometimes need accelerators, sometimes need different kinds of images.

really didn't want people to spend so much time writing YAML. And it seemed like you could really condense the surface area of what you're doing, put it in code so you can actually operate on it, just so you can operate in other code and, like, build stuff that's more expressive and dynamic.

Yeah. We we we never really got pushback for that, because the nice thing about Modo is you can bring whatever code you have. And, sure, the DSL is at the sort of configuration there for what hardware you're using, how you're scaling things up, but you still own the code.

primitives that are kind of different than if I'm doing this myself. Right? We've actually changed our SDK team to think about agent experience, sort of developer experience.

And and we think that the same benefits that apply for DX also actually apply for AX, which is why would you have an agent read through hundreds of Kubernetes files and like write YAML that's not even typed when it can basically make a couple of changes in a decorator and it gets this sort of self provisioning runtime of being able to see its changes live in action. Yeah.

Yeah. I mean, people are looking at code. One thing we actually still see is is really important is observability.

Like, how good is your dashboard? And of course, like, we have we push a lot of it to the CLI so the agents can do their own investigation, but you still need humans to go interpret what's going on and, you know, make judgment calls and whatnot.

Model is a cloud platform that's built for where we build the primitives from scratch for AI applications. And right now, it basically covers inference training, batch crossing, and sandbox workloads.

of of the world because, yeah, we we think the differentiator for us is the our other workflows that need specialized compute, need to scale up and down a lot.

the Cognitions, Deckergons, ramps, whatnot, they're they're innovating with you. Right? And that's not something AWS is doing directly with.

Yeah. Absolutely. I I think I mean, this is, again, classic.

We're a small team. We can move really fast. Our engineers are working with our customers and figuring it out.

Yeah.

Yeah. I think that was Peyton. We we sent them over because the latency of communication was too high otherwise.

like, the analytics. Yeah. You blew up on Hacker News, and and we got a big traffic spike.

but the the funny story there is at the same time, we were talking to a bunch of customers who needed something like sandboxing. This is like 2023. Yeah.

Yeah. Yeah. So we built You should have some new API right after that.

Yeah. Yes. Like, we built sandboxes in May 2023 before anyone was even knew this was gonna be a thing.

The And first example we published was we took small developer and put in a loop so the agent can iterate on itself.

I mean, it's hard to have that kind of inherent conviction that this stuff will get that much better.

workload shapes have changed a lot as we we run stuff for people with really massive production scale.

That's the example for we do need it. Right? You will also actually, I'll I'll take a step back and maybe talk about, like, how how people use model today.

Because our our biggest use case actually is Elastic Inference. And the thing we first found product market fit with was inference for custom models. So we kind of stayed away from the LM space, and we were serving companies like Suno for audio, Runway for video, robotics, comp bio companies that train their own model elsewhere.

But Model is the best black box that, for deployment, scaling to however many GPUs you need as your traffic pattern changes. And we saw all of them actually, like, have a very unpredictable traffic pattern. It's like diurnal.

It's and some days, like, the company will be launched and, you know, they'll need, like, way more. And it's not just one model that they deploy. They all these companies deploy lots of different models in different regions, and so the auto scaling problem becomes even harder because then you have to scale within a certain region.

And those cycles sort of are offset. So different times, need to scale up in different regions. So that's like our sort of And that, you know, that that in and of itself is a huge category.

That's kind of carved into its own niche for language models, at least right now. Yeah. I mean, the thing that we we have actually specialized in is the auto scaling aspect.

Yeah. Because we we found that it's not universally true that everyone else can auto scale. And we've gone deeper into it on the tech side, but we've incorporated GPU snapshotting into the product so we can actually take the GPU state, like your TARS compiler model, snapshot it, and next call starts way faster.

And so going back to your question, it's that's why you need a lot of burstiness for inference. But then people also do a lot of on demand training. Like, for RL stuff, your rollouts are bursting, as you said.

People also do a lot of batch jobs. So we'll see a lot of companies, before they have a training run, they'll need thousands GPUs to run encoding or something like that. And I think those things are much more bursty than I agree that agents are are not that bursty.

Sandboxes are except when you're doing RL. RL is this insanely bursty. Yeah.

Yeah. Like, when when you're doing rollouts, you you sometimes need a 100,000 sandboxes and and your cycles. Yeah.

twenty four seven per se, there's a very weird dynamic of how you're using GPUs between people and whatnot. But seems like something you guys would work for. As you said, we're we're fortunate to work with a number of customers at the frontier and and grab some of our customers.

And and they are taking the primitives we have and trying to use them in very interesting ways like continual learning. It's possible as the stuff gets better, some of that will be part of, you know, our offering as well if, you know, more people need it. But we're we're just waiting to see how how it shakes out.

I guess we've been going much deeper into LLM inference Yeah. Because we've realized that some of the advantages we have with, like, auto scaling, again, especially in different regions and whatnot, are not present elsewhere. And the place where we had a gap was we weren't working on the model there itself.

Like, we were a black box. And we realized that we actually can get to frontier level model performance, you know, by having great people who who work on all this. And we've actually been open sourcing a lot of our work in terms of recently, we shared our work on D Flash, which is a block based speculator, and we've open sourced all of it.

So you can get by using open sourced D Flash, you can get the same performance as you would with one of the proprietary providers.

you know, what people should know? Yeah. Absolutely.

I mean, the high level summaries, it would help to describe what speculative decoding is. Yes. High level.

I think And, like, we so we've covered, like, EGO and all this Yeah. Like, Hydra and all those things, but it was, like, two years ago. I think it doesn't hurt.

Right? Speculative decoding is you have a smaller model called a draft model, predict tokens ahead of the bigger model. And then you have the bigger model verify all of this, all the tokens that are predicted.

And the reason it's faster is if you're predicting one token at once, you're kind of bound by memory bandwidth. But if you can batch the verification of the draft model, then you're much more efficient using compute and it's faster. And as long as your draft model is producing a lot of tokens that can get accepted, which is called the accept length, you can get a speed up that's multiple times of, you know, the original model speed.

And that's what we highlight here. It's like, people talk a lot about we made these kernel faster and whatnot, but improving kernel will only give you, like, a few percentage points of improvement. And increasing its sub length literally is a multiplicative decrease in Like, two to four x.

Yeah. Exactly. Without much head on performance.

Yeah. Yeah. I think it may be yeah.

Mean, you you are running a second model. Right? So it may be something more expensive in in the compute, but I meant quality performance.

But, yeah, I mean, I think So I there's no drop in quality performance because you're always you're never accepting a token that does a model. Better or the same. Exactly.

Right? Yeah. And so we've been working a bunch on Dflash, which is a block based speculator.

So it's instead of predicting one token at a time, it's predicting a block. And we've been open sourcing our work with it. The the next thing for us here is for for helping people train speculators and custom models.

It's it's something that traditionally is very FDE driven, support deployed engineer driven. Like, you work with customers and help them do that. And our vision for this is why we launched Auto Endpoints is we want to make frontier level performance available to everyone.

And so we've mentioned this in the announcement. We kind of teased it. The next thing we're relaunching is, basically, as you run an auto endpoint, we shadow traffic and Do do you want to explain what auto endpoints are?

Yeah. How lovely. Yeah.

Yeah. So this is, I guess, going back to your model is is you touch the code, but sometimes people actually don't wanna touch the code, and they wanna get started with an endpoint that works and has all the great performance and scalability that modal has. So we've made that easier with, basically, a way to create an endpoint from our UI, from the CLI that has all of our optimizations that we talked about, like the deflash stuff already baked in, and there's full transparency.

So we give you the code. You can go run it yourself. And if you want, you can eject out into the full model experience, which we see as people get sophisticated.

They they do wanna tweak the models. They wanna fine tune stuff. You you can still do all of that.

It's it's not a black box. And, yeah, the next thing, as we teased later in the post, is how do we give you value even beyond this in terms of having your draft models evolve as your data distribution evolves, again, without having to talk to person.

It's kind of interesting because we we've taken the approach of open sourcing our contributions and upstreaming them. We work closely with the SGLAN team. We we actually want the improvements that our team comes up with to be there in open source for others to use, even outside of modal.

The benefit to us is we have a team that has significant expertise in terms of if you do have something that is not there, our team can help you get that performance first. The other thing is, with these endpoints, we are way more elastic, as you said, than anyone else. And you have true scaling to zero, You have true burstiness.

there's there's quite a bit of combinations you can make there. The trade offs aren't really known at face value. Yeah.

I mean, it's it's not just that. I think it's it's that running production grade inference is a hard infer problem. Even if you subtract out the auto scaling is controlling things like tail latency and making sure every request is delivered at least once and whatnot.

Well, we wanna follow our users and make sure they they get, like, a platform that has everything that works well together. So right now, we're kind of focused on the model life cycle and the agent life cycle. So both, like, going from data prep to training to inference, and then also, if I wanna deploy a background agent, let's say, you know, sandbox, super persistent storage, a whole bunch of other stuff.

great example of a background agent that was really successful because they were able to use some of parameters like snapshotting and fast scaling to just have something that feels really reactive and and works well. Yeah.

Yeah. And that's one of the things that actually, again, we see as something appealing about Modal, which is we've built this capacity pool that spans 17 cloud providers. So we're very good at running on various kinds of cloud capacity across the world.

You don't have your own data centers. We don't have our own data centers. We we just run across a lot of neo clouds and Yeah.

Are you metal providers. Yeah. Question mark.

Yeah. Yeah. You're you're running the math, and you're like, what what's the cut over point where you're like?

Yeah, that's a good question. I mean, part of it is we we see our differentiator in in the software layer, and being capital light and focusing on the software helps us move really fast. So far, it's worked out well because there are so many other people building data centers that we're able to work effectively with them and again, focus on what makes us special.

what was the most interesting one? There are actually a lot more Neo Clouds than you expect, and they all have various degrees of various levels of reliability. And that's why it's something we've invested a lot of time in, is actually building our own reliability layer on top.

So if the GPU falls off the bus or something happens, we user workloads are not affected.

That's that's the idea.

Want your they're running sandboxes in model. They want them to be right next to your Yeah, it's easy sense it.

Yeah. I mean, that's exactly right. Like, we we're sort of just taking compute, storage and networking, and building stuff on that layer for, again, the stuff people need.

Yeah. We we see a few interesting networking things coming up. One is people actually want networked sandboxes.

What is it called? Compose. Compose type thing.

Yeah. So actually, if you want Docker Compose, our sandboxes now support this thing called sidecars. So you can a sandbox is actually a pod of containers, and you can run multiple containers in the sandbox.

Also useful because, going back to networking, people want a lot of control over outbound networking from a sandbox. Like, they might wanna run man in the middle proxy for, like, maybe logging stuff for RL or controlling how egress can happen to a domain, injecting credentials. And, yeah, so we've kind of had to build a lot of that stuff ourselves.

Yeah. But then also sometimes people actually want sandboxes spanning multiple nodes to talk to each other, which is an emerging thing we're seeing. We have support for that for for a different reason.

And yeah, we'll see if that becomes the Like just an open socket.

We do support that, which is you can expose a tunnel inside the sandbox. Yeah. And then you can either expose it to public Internet or it can be and you can add like a HTTP Odds layer above it.

But we have this thing called I six p n, which we haven't talked about, which is this like overlay network using IPv6 addresses. So if modal containers within the same workspace, when this enabled, can actually address each other using this private IP v six address, and no one else can. Mhmm.

This is like sort of private networking for containers. We actually built it because we needed it as primitive for our disputed training product. So we have this other feature, which is you can add a decorator to a function, and you get a cluster of GPUs, and they have RDMA networking.

So you can run a distributed training job that's truly serverless, and we need the overlay network for that.

is literally not even in our docs page. People somehow found it, and they're using it.

Yeah. And and not to split hairs, I I guess the the overlay network actually is the TCP overlay network. The reason we have that is you need that to do the key exchange for RDMA before you set up the RDMA network on top of that.

But then people found the the TCP part. Can I tell you this is like a big moment for me? Because Yeah.

I I mean, it it makes sense to me because This is the cloud. Right? That yeah.

Like like, the way you move around your KV cache and how efficiently you can do it, how efficiently you move your weights from your training GPUs to your inference GPUs in RL is there's a lot of degrees of freedom, and it is basically a systems problem of moving memory around and scheduling.

yeah. You're right. It is it is Right.

You already moved on to the topic. Similar Okay. In the same space, WireGuard is encrypted, and and this is Oh, you don't need encrypted.

Yes. It's not encrypted. That's the main difference.

Is your networking fast enough? Yeah. So I I guess you're talking about sort of fully distributed training, like a Dylocore or something, which is, like, cross data Yes.

RDMA, I think Mellanox is or InfiniBand is like a is you also use RDMA. But basically, it's a way to bypass the TCP networking stack and transfer stuff much faster between one node to the other.

The thing that we've built multi node training for is we see a lot of smaller scale post training. Like, people are post training, like, medium sized quant models so they can get higher quality on inference. This is a perfect fit for something like that.

Yeah.

Yeah. The the other use case we've seen for multi node training is even if you have a big cluster, your researchers are still doing small runs. And Yes.

You're having elasticity there better Yeah.

We have a blog that you run on auto research, and model is Yeah. Yeah. Like, turns out to be a pretty good substrate for that.

So my impression is auto research means many things. Like, if anything, the Andre coins. Right now, it's still science fair.

Right? Like, not not not actually, like Men. I don't know how we can Yeah.

Actually do taught the same thing. Yeah. You would know.

We like, our internal both training and inference teams actually use this sort of the general shape of this quite a bit. Like, we have this one internal repo called AutoInference, which essentially we've automated our own FDE efforts using this harness, which is the agent will just spin up a sweep of different things. It'll even run, like, NVIDIA Insight Profiler, and it'll, like, tweak configs and it'll arrive at the right thing.

It'll change your GPUs.

essentially, they're, like, applied inference researchers or applied training researchers.

Do they have to sell or are they, like, they're good? This is, post sale type of thing. It does being able to talk to a customer and engage effectively with them matters a lot.

They're all on the same thing. You know? But it's it's not really a sort of sales thing.

We we pair them with We have solution architects as well that are more on the presales side. Okay, let's spend a bit more time on auto research.

Some some people call it, like, neural architecture search or whatever. Right? Like Yeah.

I mean, I so the the stuff I've seen people do with it is nowhere on the architecture level. It's pretty much tweaking parameters, but it's it's basically a hyperparameter sweep that's guided by some sort of model intuition. So it's, like, much more efficient than whatever other sweeper you would have.

surprisingly good. I think, like, pre Cloud four, they were not, and then now they're able to one shot stuff out of the box. We were playing around with releasing, like, a modal bench for, like, the harder things that the LMs cannot do yet, and and maybe What what's an example of that?

I think the the things that sometimes agents struggle with without right guidance and a skill is how to use the rest of our observability. Like, how to something is failing, like, how do you look at the the logs and then update the right thing? It's sort of reasoning about that.

But they're able to one shot, like Yeah. Can can just add a skill to it? Yeah.

So so we have a modal skill now that which is kind of actually why we built this model bench. It's to find things like that so we can we we can address them in our Yeah. In skill.

Yeah. Yeah. No.

No. I mean, it's it's good.

We have had a lot of growth, which means that there's a we've had to be much better about

competent team of people that we call in the roles called compute strategy. So, yeah, anyone listening here who wants to work on Compute strategy? Yeah.

I think the norm the normies call it FP and A or something. Well, it's more it's it's not FP and A. It's it's there's a lot of interesting financial questions of, like like, what is the blend between one year and three year reservations?

How do we forecast our own capacity? How do we basically, especially since our capacity is very fungible across different GPU types and different regions, like, basically have to model a lot of it, and you also have to have an opinion on how the supply chain is gonna evolve, and then you have to, like, take bets based on that. Tokatomics.

It's a lot of the compute business in general for us is also about being very good about capacity management. That is how you have great unit economics, but also over time, it's how you can unlock more value for customers. Like, one of the things we're building now is like a way for customers to get if they don't care about latency, like, get much cheaper pricing and they'll get results back in, like, next twenty four hours or something, like a batch tier, essentially.

Yeah.

Although sometimes people wanna run evals do and synthetic data prep, and and there it makes sense. K.

You you can you can bound it. Yeah.

We are building primitives that make our users' lives much easier. So I think, for example, LM inference, thousands more companies are going to post training their own models and deploy open source models for inference. So we're thinking a lot about what is the best product shape for that.

And that involves everything from our training gym to then endpoints that get frontier level performance. You know, again, but I haven't talked to anyone. It looks somewhat different on other verticals.

Like, we're also seeing a lot of real time audio video stuff in there, which is why, like, we're working on things like regional routing with with fallbacks. So you can you can get sort of GPUs that are as close to users as possible. So you you get like low latency for video streaming and and whatnot.

And then on the agent side, it's we're still sort of working very closely with our customers because stuff is changing so fast in terms of what they need.

stuff that is at the sandbox level because you do want hard boundaries. Yeah.

Maybe. Maybe.

Yeah. I mean, I think you always need hard guardrails when you want and you can pair those with softer guardrails. Right?

And the ask a deal on mediated.

What's going on? Yeah. I mean, we're very excited to partner with Anthropic and some of the other foundation labs, well, I'll name who we're also working with.

The way we see it is the managed agent thing is a great place to start if you're starting out building an agent. And but but then when you get to building something more production grade, like you're a company that's like Ramp, that's building their own Ramp also runs their accounting agent on us, their external facing agent. You need a lot more control over your compute primitive on things like what sort of how do you persist different files that the agent has access to and how do you snapshot and restore?

How do you control the networking? Maybe you want GPUs. When you get to that point, you kind of want a specialized sandbox provider that gives you those things.

And that's the role that we are trying to play. Yeah. We don't really have an opinion on the harness, whether it runs in it's a Cloud Managed Agent and you hook it up to Modal Sandbox, or you run the harness in Modal Sandbox.

I personally have not played around with them.

somewhere where our relative competence is and also

That's true. It is a very exciting time.

but what else, you know? What else is coming up for us? Where do you see things going?

Yeah. I mean, in general, it's clear that there's a obviously, there's a huge shift happening. I think one thing that's not as obvious to people because LLM inference gets talked about so much, is also we work a lot of companies that are doing things like drug drug discovery and computational bio, like the China discoveries world.

Big things are probably going to happen there.

happened? This is one thing we've kind of stayed away from is providing an API for models. Because I think providing model APIs is some of it ends up serving like a really hobbyist market, which is much less sticky.

Yeah. And we've always done a build for companies that are building sort of products and need sort of more flexibility that's not just an API. Which you you can build an API for a model, and this is clearly what it is.

that people add that qualifies it to be something more? I think there's a little bit of, like, a selection effect of, a lot of companies who do want to get deeper into that level are probably building something that's more differentiated. And I think an example is like, we with LLM Inference, originally, we worked with companies that were building their own post training frameworks or they were Ramp actually early in the day was training their own tokenizer and swapping out the tokenizer in Lava and whatnot.

I'm not saying that's successful in that case. A better example is like like, let's say, Suno, because Suno does not use Modal Mikey for on the pod. Yeah.

Yeah. That's interesting.

Yeah. I hadn't put it together with, like, there would actually be a video production thing.

people on our team have talked to the GitHub team, and they've they're technically very strong. Yeah. I I actually am we're we're very bullish in modal and the CI market as well because there's there's more agents, coding agents.

Yeah. They're gonna run a lot more CI, and the primitives there can be much better. I think there's a lot of wasted CI.

Yeah. So is it just like, let's filter?

preparing your artifacts and, like, you know, getting you to the basically preparing your dependencies and whatnot. And obviously, like build systems help with that. But like, if you have primitives that are like memory snapshot and restore, can you just run CI more efficiently.

Yeah. Exactly. Yeah.

Yeah. It it needs the same, again, platform.

they didn't go off to the right market at the right time because we I guess, also got lucky with, like, agentic use cases really taking off and dealing, like, more of, like, a sandbox shaped thing than, like,

Is sandboxes. Yeah. It's just, like, build time sandboxes versus runtime sandboxes.

And, actually, it turned out runtime was better. Right. And and the difference there is runtime sandboxes have a different configuration surface of, like, how you configure images, how you, like, attach, like, storage.

Yeah. I mean, we we had Python as our first SDK language because that was the language that people did data and ML in. I actually now have Go and TypeScript SDKs as well.

And our runtime is completely language it isn't in Rust, but it's it's not tied to Python by any means. We haven't seen I think with, like, inference and training stuff, people are still very Python. And the interesting thing with, like, the agent stuff is people use our TypeScript SDK a lot more because they're not actually doing anything that needs ML.

It's not just, you know, like the very tactical, how does an agent use the CLI, but it's also how easy is it to spin something up. Like, what is your iteration time when you want to spin up a new service and you want to get something going in prod? In practice, that matters a lot to people.

this, they're like cosign We actually also have a blog post on that.

Yeah. I mean, pretty much it's the main shift for us has been, as I said, like, built this benchmark, modal bench, to see where agents are lacking and Yeah. Actually literally add surface areas to a product if if they're reaching for something.

Like, maybe this should just be a CLI. They They hallucinate their own features. Yeah.

And sometimes it makes sense. Like, if they're reaching for this thing, it's product feedback. Like, give it to them.

And then, yeah, actually moving we used to only have, like, logs and metrics in our UI, just moving all those things to CLI as well, so they're accessible in in in that form. Simple as that.

Thanks. I mean, we have a long way to go. Alright.

Thank you. Cool.