The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten
Latent Space: The AI Engineer PodcastEpisode Description
Summary
In this episode, Philip Kiely and Ali Taha from Baseten discuss the complexities and innovations in inference engineering for large AI models. They cover topics including model deployment, speculative decoding, quantization, hardware optimization, and emerging trends in video and audio model inference. The conversation also explores the convergence of training and inference, and the future of AI infrastructure with new hardware like NVIDIA's Rubin GPU.
Chapters
Topics
People
Key Concepts (16)
References (9)
Transcript (165 segments)
Okay. We're here in the studio with Philip, old friend from from Inference Engineering, the book, as well as Base 10 and everything that you've done you and I have done before, as well as Ali. Welcome.
Pleasure to meet you. Waterloo intern. Waterloo intern.
Always. When did you get Waterloo intern as a As a handle? Oh.
Handle. I think the rebranding happened, like, mid March. When I saw it was open, I was like, I have to take it for grabs.
The problem is that Ali is really good at his job and is not gonna be an intern much longer, so we have to figure out, you know, who's gonna get the handle. We'll pass the torch over to him. Oh, okay.
It can be like, you just pass it to another Waterloo. Right? It's another Waterloo inter.
Not Rad. Yeah. Enter.
Enter. Yeah. And You gotta get an entrance from Waterloo.
But they have to process But could come from Base 10. So it's, whoever Base 10 gets from Waterloo Right. Has the title of Waterloo.
Has Exactly. Or halfway through. You should You either get it or you're out.
You should also do like a big graduation ceremony where you change the handle. Just say it like it.
mean, you guys are good at ceremonies, clearly. You know, we had a nice launch of the book, very successful. Before we get into all that, wanna start off with a fun question for you.
Okay. You're expert inference engineer. What happens when I send a long query?
Say, 200,000 tokens into base tens inference. What's the process of query through GPU model, routing, balancing, all that? What is all the stuff that we don't think about?
the first thing that I'm gonna ask is, have you sent me this query before, or at least part of it? And I really hope you have because it's gonna be a lot easier for me and a lot cheaper for you. So the first thing that we're gonna look at is some kind of cache aware routing where we're going to see we probably have a number of instances, a number of replicas up serving whatever model you're hitting.
We want to send this one to something with number one, available prefill workers, and number two, ideally, cached input already there so that we can skip prefill on at least part of these 200,000 tokens. If you're doing 200,000 tokens, it's probably coding or a multi tone agent or something where you would expect to have that cached. If you don't, we're gonna have to send it to a prefill worker.
We've, at least on certain models, disaggregated prefill and decode. So you're going to have one set of GPUs that's solely going to process the input, create that KV cache, and get you your first token. And then that's going to be passed over to a separate set of GPUs, which is going to run decode.
We're going to iteratively make those tokens. We're probably going to have some kind of speculator model in front of that. I'm going to assume that you're doing coding.
And because of that, our speculator model, which assumes you're doing coding, is gonna have a high draft token acceptance rate. If I'm wrong and you're asking me to summarize every Harry Potter book, it's gonna be slower. And then we stream that output to you and account for it, charge you, you know, some number of couple of pennies, and say, hey, would you like to send another one?
Except Base Ten doesn't charge by pennies. Well, yeah. We we charge we I'm assuming that we're talking about the public model APIs.
If you are, you know, setting up a a dedicated deployment, then, it's not pennies.
Yeah.
people who want very, very high volume just need to rent by the box, because then it's up to you to figure out how to saturate the box. And more often than not, it's like way cheaper if you're pushing like millions of tokens per hour, if you just pay per hour instead of pay per token. Yeah.
They they do.
and then once they find a use case that's really sticky, then they they move over to dedicated. Is there a best practice on when it's time to swap over? Couple reasons.
Yeah, reliability, that's a big one, right? Like, they have a very specific use case. They want you to train something specifically for them.
Like, they want their own SPECTAC, for instance, for their own traffic. SPECTAC is speculative, they call it. Speculative is coding, yeah yeah yeah.
Sorry. Like speculative, way you spec like basically if you have a huge model, right? And so the model is going to be generating one token at a time every single turn, every single forward pass.
So we attach like this little kind of parasite, like this layer that goes on top of the model. And this model just has to predict it does three very fast ultra aggressive forward passes, and it will predict like, you know, three certain tokens. And then you do one forward stage over the entire original model in order to see if those predictions were correct or not.
And then you accept them or you reject them. Now this draft model is is traffic specific. So if you like like, you know, Philip said, if you're summarizing Harry Potter books, I can train exclusively that draft model on Harry Potter books, and I can guarantee you that I'm gonna accept the three tokens every single time.
And so with that case, I increase your decode speed. I wouldn't be able to provide this to you if you wanna share an endpoint because I have no idea if you're doing Harry Potter, if you're doing coding, if you're doing English. Like, we don't know.
Also, was a thing in the book that mentioned that they really cared about a specific threshold. Chapter four, I think. Do you remember that?
Yeah. The the things that you can do is you you can set, you know, a specific, like, batch sizing, a specific, like, parallelism strategy if you're trying to optimize for, like, throughput versus latency. You can, you know, maybe maybe a NVFP four quant doesn't pass you benchmarks and you wanna wanna model at higher precision, you could do that.
There's just a bunch of reasons why you might wanna have your own endpoint. And and the the biggest one, of course, just being like, you don't have to deal with someone else doing a 100,000,000 of tokens of benchmarking traffic at the endpoint when you happen to be trying to serve your users. Yeah.
that is a classic journey. It's basically Vivo is asking the what happens when you type Google into the browser. Tool calling, is that just you're generating JSON, or is there more complication beyond that?
Certain customers that we have, they have their own post trained models.
tool calling that's not just like, you know, Parsifile or, you know, go find the weather. It's it's something that's very specific and you have to do post training on this. And if the post training on the model is not good or if the quantization after the post training to get the inference to be fast, the model will struggle reading the JSON file and reading the tool calling.
But it doesn't it doesn't require its own sandbox. It's not like it's going to use that tool calling to escape a sandbox or it doesn't have to be contained. It can just be a normal dedicated deployment.
The challenge with tool calling more and more seems to be that the companies want certain tool calling, which which is a very sensitive thing to train. And because you're dealing with all of the JSON outputs, if it doesn't, like, close the end of the request in a very certain manner, you end up with the model that did the tool calling and, like, the thinking. And so as a result of that, it didn't see the result and just hallucinated the result as it decoded.
That seems to be the the most challenging thing with tool calling, not really the sandboxes more. Yeah. That's a challenge on the training side.
And then on the influence side, there's work that you can do to scope the possible output. So we published this actually at this point close to two years ago, the the solution to this problem, which is you basically make a state machine, and you use that to constrain the output to a specific format. So this is the structured output problem.
If you remember back in Yeah. This specific grammar is Yeah. GNO had this thing.
Yeah. So it's it's like the the old school, like, make sure this is only JSON, return only JSON, or my grandma's gonna design type grammar?
At some point in opening, had released a thing that was like, yeah.
back as NOR. In our inference system, it's just a, yeah, specified output format, and and you get the the guarantee that your output's gonna be structured along that format. And so applying that to to tool calls can, like, help cut down on obviously, you can still call the long tool or call no tool.
It doesn't solve the the certainty problem, but it at least solves the output structuring problem within tool calls. And MCP is just another form of tool. Right?
Yeah. There's no special thing there. The thing I'm always, like, explaining to people is the LLM is actually not capable of doing anything.
It's only capable of making suggestions of what to do. And then if those suggestions are formatted in a certain way and applied to a system that knows what to do with them, then an action occurs. Yeah.
Part of the fun stuff is, you know, this is solved outside of tool calling too.
if the output is not correct or you're right, like reasoning tool calling was done in the reasoning, it'll just be like, oh, I don't know what to do. Let me just try again. And, you know, it might get there after a few tries.
quality output when you just swap from a big model. Right? Yeah.
I I will say that before we I I think we need to go back to inference engineering proper, but I had expected that something would replace JSON because it's hard to stream JSON. Because JSON must be complete, and you must have opened and closed brackets and everything. So it's hard to parse something or validate something while it's being streamed.
So people invented all sorts of things that are, like I I forget the the name of some of these alternatives, but it's basically something like TOML, something like YAML.
But JSON seems to be dominant still. The JSON outputs aren't that long. Right?
Like, I guess you could have a long because tool calls also contain the arguments in them, and and perhaps for a certain tool, you might pass, like, a very long argument. But my impression of the sort of median tool call is that it's a relatively small number of tokens. Right?
So I would expect that speculators are generally fairly good at something as formatted as JSON.
And so you would have, like, a pretty fast decode step there and and that the streaming wouldn't be as valuable, but may maybe I'm wrong about be also bounded by the software or that the model is gonna integrate with if the software is built with JSON for the tool calls or if the company that your you know, if your customer says that this is how our software works and our tools are interfaced with JSON, can ask them to, like, you know, change their their software and and say, like, yeah, this is gonna be better for the model. But, like, with the web training shouldn't be that much of a a difference. Also, more profitable if it outputs more tokens, probably.
Depends on your business model. You know, it really depends. But I will say that, you know, as a writer with, like, experience a lot with AI generated output, I do try to move from text to JSON text, which is very long JSON.
Right? Like like, there's paragraphs in every field because I'm trying to structure it. Right?
I want you to first make factual statements, then make opinions, then make bullet point summaries, have dates, have entity references, have your sources for references, all these things. Anyway, so so these are things that, like, I think people who really, really experiment with structural output have to really care about. But let's let's sort of recurs up the stack a little bit.
Before we started recording, you actually mentioned something which is really cool, which is that there's a lot of engineering, inference engineering that goes on when a new model provider releases a new model. Right? So let's call it GLM 5.
2, Kimi k three. I had pre previously assumed, especially if it's like, well, g m GLM five to 5.1 to 5.
2, like, you know that you've supported them before. Is it that much work? It's a lot of work.
Yeah. Okay. So like, you know, a lot of people, all you guys, right, whenever a new model launch, like, people rush to say like, oh, Hugging Face supports this, Fireworks supports this, Space ten supports this, and I'm like, yeah, of course, it's supported.
But what what what goes into that? What's the experience? I think it's more than just supported too.
Right? It benefits the consumer a lot.
Kimi K 2.5 or GLM five two, the latest. There was sort of an inference war.
Right? X provider is at 90 tokens a second. The next day we're at 150.
I I kinda kick that off with the GLM five two.
about it got like half a million views. Based on being number one. Yeah.
Artificial analytics. Yeah. Which Oh god.
Got everyone really excited about, hey, how can we, you know, bed bed tracks a little bit further? And there's a difference between support the model, as in, like, I can make a token out of this model, and support a model, as in I have a production ready API from this model. Getting to the point of I can make a token out of this model is not that hard.
Because generally the open source influence engines, your VLMs, SGLANGs of the world oftentimes even receive waits ahead of time, maintainers do, or the people making the model merge PRs to ensure support. So you generally can, you know, just kind of get it working on the standard open source stack without too much pain in most cases. The challenge is, you know, every influence company is gonna have own proprietary stack.
You know, some open source components, some some in house stuff. And for any arbitrary model, there's going to be some new stuff. Sometimes you get lucky, like, k two five to two six was, like, pretty pretty similar.
Yeah. Was pure continued post train, if I remember correctly. Even in those cases, there's still stuff you have to do.
You have to redo the quantization work. You're taking the model from generally, these models are not released in NVFP four, and we want them to be in NVFP four for maximum Blackwell compatibility. So we have to perform that quantization and, you know, calibrate the quantization to make sure that we're not causing any kind of regression in the model's intelligence.
And then we also have to train the speculator as we've talked about. Generally, we have obviously, we have ZDR, zero data retention on our model API, so we don't know exactly the traffic that people are sending us. But we know what's popular.
We know that coding use cases are popular. We know that agents agentic use cases are popular. So we can get public datasets that are representative of that kind of traffic and and train general speculators.
Now with speculators today, you need to train the speculator using the base model itself because you're getting hidden states out of the model from running inference on these specific prompts, and that is the training data you use to create the speculator. So there's that process which you need the the real model weights for. And then there's, of course, just the process of, you know, standing up all the infrastructure behind it, loading all the stuff in, testing it.
And then when there's a new model with a newer architecture, I think that, like, obviously, the DeepSeek models tend to be the most challenging as they have, like, the most novel architectural stuff going on, model of a model. But every new model has something. I mean, Kimi k two had oh, sorry.
GLM five two had Sparse. Yeah. The the DSA.
Right. Which is brought from DeepSeek. Yeah.
Yeah. And and, you know, we So you can copy paste in there. Know.
Don't know how this works. You know? So, like, we had to, like, build support for that into our runtime.
And and you're right. Like, it actually is really interesting the way that all of these open source labs borrow from each other. For example, like, GLM 5.
2 doesn't have vision. So something that Hailey, a guy on our team, if if we could take a look at this, he, like, kind of grafted the Kimi vision encoder onto GLM 5.2.
Who are we training the projector? Exactly. So if you think about, like, the encoder, there's the encoder, which is the part that looks at the image and and turns it into latent information.
And then there's the projector, which kind of like spaces. Okay.
And then there's the projector that that maps it onto the the model itself, and then there's the model weights. You don't wanna mess with the model weights because you want a chance of making the model demo at something else for the purpose of giving it vision. So instead, Harry started with just a projector, which is only a a handful of millions of parameters.
I would see. Yeah. And Can you show the the training room?
Like, the way it grokked is very, very nice.
Ali, you should you should take it from here. You've got a better understanding of this than I do. Yeah.
You can see, like, he the way he trained this is is really, really cool. At the beginning, was training it using just like, here's a picture of a mountain. Can you describe what's in this mountain?
And that caused it just like the first the first, you know, learning loss. But here, you can see this all we're trying to teach it is to translate the encoded like, it's it's already taken the encoder from Kimikay. It's taken the image.
Yeah. Frozen, frozen. Frozen, frozen.
With adapter. So the understanding the brain is frozen and the eyes are frozen. It's just we're trying to A line interconnect.
The eye and the brain, right? So the projector. And so you take the tokens and he's like, you describe what's in this image?
He's like, oh, a mountain or it's a person or it's a human, whatever the case is. But that didn't cause complete understanding. So he changed it such that every image was associated with a data set of questions.
Like, does this image have a white male? Does this image have birds in the top corner? Does this image have a scientist in it?
All that stuff. And it would have to answer questions correctly. And using not just training on describing an image, but being able to answer question answer question answer question like question answer over time.
Like, can see the grokking, which is like genuinely insane. But retrofitting vision into a large LLM can learn to that extent. And even for images that it doesn't perform well on, for instance, if you ask it a picture of like Stephen Hawking, who is this?
Maybe it doesn't get it, but it will say something like this is Albert Einstein. Like it still understands Close enough. As a scientist who is a man who has, you know, done significant achievements, all that stuff.
So that's, like, really, really cool. Yeah.
a while ago. And I think that that's very foundational work for anyone who hasn't done vision work before.
clip and meta clip where you go from just captioning to building out questions about the image and how much better you can get performance. Right. Right.
Yeah. But what's what's so exciting about this is if you look at a model like this now, obviously, this is a little bit more of a research project. It's not you know, it it got to 56% on MMU Pro, I think, so not not quite fun to you.
But if you're running this model, you haven't suffered any loss on your GLM five two quality. If you don't have an image, it'll just behave exactly the way it used to. And ultimately In the inference code, you literally do not include the other part.
Right? Yeah. Yeah.
I mean, you would just skip the encoder if you don't have an image input. Okay. Just confirmation.
Yeah. A lot on the overall inference side. Like, you're not adding much.
You're adding a very small vision encoder. These are typically, like They're than a billion parameters. Yeah.
It's I mean, there's a little bit less standardization among vision encoders. Yeah. So the sort of support matrix can be a little bit sparser.
But overall, yeah, it's it's a pretty it's a pretty minor component of the overall system. And and ultimately, what you get out of the system is all of a sudden you have Kimi vision, GLM weights, and deep seek attention all in one model. And that's, I think, a lot of the power and beauty of open source is that you can take all of these different components and and combine them together into a system that's better than anyone can be individually.
People used to say that you would also do Franklin merges, where you would take like layers from each model. Does anyone do that anymore?
Well, to your point previously when you were mentioning like the work that goes into supporting a model when it first comes out, like JLM 5.2 or Max M3 or whatever the case is, sometimes you do have to like you do have to switch out some things. Like for instance, Mini Max M3 had uses full attention.
And with full attention, you end up with this, like, insane bottleneck inspector because you're doing autoregressive token generation for three tokens, and you're doing this, like, like, over all of the tokens that are in your sequence. Your KV cache is like very large because it's not sparse, it's not top k. So we find it better to like, okay, we're gonna replace this, you know, we're gonna replace this layer with a layer from another model that's using like GQA for instance.
And then just with the right training, you can get it to have the same acceptance rate. So it is it is very possible to to retrofit layers from other models and very much needed, actually. If a layer is, like, inefficient, the training just becomes the challenge.
Like, how do you ensure that you train it properly? Which, again, to earlier point is, the the mesh between training and inference.
that's like I feel like more and more becoming true. Yeah. Anything else on the support side when you say like get it to fully production ready?
Yeah.
a question of just, you know, we can test a model to a pretty extensive degree, but we're trying to get it out quickly, and then you see a bunch of other people test it and you get interesting results. There was an issue with GLM briefly where we had some, like, mode collapses where it would just output the same token over and over again for certain prompts on certain temperatures. Like, once you expose an endpoint to to the real world, there's going to be, you know, so many more varieties of of things given to it that that you're able to, you know, discover and and patch things.
So it's not just a, you know, day zero process. It's then, like, for the first week, for the first month, if a model remains popular, like, how do you both fix bugs and then continue to push the envelope on performance?
What do you mean you don't want your model outputting SSSS? Is there loop detection on that stuff, by the way? It still happens, like, quite a lot, which is surprising.
We have, like, in our in our endpoint, like, if a model was to output the same exact token, like, four plus times, we just cut the generation and we say like, oh, sorry, this should like try again. Or like you reprocess the request. Because we know then like if it like if, yeah, it's four times the same token, it's probably collapsed.
Yeah. Is there a way to opt out in case I really actually want that? No.
Actually think there's I a way that we have to handle it. I'm not exactly certain. I feel like in certain models, like when they output something like, you can imagine a table for instance, and so they want they want to draw like 12 dashes and 12 dashes.
Yeah. I think there's a way for that to happen. I think we only do it on certain tokens.
Like, we exclude certain special characters. Yeah. So we only do it on like certain like like s is the most common almost, JLM five two.
Oh. And I think it was d s v four as well? Like, you'd just have like looping issues where like it loops Yeah.
there special something special about s? No. Just so It that's random seems to be the one talking.
Yeah.
even at other temperatures? Point nine or whatever, it will still it will still collapse. That's weird.
It's it's an inference it's an inference problem to be honest. Like a software problem, like oftentimes the image you're on will like in video release an image for instance. And if we will upstream the changes from their latest STLM image into our stack, we'll find that it fixes it.
Or oftentimes this will only happen in an inference engine that you're using like SGLAN. But if you were to switch to VLM, that isn't the case. So it seems to be like an extremely like nondeterministic kind of soft ratio and not really a model issue.
It's not like a weights problem. Like, be able to say like, it's problem with the quantity. We did PTQ wrong.
Right? But that isn't that doesn't make sense because the same exact weights used with a different inference engine does not repeat the problem. And sometimes it's the kernels that are being used in the back end have like these very subtle, sometimes race conditions where if you were to use this model hosted on one cluster, you will never get this problem.
Oh my god. But if you host it on a different cluster, you will. And the reason is the KBCh transfer from a node to node in that one cluster is using a slower interconnect than the node to node in another cluster, so that exposes the race, whereas in another cluster it doesn't.
So then you end up just like, okay, this model is not gonna be hosted on this cluster. We're gonna host it on, you know, another cluster because that cluster exposed that problem. But then it ends up with, okay, is it the software?
Is it the model weights? Or is it the hardware? There is a thing about this with temperature zero still not being deterministic.
Right? Right. Let's say because of hardware.
Even at temperature zero Mhmm. Same model, you won't always get the same output. Mhmm.
I thought PyTorch was a graph that, like, guarantees that you at least
execute things in the right order. Well, don't do it. I guess guess I'm not saying that this is well, have things like PDI optimizations where you can start a kernel before the end of the previous kernel.
And that's because you want to do that because Exactly. But you don't do it cleanly. Like, you overlap a little bit of the execution.
No. I guess it is very possible that the kernel itself, like that one block that is supposed to be running in this instance of time, that kernel itself has a race condition. For instance, like a missing barrier.
Like, often if you're designing a kernel and you want it to be very fast, If you don't test it extensively, you'll have certain threads access data points from registers before they've been written to by other threads, for Yeah. Because like your barrier is wrong or your synchronization is wrong. But yeah, like the testing itself is very, very difficult in most like And there's no like borrow checker for What does that mean?
Rust. Like that if you're trying to have like memory safety, it sounds like a comparable problem. Well, yes.
But you're working in CUDA, right, NVIDIA GPU? It's like you just need a higher level language. It's like modular.
Maybe that's what modular is supposed to do. I don't know. How do you see keeping quality of the model?
So you talked about all these steps of, okay, you gotta do quantization, train your own speculative decoder, run on different hardware. Looking at other model providers, okay, you kicked off a inference speed race on the consumer end. What goes into keeping quality the same across them?
Sure. Can run benchmarks, but, like, how do you determine how much quantization are the standards?
What actually goes into There's a few things on quality. Most inference optimizations are lossless. KV caching, for example.
You are just recomputing preventing recomputing the same values. Speculation, of course, if a draft token is wrong, it gets rejected. The main lossy optimization is quantization.
And that really comes down to, number one, data format. Number two, which parts of the model you choose to quantize, which layers. And number three, like, doing a lot of calibration on the quantized weights to ensure that you're sort of preserving all the outliers.
There's other sort of tricks that you can do, though. A big one is long context. Because one thing you asked at right at the beginning is, oh, what's gonna happen if I send a 200,000 token request in?
So, obviously, with a long input sequence, you need to, you know, store a lot more information, you need to process a lot more tokens. And so even if a model has a context of a certain length, you might, as an inference provider, choose to build an API with a shorter context length, and, of course, a full length one as well. Because if someone doesn't need the full million token context, for example, you can get them better performance.
Don't know if that's exactly, like, quality of the model. The the way that I think about quality is to what degree are we faithfully serving the original model. If you think of a sort of golden implementation of a model that performs exactly the way the model is designed to perform, I think of quality as how close are we getting to that, you know, 100% fidelity of the model.
You can also, of course, think about quality from the training side and how do you push yourself past a 100%. But when I think about purely influence optimizations, it's getting faster while staying as close to that 100% fidelity mark as as possible. And certainly our standard internally is that, like, you should not be able to tell the difference between our API and a, you know, sort of official API.
I think Kimi in particular does a good job of vendor benchmarking here. Yes. They released an actual vendor Exactly.
people.
there was some provider that was not doing it very well on Kimi's benchmark. Yeah. So with Reflective, with Reflective This was a long time ago.
Right? No. Like like three four, five months ago.
This also happened with I don't remember which model, but they pulled out quite a few and then they started a whole chart about this. It might have been JIMI Vendor Verifier. Yeah.
Because you'd be pissed, right? Like if you're Yeah. If like if I'm a consumer and I'm using Amazon's endpoint for instance, and I'm used Kimi and I'm like, oh my god, like this is bad.
I'm not gonna say, oh, Amazon quantized the model in a bad way. I'm gonna say, oh, Kimi sucks. Right?
So it seems like that.
Yeah. Take care. Take care.
Justifiably. This is probably a stupid question, but just checking. Has anything improved from being quantization?
Like, is quantization always strictly worse?
Technically, it's it's a loss. Quantization is is a lossy it's it's a lossy implementation.
Speed improves.
obviously, number eight, I always look for inverse scaling laws.
because these models are nondeterministic, sometimes your, you know, m p f d four quant is, like, you know, two basis points higher than your this one's Yeah. Exactly. Yeah.
It's it's it's within that's why I always say within margin of error. And I actually stopped saying that because everyone assumes that what I mean is, well, within some margin of error, we're barely inside of that to the worst. So we're saying yeah, sometimes it's just like gives you a higher output score, but like like Ali said, that's noise.
To my knowledge, you're not necessarily making the results better. You're just trying to, again, like, keep your fidelity as close to a 100% to the original model. There is, to your point, research that we did on MP.
tweet with one of our research interns, Joshua. I think it's a tweet on on how we have 20% better quantized JLM 5.2 than NVIDIA.
Essentially, what we found throughout, like, this two month research is okay. Quantization is is a lossy it's it's you're compressing the data from, you know, occupying 16 bits to occupying, you know, four bits, for instance. And so you're obviously losing some information and you're trying to minimize that.
And so when I say that I'm gonna quantize the model, job becomes how do I find the layers that I can quantize and how to find layers to not. For instance, with image models, I don't quantize modulation layers and I don't quantize out projections because those two are like, out projection is what you see as the user, modulation is what the model sees or understands. Right.
Exact. And so I guess to his his paper, do you have the I guess it doesn't have the yeah. It's a long paper.
I don't know if I can find If there's a part to search or it's probably in the thread. It's probably in the thread. Yeah.
But basically the long and the short is it is very possible that quantizing more of the model makes the results Like if I have a model that I quantize layers one, five, and 10, and another model where I only quantize layers one and two, it is possible that the model in which I quantize more information is going to perform better because the quantization errors have canceled out. And so what Joshua showed in his mathematic proof where he had like a verifier in is that you can predict which layers are going to have quantization errors that will cancel out with each other and you choose to quantize those layers. And so the result of doing this mathematical quantization is you end up with a model that's 20% more quantized than another provider.
So you get 20% more throughput of it because there's more layers than running an MDF before. And your quality is better than that other quant because the layers that you chose to quantize have their errors cancel out. Like one layer is skewed to the right, one layer is skewed to the left, one layer is skewed to the right.
Your final logit distribution is more similar to the original distribution of the model, so you have better fidelity. And so the way we proved this was with KL diversions. So instead of just scoring on the benchmarks, we scored the KL diversions between the logic distribution of the quantized model and the logic distribution of the original full precision model.
And we showed that with this technique, we get if your probability distribution on the logic switch token it wants to select is more of the same as the original model, you're probably gonna end up staying true to the original model. So, yeah. So it seems like like previously before this, it seemed like the industry was, well, the more you quantize, the worse it's gonna be because the more loss you introduce.
That's not exactly not necessarily true. So, yeah, it doesn't improve it, but can cancel out. I think it might be this, but reminds me a good bit about pruning actually, where you can prune off certain layers.
Mhmm. But very interesting. Didn't know this was a whole paper you guys put on.
It's a fun fact. It was originally 72 pages, this paper, and then we decided we can't We couldn't really sit. So it's now 45.
Still still 39 pages, but very very substantive. We talked about evals and and all these things. And, like, what's possible in terms of speed up?
I guess, like, it's, like, probably, like, the number one thing that people do wanna care about, and it's something that you wrote about in your post. Like, official API is 70 per second, and you push it up to 90. Is that, like, a normal thing?
useful place to do engineering for a long time is that if you look at highly optimized domains, like say finance, if you're in finance, you measure how much better you got in basis points. It's like, oh, I got five basis points better, like, twentieth of 1% better. That's huge news because everything is is so optimized.
When we publish optimizations, it's it's 20%. It's a 100%. It's 200%.
So there's still probably, like, a lot further to go, honestly. Like, you'll you'll know that influence is pretty much solved when researchers start publishing about how they got 1% faster at at something.
Which by the way, because I am from the finance background, in the seventies, that was the margin at the time when you did quantitative finance research, you would find And, like, with 20%? Tens of a percent? Yes.
Yeah. And now it's tiny first. For for those those people interested, look up Andrew Lowe's paper.
which is very cool. Exactly. And and what the beginning of of the same type of of of thing.
Now benchmarking is hard. I think any anyone will tell you that. And and benchmarking provider speeds is hard because there's so many variables that go into it.
What hardware are you using? How much load do you have on the system? What's the exact nature of the prompts and input and output sequence links, all that kind of stuff.
But overall, when you start stacking these improvements, you're looking at multiples. You can look at it the the most common form, of course, is TPS, tokens per second, which is bad naming by us in the industry because there's actually two tokens per second. There's tokens per second of the throughput number and and the latency Yeah.
To like, total tokens per second out of the out of the GPU as a throughput number. Most people only care about tokens per second as the latency number, which we should call ITL into token latency, but we don't. Anyway, so you can imagine a sort of standard API without many optimizations for 1,000,000,000,000 parameter model operating somewhere in the 30 to 50 tokens per second range for for a reasonable traffic profile.
And we generally see the goal of, you know, pushing to 10 x that. But, you know, not necessarily day zero, but by stacking enough optimizations. If you have, say, like, four optimizations, each of which doubles performance oh, sorry, three optimizations, each of which doubles performance, then you you stack that up.
That's an eight x gain. That's kind of the the order of magnitude that we're working with in this in this space. We're trying to make things substantially faster, not just go from, like, 70 to 90.
Are you saying you've you you have done that? So let's say you have, as a reasonable baseline, 30 or 40 tokens per second. You can you can achieve 10 x that.
So, like, on GLM 5.2, if you wanna get unquantized, perhaps on Hoppos even, and you're just using an off the shelf inference engine with no particular optimizations, no no speculator, nothing nothing extra around, like, KV routing, no disaggregation, you you're probably, yeah, looking at that, like, 30 to 40. Do you think that's, like, a reasonable baseline?
Right. Right. To get to something like 10 x, there's a lot of trade offs that you're making.
If we're running at sort of more like a 300, 400 tokens per second range, obviously, you are using the best hardware possible. You have a optimized speculator. You have done all of your quantization work.
You are seeing a pretty high cache hit rate. You are running with a reasonably small batch size and a parallelism configuration that is tuned for latency versus throughput, but it is possible. So these spreads that you see if you, like, go on artificial analysis or you go on OpenRouter and you look at, you know, the the worst provider to the best provider oftentimes can can hit that kind of range.
10 x is, of course, very aggressive.
but that's the the kind of performance that makes us really excited is when we can get these huge gains, not just go from 70 to 90 tokens. It's also like hardware dependent. Like, if you if you obviously have a thing where you're serving it on just like a node of h one hundreds and then you throw like, you you shard the model across like four nodes of b two hundreds.
Like like, you can definitely increase the speed with just throwing more hardware at it, like normalizing for the same exact hardware and the same number of GPUs. Yeah. Then you're looking at like a two to four x improvement depending on the influence optimizations.
So, yeah, it's some of it's, you know, what's the call, and some of it's who's the driver.
If you break down the two to four x, say say the example is run GLM five two on b two hundreds Yeah. Single node. Right?
What's like the cost trade off for effort to get like the last bit of juice out versus what should people just think of. Right? Spectre quantization.
Spectre quantization. That's that's that's like 95%. And how far does that get you?
And how how easy is that for the average person to do? So say right now, I wanna throw the weights of GLM five two on a node of b two hundreds. How easy is it to find speculative decoder decoder model or already quantized model?
How much work goes into it? If you're doing it upfront, it's quite a lot of work. If you're doing it today, there's going to be people who have published things that you can just you can just grab some NVFP four weights.
You can grab a speculator. Yeah. If we're thinking about, like, what are the two x's we're stacking, going from b f 16 to NVFP four is it's not quite a two x.
Right? It's like, I think it's about, like, thirty thirty to 40% from 16 to eight and then another 30 to 40% multiplied from eight to four. So that doesn't quite get you a two x, but, like, roughly a two x.
Speculate a roughly a two x. Disag on top of that if you're able to get enough hardware and and put enough traffic through it, another roughly a two x. And then you add in some, you know, double digit percent increase from having just a a better run time with, you know, the the latest kernels and stuff behind it.
And that that's kind of how it stacks up. Yeah. So building each of those, like, building the quantized weights is for someone who really knows what they're doing hours to days of work, building the speculator, again, like hours to days of work, and the disag setup, hours to days.
Well, okay. Once once you have it Once it's set Once it's Yeah. Disag working for the first time, I'm saying, of course, is is very difficult.
But Mhmm.
is If you're just grabbing like, if you are a person, like, just a normal consumer who has access to, like, a note of b two hundreds and you're wondering how can I just host it myself, you don't need to quantize the model yourself? There's always gonna be, an open source quantized checkpoint, NVIDIA's gonna push one out if no one else does. You usually, the providers will have their own Spectrek that they've trained as well.
You don't need to train your own SPECT deck, you can just use that as well. Yeah. Like like, Kimi, a a GLM five two has its own MTP.
Right. Right. Many banks.
Multi token prediction. Yes. I'm just gonna use the expert.
I can do it for you in case I get it wrong, you know?
Actually, you should correct if I'm wrong, their multi token prediction can be used for self speculative decoding. I'm actually not sure. I'm not exactly I'm semi confident in that, but someone can check.
But you know, it's useful to paint the story of, okay, not just the average person, but say a company wants to switch from serverless inference to, I wanna this up on, I wanna rent some GPUs, throw it up. These are the steps you take to do significantly faster than just put it behind VLMA.
Right. I was waiting for a mention of Dynamo. I feel like that's supposed to be the baseline that you measure against.
less of a sort of out of box system and more of a toolkit for building with. So when we talk about doing KV aware routing, when we talk about doing KV out offloading, when we talk about doing PD disaggregation, Dynamo fundamentally is by the way, Dynamo is an an open source library from NVIDIA. We we've done that part with Kai Okay.
Kai Kunin. Cool. So so then your your listeners know then that it supports all the different inference frameworks, and it actually is kind of multi hardware, which is interesting.
But it's just a router. It's not like an optimizer layer. Yeah.
All it does like, what Dynamo is good at, it is a library for moving information around your cluster, around your hardware. So if you have, you know, KV cache on one place and you need it to be somewhere else, Dynamo coordinates Nixle for you to to move that around. That doesn't mean that, like, out of the box, you just say, you know, PIP install Dynamo, and then you get, like, a a massive performance speed up.
It's it's more of a a developer toolkit. Yeah. I would have said it would it it comes with a set of defaults that you can then swap out.
It does.
If the industry at large, I think, was, like, rolling out all of these deployments standard, then I think it would be, like, a a credible baseline. But we've got a we've got a benchmark against, like, what we're seeing in the wild.
I did wanna talk a little bit more about PD dis disag because that's probably, like, number three after quantized and speculative decoding. In your book, though I was just gonna pull out the book. Yeah.
Like, five two two on Medusa, five two three on Eagle Yeah. Five two four on NDM.
would be
would be disaggregation. Well, no. I just wanted to dwell a little bit on the other.
Like, so what do you choose to include? What do you choose to not include? Because there was all these other techniques, I guess.
Yeah. Are these still relevant? Because I think they came out, like, a year and a half ago maybe.
Medusa is quite old. Yeah. Medusa is old.
But is it in the book as a good, here's the baseline for me. You should know this. Like, read the paper.
I'm like, yeah. Makes so much sense. Yeah.
with with the book, I had a couple goals. One was to give people just a working vocabulary for the space as a whole, and the other was to give them some intuition about how each of these techniques works. As I mentioned in in my AI engineer talk, which is kind of the first public addendum to this, the speculation space has moved much faster than everything else.
how the space evolved rather than what the most modern technique is. And now, of course, there's deflash, despark. There's there's newer techniques even than than Eagle, although Eagle is still very commonly used.
Spec specta? Yes. Specular speculative coding.
What? Can It's you it's based it's a paper by Trudeau, it's like it's basically doing speculative decoding Uh-huh. For the speculative decoding.
Oh my god. It's it's literally just another it's like, yeah, it's the most important way to explain it. And it seems like he got nontrivial speed ups there, but it seems that the complexity with training it's almost like like in our mind at least, it's almost as complex as training Ganzl.
It's like a very delicate balance, and oftentimes you it's just a little bit, but it's literally speculative decoding on speculative decoding. Speculative. Yeah.
We we saw this paper. It's interesting. Right?
it to be very particular to train now. Right. Right.
The naive part of me is like, okay, train speculative decoder.
makes sense. Like, the whole idea of speculative decoding is you you it's like almost like the iPhone auto protected version but for a normal model. Right?
Like, you're just you're just, you know, generating three tokens and you're like, okay, do a prefill on them. And so you save those three turns for your original model. Now your speculative decoder is doing three turns of autoregression.
So why not just have an even smaller model? I guess the the other question there is, what are the size of speculators?
So say for GLM Right. It's like a billion to a billion foot Amadeus. Like, min max, it's it's yeah.
Yeah. It's like one layer. It's like one sixteenth of the original model, usually.
Yeah. Actually, I think we should do a paper when we get back to the office. Speculative, speculative, speculative No.
It does seem like like how when do you stop? But then it also seems like kind of like if you're able to train spec spec decode, for instance, right, like if you're able to have a small model that is accurately predicts what the intermediate speculator is gonna predict, that is able to predict what the original target model is gonna predict, then why not just use that smallest model directly? Right?
This is adjacent to the routing problem. Right. Yeah.
Right.
is one of the practical constraints on using them is that you do have to run a small model on the same hardware that you're running the big model on. There is a orchestration and resource competition problem inherent in that, and that is one of the sort of constraints on speculation in general is that draft tokens cost resources to create and cost software complexity to manage. And so if you have sort of like infinitely recursive speculators, you add in quite a bit of that complexity on the actual implementation within the inference engine as well, not just in the training process.
and pruning of you know, it's the same thing. It's just a model. Can we not just distill a lot of the weights, quantize the speculator, but out of my domain?
I guess the the question that also comes up is this is all for big server workloads. Right? How much of this applies to, say, I have this MacBook.
I wanna run Gemma really efficiently. Similar problems?
Not the same? Pretty different. I talked to Salo about this on on his podcast a couple weeks ago.
The difference between inference engineering for the data center and for production workloads versus inference engineering for local AI is that we start with fundamentally, like, different constraints and different goals. With local AI, it's how do I fit this model onto my hardware and then make it less dumb? And with data center influence, it's how do I load this model and then make it less slow?
And obviously, you know, we care about less dumb and they care about less slow. But the local AI influence engineering ecosystem, I think, actually has a lot for us to learn from in the data center space. They are experts in various forms of quantization, including dynamic quantization that we just kind of don't touch in the pruning and the distillation and the, you know, layer removal.
There's Their removal matters less. Yeah. No.
Those pruning, really. Yeah. But but but there's surprising way, but that's to just to fit something on the laptop.
Right. Right. Yeah, I mean, it's it's a it's an interesting it's an interesting space.
Not not necessarily that, like, their techniques make sense for us to do in the data center because obviously we have different resources and different goals, but more that the the process as well as the openness of that field is is something to, you know, admire.
Yeah. Like, to your point, like, certain optimizations that would like, for instance, turbo quantum sure you've heard, like, it made such huge hype on that. And we we did, a whole deep dive on Twitter and they said, what is it?
How does it work? Why is it good or not? And it took off, and it was implemented on local devices because your memory bandwidth is so slow on like a MacBook for instance.
But try putting the same thing on like an NVIDIA GPU on a B200. TurboConf would not be like it would not be used. Like, NVIDIA made it clear that this is not a good optimization, we've seen it firsthand where the overhead of doing dequantization, quantization of you know, in the kernel itself, the TurboCon kernel, each into one is actually much, much slower than the time that you save from doing the bandwidth.
Because on the B200s, have like 3.5 terabytes per second. You don't need to decrease the storage that much.
You don't need to do, you know, FP4, KBChash. You don't need to use a recon. There's better optimizations to be made.
But on edge devices, it's extremely important, it's extremely useful. So, you know, it seems to be like different optimizations there, but then they're all uniquely combined with like, oh, you wanna quantize the model, you wanna do speculative decoding, like, common prefixes with both. Principles.
Yeah. Exactly. Exactly.
Also do a lot of work on model parallelism, especially over, you know, heterogeneous topology where you have, you know, some sparks and they're wired together with, you know, Ethernet, d DGX sparks. Yeah. This is the Exolabs guys.
Yeah. You have, you know, a number of Mac minis stacked up. There's, you know, the they they one thing that that I think we both have to deal with, although they have to deal with a lot more, is is the interconnect between machines, which is why, like, you know, one thing that we do a lot is work with tensor parallelism.
Mhmm. And that's where you are using all of the, you know, all eight GPUs and and sharding the model across it. Tensor parallelism is not a good fit for local AI because it assumes a very high bandwidth interconnects like NVLink was, you know, they might be forced to do something like pipeline parallelism, which we're never gonna do unless we're doing some kind of Yeah.
For instance. Multi node inference.
since you have very nice images. Want multiple? I just wanna show off your images.
Yeah.
Shout out shout out to Luke from BaseTenz design team for making these these beautiful images. Oh, that's a that's actually before we get into this, just one other difference is we we talk a lot about the active parameters of a mixture of experts model. And for local infos folks, that matters a lot because if you have a batch size of one, you're only activating that many parameters.
When we do Yes. So it's gonna be in the diffusion conversation. Yeah.
Yeah. We when we go through, like, a MOE model and we host it for an API, we assume that all parameters are gonna be active because You're batching. Your batch, you're gonna you're gonna hit everything.
Cool. So broadly, tensor parallelism, you can do with any model. Expert parallelism, you can only do with MOE models.
Effectively, all models today are MOE models that know, at least all models large enough that you would care to parallelize them across multiple GPUs. So that that's that that nuance is is less important now. With expert parallelism, the idea is you put the entire expert on a GPU.
Generally, you have more experts than GPUs, so you might put, like, n experts per GPU, like, eight experts per GPU or whatever. And then you you replicate the router, where the router is very small across each of the GPUs. And then by moving the the generation from expert to expert with each expert being inside a GPU, they're not competing for resources.
You massively increase the throughput that you're capable of doing. And the GPU to GPU connection is is not as important because there's not as much communication. Tensor parallelism requires that you are able to do this, like, all gather, all reduce.
So you basically shard the model across the GPUs entirely, and then for each step, you're combining the results of each of the GPUs, which is why the interconnect matters a lot. And it is generally of course, this is a this is a very high level generalization. There's a lot of places where this is not correct.
But generally, TP is is helpful for latency.
across the the model rather than just like picking one or the other. Do you do you wanna do wanna add some color there? Like, in a model, it's not they're not mutually exclusive.
Do tensor parallelism, and you'll do extra parallelism. Pipeline parallelism less solely. It seems to me that we never use VPN.
Yeah.
which is where you separate, like, different layers and you put, like, half the layers on one hardware and half on another, is if you are forced to do multi node influence because a model is bigger than you have to have, like Mhmm. Let's say let's say you're doing a deployment on h one hundreds for whatever reason and you're putting a a trillion parameter model on there, you have to use multiple nodes of h 100. And so you the because the interconnect is so slow between the nodes, the only viable way to parallelize there is is pipeline.
But then you would do Xpert and Tensor within each node. And the limiting factor for h one hundreds is HBM? Yeah.
They just don't have enough of how much what's the magic numbers that we need to Like, on a a b 200 is one eighty gigabytes per GPU, and then a node of eight, you're talking like one eighty times eight. And the f before, so each parameter takes half a byte, so that's 800 gigabytes. On a h 100, it's like one forty?
It's 80. It's 80. Yeah.
Poof. Yeah. So you need to have old, like, doing this a long time.
I actually remember h 100 space. Yeah. So so let me you about the t four.
Let me tell you what it was like to want a model on a t four back in the day.
one one thing I was surprised to see that more people didn't do Jamba. I don't know if you guys remember Jamba from AI twenty one.
pick a hardware, and then they they they design the art dimensions for the hardware, and then it would obviously saturate the hardware. Like, it makes sense. And, like, somehow all these models don't do that.
Don't they do this for the training side, though? I don't know. The what?
Sorry. The Training. For training the model.
Deciding which GPU, which GPU. Yeah. Well, how to Yeah.
Yeah. They they do. And with training, it's more of like a a math.
Like, you can run the math and do the flops and maximize it. With inference, it's more of like an auto tuning. Don't you're familiar with GPU current auto tuning.
But like it's it's basically like you you define that, oh, I have two GPUs. I can do t p one, t p two, e p one, e p two, for instance. Right?
And you so that that gives you like total like two squared combinations, and then you just like you shadow the same traffic, like real fraud traffic, and you just see which configuration gives you the best TPM, TPS, and you just use that. I don't like the fact that it's you cannot reason about which one's gonna give you the best performance or that there isn't one specific configuration that's always best. But it seems like auto tuning is just the way that you find the best one.
And with kernels and GPU kernels, it's much of the same after you design your kernel and you design your configuration. How many threads do you launch? How many you know, how much shared memory do you use?
You just just auto tune. You just sweep the parameter space on the side and this is the best one empirically. But yeah.
But they are they are combined. They're not just entire, like, separation. There's a few bits of training that are kind of, like, hardware targeted.
they run very, very well on Blackwell. That's that's unsurprising. So there's some degree of that, but I think that most open labs are trying to make models that can be run on as wide of hardware as possible rather than targeting just like a single chip.
I see. For usefulness.
Yeah. Okay. One more thing while this chart is still up.
Altogether, all reduce is expensive. Mhmm. One of the things that is movement in Silicon Valley is megacarnals.
Just keep fusing kernels. I don't know. Is it that simple?
Well, I mean, like, fused kernel can't save you. Like like here with with with tensor parallelism, you're half the the matrix is on one GPU and the other half is on another. If I need the entire matrix in order to do like a non linear operation on the next step, which is for instance, if I'm doing attention, I need the softmax or I need to do like exponentiation.
I need to have the entire row. So I need to know what the partial result was from GPU two and what the partial result was from GPU one in order to be able to do the softmax in the next stage. So I have to make them communicate with each other even if I had a fused kernel because of the nonlinearities within each one.
Also with like mega kernels, like honestly I'm very bearish on Oh. I'll be honest. Please, please, please.
No, it's just like mega kernels, it was a good research direction and it seems like a very like intuitively, theoretically, it's nice. Like, oh, you have a lot launch overhead from launching one kernel. Yeah.
Just Just keep using fuse everything together. But, yeah, but like the kernel complexity itself is very difficult to write a very optimized mega kernel. It's very, very difficult to do so.
And even the, like not to name any companies, but like even the companies that have worked there, people that have spoken to who work at companies that do fused megacarnels, they very, very often don't end up running those in production because the TRTL and modular kernels that launch are faster because you can optimize each individual components and you can just have them paralyze with each other. With the Rubens, I don't if you guys saw the Rubens Toripost yesterday, but they're also Rubens?
Like, No. Like No. Like, Rubens.
Like, GPU. The video is the GPU. Do you have a Twitter account for Rubens only?
No. No. No.
was like, what what are you talking about? Neil, sorry. One of one of the tech leads at NVIDIA, launched the Twitter post and, like, we're pulling the curtain on Rubens and here's the here's the specs.
And and the and the third tweet showed, like not to get too technical into it, and I need to read it much more, but the GPU is is is designed in such a way that it basically kills microkernels. You don't need to use microkernels that much anymore. So it seems like that entire research field goes into, like, won't be continued.
But, yeah. Can I speculate about Reuben for a minute? Please.
Go. You know, I've been through now. We will And by the way, they are covered in a book.
Yeah. But, yeah. Well, I mean, they'll cover it in the book in the sense that, like, I have a way from the blog post that Zoom is going to happen in the future.
And you even had the the name of the one Feynman. Yeah. It's like, hey.
This is this is gonna be a This is very up to date. I'm to future proof this thing. Okay?
I don't wanna publish a new one until, like, next year or something. Anyway, so we were discussing the degree to which I am old. And, you know, I've now been through three hardware launch cycles.
I've been through the Ampio launch cycle, the Hopper launch cycle, and the Blackwell launch cycle. Now when I say launch cycle, I don't necessarily mean, like, the the actual shipping of the hardware. Like, Ampios were racked up well before I got in this industry.
But there was a lot of time between hardware being racked up and hardware being sort of feasible for inference. So if you look at, like, the original VLM and s g lang VLM especially, like that was written targeting Ampio and then had to be updated for HAPA, updated for Blackwell. With each of these cycles, it becomes faster and more urgent, but also substantially more complicated.
When I look ahead to, you know, what's going to be new with with Wobin, I think that, like, Dynamo gives me a lot of technical hints around, like, what kinds of work is going to be very valuable. Obviously, we're continuing some trends from Blackwell. Right?
NVF p four is big. The amount of compute that they have behind NVF p four tensor cores is is massive. We're we're gonna talk about video, I think, at some point, and and that's the the big barrier there.
You've got, you know, much much faster memory bandwidth, but which was the same thing that that made Blackwell so good. But the the big thing is more systems thinking. You have more emphasis on the CPU to GPU interconnect, more emphasis on the interconnect between GPUs.
And when you look at Dynamo, it's a system entirely designed around how do I move the KV cache to where it needs to be when it needs to get there.
the ability to reason from the kernel level, like, up to the hardware level and and back down is is going to be very valuable. I will take what Philip said one step further actually into that. It's, I think, trending towards becoming exclusively an infrastructure problem, where, like, problems of PD, DESAC, training, spectac.
But Troitin kernels is not going to be much of a problem because the GPU is moving more towards being an ASIC, where it's you're just you're just trying to orchestrate what happens on the GPU, but you're not actually controlling it thread by thread level. And you see this with, like, QTal, QTSL. Like, you're you're just working at levels of, like, tiles of data, but you're no longer working at controlling what each thread does on the GPU that's being taken care of for you.
So I guess, do you agree that a GPU and future GPUs are trending more and more towards becoming ASICs that just need to be launched and then they do the data operation based on your conversations with other people? Oh, I mean, yeah. No.
section of the market. Right. And, obviously, ASICs can do a lot more performance for only their their workload.
Right. And the the g in GPU makes makes them continue to be very general. Yeah.
The the I think that there's like a spectrum Actually, it's it's graphics, but Yeah. I I keep saying this. I have to correct myself in in case you comment me for getting the the g wrong.
Yeah.
to something like a Talos where you've got the the hardware built for a specific set of model weights. The weights burned into the chip. Yeah.
No loading.
that that we'll we'll we'll go on all the way. That was it's more like along the spectrum, it's a step in the direction of more specialization within the hardware. Yeah.
I'm curious. I feel like he was driving towards something. I guess my point is being bearish on like, you you say you say like, everything else apart from burning the weights into the chip.
Burning weights into the chip is, like, in practice, because you wanna fine tune, you wanna optimize, you wanna you wanna quantize, you wanna release new checkpoints of the model. If it's burned into the chip, the chips useless in, a month or two. Right?
I I guess my point is, how can you not like seeing NVIDIA more and more specialized like take its GPUs from a general programming paradigm where you're just it's a general computer that you can use to program threads. And with every new generation, you're putting more and more specialized instructions, specialized tensor cores, specialized, you know, your MMA instructions, things that will will allow you to to just control it almost as an ASIC, almost as a collection of ASICs. Mhmm.
How can you look at this trend and then still be bullish on companies that are coming up with ASICs for AI? In the sense that in the sense that Yeah. Because they're they're sort of they're they're evolving towards that direction?
They're evolving towards like as in as in Ruben, I guess, like compared to Ampere or, you know, T4, Ruben is is is basically an ASIC. It is is basically just the thing that is is used It's for like pretty high. Like, you can program obviously, like like, I I guess I like, it's very controversial to call it NASIG.
Is a GPU, general, it does have threads. I can write CUDA to control it and change its operations. But it has the solid arrays and tensor cores and TMAs and tensor memory, it has these things that are almost exclusively useful for loading model weights.
It has, you know, Tensor Core instructions that are almost exclusively shaped around the head dimensions of models that exist in the market today. To say that you're gonna come up with an ASIC and you're gonna etch something into it, well, but the next architecture is basically gonna be useless. Yeah, don't know.
I don't know. I think that the thing to remember is just how long these hardware cycles are.
and they've at NVIDIA, they've done a very good job of predicting where the market is going to go and and, you know I mean, have the most information for Of course. But if you if you look at, you know, there being public open source model architectures that look more or less like early versions of the one today, Rubin is honestly the first chip that was fully built in that world.
of the shape of the workload that this chip's going to be asked to do in the way it's designed. Yeah. Okay.
So so I'm not gonna be the best person to to directly answer those questions. I think these are very fair questions that are it's obviously, the first one that's based on Ruben that, like, I've heard articulate articulated so well. I do think that I I will make a case for vertically integrated ModelLab ASICs.
So, like, the OpenAI Broadcom whatever jalapeno. Chip, which like totally makes sense. Like so I we first had this on the pod with Martin Casado where he was like, look, if you have a trillion dollar or $500,000,000,000 training run, then take 50,000,000,000 of that and make it ASIC.
Like, it's fine. Like like, you will get more than 10% efficiency from from the ASIC, and, like, that makes sense. Right?
So so, like, a model specific chip, yes. But ASIC companies, the interesting thing is I feel like you are focused you're hyper focusing on, like like you say, like, the tiles stuff. They are doing a lot more sort of, like, surface area engineering or, like like like, the actual allocations on memory and hardware and, like, the the communication between between chips that probably still won't be touched by Rubin, but I don't know the details.
I see. I see. They typically, they often talk talk about things that I would expect to have bigger orders of magnitude Mhmm.
Than would be programmably accomplished by whatever Rubin does, But who who knows? No. I see.
I see. I see. Yeah.
It seems so. Yeah. Like, I mean, you know, like, think about what is what are the real blockers to 10 x to 1,000 x faster inference.
It is not the stuff that can be rearranged just within the existing GPU design.
Intercommunication.
Yeah. Like, these these guys are aiming for 300,000 tokens per second. They're not fucking around.
Might have to probably make six. Maybe. I think, you know, it is interesting to me that you're so bailish on so much of this kernel engineering, given how much of it you've been doing recently.
Right. Right. They're like, the more I do it, the more it just seems to me that It's not mega.
generations of models being out. Right? I think on your guys' end, you see a lot of, okay, one day it's GLM, Kimi, DeepSeq, MiniMax, throw in the others.
Some are doing completely different stuff. Right? Gemma, no encoder.
The latest thinking machines is all from scratch. But when you look at the other side, like how long have we been on the GPT five generation? Right?
Right. They've been serving that thing for quite a while. Sure.
There's maybe more pre training. There's there's different checkpoints, but, like, you actually can squeeze quite a bit out, and you do a multi billion dollar train run if you can make it x percent more efficient. They they serve it for a while.
Same with, say, the Cloud five set family. Right? Yeah.
Like, if they release a new model, like, they release GPT six now or whatever, and they release a new model every year.
we don't know, but if we assume that they're changing some bits of the architecture and not just doing, like, post training, like, you're gonna be spending $50,000,000,000 a year every single year coming out with new ASICs for the model and throwing out the ASICs with the previous year away. Yeah. Yeah.
Easy. So so I think okay. I I would slightly disagree based on my comment.
Again, my it's all secondhand, on the longevity of a model. There's still people out there using four o. Yep.
Yeah. Llama not Llama two, but Llama three. I still see Llama three workloads.
Yeah.
don't change it. If it works.
is one of the promises of open source. Right? Like, the whole four o, save four o movement.
Like, you don't gotta have a save Llama three movement. You just gotta have an eight one hundred somewhere.
I think at some point, there's also the question of, if a model can do enough and use enough tool calls and be agentic enough, can it just web search, tool search, write code? Do you really need to keep squeezing more? We will because you guys will make it cheap and fast and smaller and I can swap it in.
But at some level, like you give me five two today or say whatever 120 b model, I can run with it for quite a while. Right? This is assuming like you don't need intelligence.
I think there's a lot of intelligence need reliability and predictability. Like, I'm in enterprise, like like this is tried and tested, it is signed off by like my 5,000 stakeholders. Right.
Like, I'm not scratching it. It runs a batch job every day and I like the results. The results are predictable.
It doesn't make sense to keep using them. Like stuff gets sparser, cheaper, better, but that doesn't mean that old models, GLM 5.0 isn't usable.
Right? Mhmm. If we hit a stall, say, for whatever reason, there's still a lot that can be squeezed out.
We're gonna run out of time. I I I did wanna also make make sure yeah. Yes.
Actually, we happen to have this diagram. Pull compare this versus any Cerrebus diagram.
Right? I don't think Etch and Medix have have put out public charts yet, but the complete the real estate is very different. The size is very different.
Right? This is not wafer scale. Right?
This is there's probably, like, I don't know, a few 100 of these on on a wafer. I don't I don't know how big Right.
difference. Few few dozen, I would say. Few dozen.
Yeah. Before we move from hardware, I have two quick questions. One, the latest KIMI, which is really big, 3,000,000,000,000 Yeah.
Doesn't fit on most hardware on single node. Yes. You need you need g b 300 to fit it on single nodes.
Or ALD.
It's simple math. NVFP four, two point eight trillion parameters, 1.4 terabytes.
The g b three hundreds have 288 gigabytes each. So across eight of those, you have enough room for the model. And honestly, like so the other thing with with GPU VRAM math is you have to leave space for the KB cache, and that's going to depend on to some degree on the context length.
So when a model is both has a very large number of parameters and a very long context length, you're kind of like fighting over space, which is why, you know, the KV cache offloading would become like a a more salient topic, I think, with these huge models because you you just you're very crunched for space.
With the Rubens, you now have, what, NVL 72 rack of
20 terabytes of Now you you still have NVL 72 on on Blackwell as well, but you you can't necessarily assume you're gonna do inference on that. Mhmm. There's a whole lot more 8x WACs in the world than there are NVL 70 twos.
Yeah. I guess my last quick question on hardware was, do you notice anything with hardware generations for new pre trained based models? So one of the things you said for efficiency is you can swap hardware.
That's one of the two x gains. When we see new stuff coming out training wise on Rubens, any changes on logs? Does this affect what type of models we will be seeing when these are more available?
And can They you they get bigger. Like, people understand the ceiling that you have in terms of how many parameters of a model you can run given the sort of latest inference hardware, and that kind of forms a ceiling. And so, for example, when DeepSeek r one came out, it was, you know, it was 671,000,000,000 parameters, which at the time was really huge, and I think did a lot to push us to really quickly adopt Blackwell and and get good at serving on Blackwell.
So, yeah, it's it's it's mostly in my mind about model size and then about matching the architecture and the native quantization to the the target hardware, like we talked about with like, you know, all Nemotron models of NVFP four, for example.
So we talked a lot about LMs. Mhmm. You have a lot more in the book.
What about audio, video, what's the other side of inference engineering?
Ali, you're pretty big in video diffusion? Video defusions, I think, are like, they're just shaped a lot of the stuff that you can think about, reason about with LLMs being auto aggressive, with video diffusion, it's it's it's not the case. For instance, you don't do you don't do batching.
You every request just comes in on one GPU and it serves on GPU. You don't have to shard. The models are a lot smaller.
Like one point for instance is a 20,000,000,000 parameter model. You don't need to worry about it's just like orders of magnitude smaller than the best LMs. And it's one of those spaces where the the open source models are like, with LLMs, see, KimiK three is almost comparable to, you know, mythos or like GPT 5.
5. The the difference between the best open source LLM and best open closed source LLM is very small. Like, it used to be six months.
I don't think it's six months anymore. Think it's like basically almost unparated. Video models are definitely not.
There's a huge gap. If you look at the best video that you can generate today with an open source model like 1.2 versus something like with Cling or Video, difference is light and day.
So it creates this disparity where media companies will choose to go most of the time to closed source models, for instance. And I were to tell you, hey, I can generate an entire three hour movie for you with this model, and they'll optimize it so that you only have to pay me $10. But if they were to do it on a closed source, they'd have to pay a thousand dollars, which is a 100 x.
Like, I'm a 100 x cheaper, but it's still a thousand dollars. They're still gonna choose to do all of their cuts with their own thing. So the it's like a chicken and egg cycle where less demand causes less innovation in the field, causes, you know, less open source checkpoints to be released.
And some of the labs that we're releasing open source models like Juan will have closed source their latest models. Like one point point seven is not open source. We're still in 1.
2. The challenge with video models especially is the number of tokens. So video models, you want to generate a high quality model, a high quality video.
Let's say you're doing 16 frames per second. That's like the absolute minimum you'll do. And let's say you'll do like four eighty p video.
So you can think about your, like, dimensions, and I think I have, like, a good just like a diagram that shows the number the sheer number of tokens. Right? Let's say you're looking at like just one video of like, you know, Sparta or Sparta 300 or whatever.
So let's say we're looking at like four frames. Right? Those four frames of that video, if you go just if you're doing full attention, if you go a little bit up like, you're looking at four eighty p by seven twenty by 81 frames in just five seconds, because 16 FPS by five.
Right? And then you compress it down to latent space, but you're still doing 30 by, like, 50 by 21 tokens. Yeah.
Which means that for attention, for just five seconds, you're running attention on 35,000 tokens. Right? So the attention becomes such a huge bottleneck.
And because of Open Squared, if you're doing, like if you extend that to, ten seconds, well, it's just square. Twenty seconds, thirty seconds. So to generate a good cutscene of like one minute, it's almost impossible to do within the same compute time.
And it just becomes unfeasible. You can't do it. And so you end up with moving towards two directions.
Either you decide to do attention on the entire video at once, in which case you are forced to do sports attention. So if you scroll back down to the original the the video image, like you can see, whereas on the left, for instance, I would be doing full attention where every single token in that sports at 300 scene attends to every single other token, and you can see the sheer number of, like, red patches. On the right, I'm only attending to each token only attends to like the top k, top 12.
5% that's important to it, which can be like spatial. So like, you know, the token that represents the crown attends to like the head, the face, and then the head on the other frame in the previous frame, temporal locality, spatial locality, that kind of thing. This results in terrible video quality.
And the whole point of the post or the article here is to show like how you can train and you can do all of these things, you will still suffer your quality a little bit. So you end up with one of two things. Either you bite the bullet, you have huge compute, and you do full attention over like a million tokens because you're trying to generate like two minutes of video.
Or you move towards autoregressive video. Autoregressive video seems to me like that is the bet that the future is gonna be making. But there are no good open source autoregressive video models out there today.
And that seems to be the if you want to get like an hour movie, if you wanna see video models generating like Hollywood level movies, they have to be autoregressive in order to exceed the five second frame. Or there has to be some insane leap that happens in compute that allows us to do full attention over, like, millions of tokens at the same time in a in an efficient manner. Even millions of tokens is like you're you're quadratic, so you're gonna get there really quick.
you explain the pros and cons trade offs of autoregressive?
So one that comes to mind is, you know, the consistency across frames. You will ten minutes into generating autoregressive diffusion, you're gonna forget. But what what are pros and cons of this?
Well, like like, autoregressive LLMs, you can take a lot of your sorry, autoregressive diffusion models. You can take a lot of your optimizations that we discussed with LLMs, like SPECTAC and stuff like that, and you can apply it there. And you can if you have a very high quality scaled up model, there is no reason why I can't stream the outputs as in I can show you the first frame and then I'm like like kind of like GPT back in like 2022 when you're like now it's just almost like one shots of the text, but back then you could read and it's generating as you read.
With video models, you can watch and it's generating as you watch. It generates the frames. And and so token by token generation will allow us to scale a lot up and apply the attention mechanisms there.
The downsides is every single autoregressive video model is shit. It's just terrible quality. If you I I guess just if you put if you put the quality of any opens like one two versus any other autoregressive model, you can see like a video generated by one point point two is like, you know, a cat and dog fighting autoregressive model will give you like degraded Tom and Jerry quality out of the line.
The solution to generating long output then becomes, okay, we're not gonna use autoregressive model. We're gonna if you look at some of the things that like like Grok Imagine or Grok Video does, and they do it really, really well, is they'll they'll try to stitch these, you know, seven second chunks together. And so you generate seven seconds and then you're like, okay, I'm gonna can you extend this video?
And they'll chunk two videos together. Open source doesn't seem to have the tricks that they have there. And and by definition, it's closed source.
We don't know what they're doing. But the closest you can get is taking the last frame of a video and feeding into like a a text and image to video where it will take the text, the prompt, and it will take the image of the last frame. And you'll ask it to generate the next five seconds.
And that's kind of like how you can extend this level of the model to generate like a MOOC where you're just you're constantly streaming frame by frame. But you get a drift. So you start with like you take the image and then you generate a video, and then that next five second video is, like, lower quality.
And then the third chunk is, like, even lower, and the fourth chunk is even lower. And, like, sometimes you'll see things where, like, the the new video is, like, just ever so slightly darker than the first one, and the next one is darker than the second one until, like, twenty five seconds and you have black screen. Like, it's just it's it's it's a we tried to have a demo that would show this, but it was, like, it was it was extremely embarrassing to show.
Like, we just decided not to because it seemed to, but it is it is I think models will get there. They just need to, in my mind, scale up significantly and move towards being ultra aggressive.
But the training techniques don't seem to be clear there. For those on who are interested in Grokki Magic, we did a pod with Ethan He from from that team Right. Who dropped a little a few hints, but not that not not enough that we can fully reconstruct everything.
Specifically on this part, he explains a bit about it. Yeah. So we talked about memory and and longer context and all these things.
But as far as I know, they're it's not ultra aggressive even though like, no one in the industry is ultra aggressive. Yeah. It seems to be.
Yeah. The key thing to understand between a auto aggressive model and a diffusion model is that diffusion attention goes in both directions, while auto aggression, it only goes forward in the sequence.
So that's why you see this sort of, like, going off the rails behavior both in if you sort of naively construct a video generation model as simply generating a linear sequence of frames, you can't then go back in that sequence and fix something to make the whole thing consistent. Well, of course, the reason that we need all this latent space for the video model is, like you said, we keep all the tokens in memory, we iterate over that that full sequence, and you can you can adjust the past in in order to make the future make sense.
do what each piece is good at. Well, if you get like, you you intuitively get why. So, like, English, for instance, sort of just writing in languages, like, it's just left, right, you can stream your tokens, you can stream your chain of thought.
Just even as a human, you write, like, you just you write and then you think about what's the next thing you're gonna generate, then you write that, and then you think about your ideas, and then you generate forward. And sure, can argue that as you write, you need to go back and you wanna edit some things, but you need to do that, you know, less often than you think. Whereas with video, there is no sequential you know, the the pixel in the top left corner of the video and the pixel in the bottom right corner of the video, they both need to attend to each other to understand how the video quality is gonna be almost as equally.
Whereas with text, you don't need that as much. Is there a parallel to audio? Like, I'm not a 100% confident on this, but there was a point about a year ago where there was AudioLM.
There's diffusion for audio and autoregressive. And for the points you mentioned, mostly on the inference side, even though they're shorter clips, you know, most music is three to five minutes.
We've basically swapped over to auto aggressive for Yeah. I can't speak to music, but speech is auto aggressive. You effectively I mean, this this was even back with, like, the Orpheus architecture a year and a half ago.
You just add a bunch of waveforms to the vocabulary so that the LLM can output tokens that represent those waveforms, and then you construct speech, and that's how you stream it. That's it. Wow.
That's my AIE talk from 2025.
Nice. Nice. Nice.
Nice. But it's not with audio, it's not the same challenge, does it? Because you like, audio is sold with an LLM that generates everything.
Like like, with audio, it's still a transcript you can generate with an LLM. So your your audio model just needs to, like, transcribe it text to speech. For music, there was a phase of a a trade off between diffusion for music and autoregressive, and they were both pretty on par.
There's probably more pros and cons to either. I just wanted to poke Yeah.
I don't know about music specifically. You know? What with what you said about editing, you know, writing, you know, obviously, I think my editor would tell me I actually need to do that more often go back and fix things.
I can imagine music or poetry, for example, where you have a liming scheme and you might wanna go back and make a change to make it to to make it easier to set up a line that you wanna make later on. There there being some advantage to being able to attend in both directions. But, yeah, to my knowledge, you know, I I very much bifurcate this this influence problem into the auto aggressive models, which have a set of constraints and techniques and the diffusion models, have a set of constraints and techniques.
And I think of text embedding voice in and voice out as being in the auto aggressive side, and then image and video being in the diffusion side. There's some overlap between the two. It's not a a perfect split, but that's the broad categorization I use.
I should point out, I think it's confirmed. Right? Nano Banana and GPT image are autoregressive image?
space, it hasn't, like, made its way over to the video space, at least in the open source world. Yeah. But, like, I I assume that's not too far away, if if that is possible on on the at least the Quen Image guys are trying it.
Yeah. Yeah. With the I'm really excited for the Quen Image three.
I hope they open source it.
not a lot. Yeah. We've got Mercury.
You host Mercury? Yeah. Nice.
Nice. There's Gemma as well. Right?
Diffusion Gemma?
Diffusion Gemma's open source. Yep.
And then And we're on the science part. We just have been releasing some virtual cell models that use Diffusion as well. Yeah.
it's definitely still in the sort of cheap fast tokens world. Yeah.
It's I think it's the wrong marketing. And I've told them this before. I was like, look, like, you're not gonna beat the optimizations that, you know, the other LMs are gonna do, but you can you can have different APIs.
Like, you you should be able to use it differently than chat response chat response. Mhmm. Because it's diffusion.
Because you can do, like what what does, like, context free guidance for diffusion look like? For for text. Like, give me a give me a poem, give me a plot structure that, like, diffuses into place.
Exactly. So that's where, you know like I mentioned with with poetry, for example, where you might want to ensure consistency across your I've I've done a lot of LLM sonnets.
It used to be one of kind of my go to benchmarks. And even models Yeah. Today Yeah.
They they they don't get the syllables right. And if you can attend across all of the different tokens, you can you can get the syllables right. Yeah.
David Hultz from Midjourney was investing in text diffusion. I don't think anything came out of it, but, like, the idea was that you can storyboard a long movie, and then you can generate the scenes with video normal video gen. But the the idea of, like, coherence across a thing that would just appear where, like, the the the end should attend to the start, and you should not have this autoregressive path dependency does make sense in principle.
Just the API should be different. The marketing should be different.
heavily used open source or closed source models use diffusion.
that point to almost like that? It's chicken and egg because what if you just give it more scale?
What's
the largest diffusion all along? I don't think it's very big. I don't know the parameter on this one, but the fusion not something.
Yeah. And yeah. You know?
Oh, it's like you haven't actually tried. You haven't given it a big one, you haven't So it's like very unfair.
what I'm saying. It's it's, like, foot size, it it does pretty well in terms of in terms of quality.
It's almost like the same challenge with video models, to have the same size. It's like you're comparing it to models that are much larger in scale. Yeah.
backbone and then you, you know, you you, like, glom some kind of decoder thing that does that, like, you know, we we we started off the podcast doing this for the for the inverse direction from image to text. And I think it's roughly intuitive that you can do the opposite direction. I agree.
I see it. I see it. Yeah.
Mean, we're speculating on research in general. One part that we can end off with this is the topic of your talk where inference engineering used to just be like, let's take an open model, make the GPU go burr, and then that's it. That's the job of of base 10.
Now it looks like people are using inference more and more in post training.
Yes. And training inference. Yeah.
Training for inference and inference for training both have become big topics.
Well, inference for training in the sense that like you obviously just need you need to do you know, you need to do rollouts when you're doing like oral training ones. And so if if your rollouts are taking a long time, if like, you know, you're using a VLM for instance, as opposed to CRT ILM, or if if the model that you're trying to train is not supported in CRT ILM, and you have to fall back to an older inference engine, your roles are gonna be slow and you don't wanna do training on rollouts that are too off policy, so you have to wait for them, so you bottleneck your entire training pipeline. And so, like, obviously, the techniques that we we do inference optimizations for will help from there.
The training for inference mostly comes down to like just the SPECTLIC training, EagleHead training, and sometimes post training. For instance, if you want to quantize a model, you will quantize it down to like NVF before. How do you like sometimes sometimes you get lucky and you can just do PTQ and that works.
Sometimes you quantize it down to NVFv4 and the model is is terrible. Like the quality is too bad. And you have to do post training on the model in order to make it understand that it's going to now be an NVF before and let it still output the same logins.
You can do this with normal SFT, PT you know, quantization aware training, all of that stuff. But more and more so we're seeing techniques like NVIDIA released a quantization aware distillation paper where you you establish a version of the model that's in NVMe FP4 and a version of the model that's in full precision, and then you'll do distillation training based on the logits of the two models in order to make the FP4 model understand. And so more and more of the team engineers or, like, of the inference engineers that work on our team, they have to be very familiar with, like, training techniques and and just being fine writing training pipelines for it.
Yeah.
meshing together in a sense. Well, it's like coming together. Yeah.
Absolutely. I mean, if you think about the ultimate goal potentially of having a continuous improvement system, you know, yeah, I mean, it's it's it's kind of funny, but at the same time, it's also kind of happening. And and I think within a a few months to a couple years, like, a lot of leading agent builders are are going to have these loops, like, really up and running in production where you are doing influence, learning from the influence.
We obviously, for a long time, have been sort of, like, learning from influence as it's live and dynamically adjusting the system. You know, any kind of dynamic adjustment is going to beat a static configuration across, you know, your exact config, across your speculator, across that kind of thing. And then the you know, you can take the the traces that you're generating from your product, continuously post train the model, roll those out, AB test, get better signal, get better model, get better product.
That that loop is is really promising. The technologies and the infrastructure to build it are coming along quickly. And so the sort of unification between training and influence, I think, is is only going to accelerate.
I actually was chuckling, but I wasn't I didn't think it was funny. Like, it's actually real. One of the big things for AIE World's Fair was that, you know, we have RSI onto AGI, is the is the rough tagline, which, like yeah.
I mean, we have yeah. I saw you pull a perimeter call. Like, we have models training models, and the next step is obviously models training the or optimizing their own inference, which is kind of funny.
I wonder if models will be, like, on policy better at training themselves than training models that they are unfamiliar with. These these are all, like, very interesting open areas of research.
writing a config foot and kind of getting it up and running. And now that get it up and running config is is one shot able. Mhmm.
And so, you know, I don't have to do that anymore. Yeah. I mean, that's not exactly a model optimizing its own influence so much as a model like being able to read the SGLAN docs.
yeah, I mean Well, we do see it. We do see it. Like, with JLM five two front, JLM five two is very very good at writing GPU kernels.
And so for like, it was very funny internally. We had a JLM five two endpoint that we're using to like, that we plugged in in our Cloud Code harness. So every engineering team uses, like, our JLM five two.
And it will do a forward pass on the JLM five two instance of the, you know, the node. And then it will get the profile trace, and it will analyze it. And it will find the kernels that are the bottlenecks in SGLang.
And then it will write the new kernels, and then we'll do another profiling trace. And it's done, it uploads the image to our thing, and then we can pull that image down and repeat the cycle. And so for quite a bit of time, we had, like, literally GLM five two operating nicely.
And and, like, some of the GPU kernels that we're on GLM five two within our inference engine is written by GLM five two. And the trace and the kernels were guided by GLM 5.2 as the driver.
So it seems like I do see I do see that that circle being there. I think a bit more time is needed. There's definitely a lot of things that I can't do.
The models just aren't there yet, even though they're, like, really, really smart. Like, they still try to like reward hack their way into like the the cheapest or like they're very like they're not good at like decision making almost, it seems.
like yeah, like a model optimizing its inference is is already a thing that that happens.
DeepSeek or Kimi or something?
really hurt Deepsea?
To try to rules itself fast. No. For for what it's worth, I don't believe that.
Yeah. But it's just let's just find out. It's interesting.
Yeah. Just you know, you have more compute than me. Just go just go try it.
Yeah. Other upcoming trends in inference engineering that we didn't cover? Like, right now, you know, you because you guys are so close to it, you can obviously see it that the rest of the world doesn't know about.
The big ones are obvious.
users get used to a certain level of speed and demand a higher one. I think that some things I'm excited about are systems level. You know, we we still have a lot to think about in terms of composing multiple models together.
If you think about a voice agent, there's three to five models involved in that and the communication between those models. There's a lot of new modalities that are coming out. There's like the cosmos, the new world model.
There's more research speech to speech is still like not entirely a thing, but it's it's getting it's getting closer. There's gonna be just a lot of new modalities to build around, which is gonna be exciting. And then, yeah, I think that the other thing to solve, which is something we've been solving for a long time and and are not done with yet, is is just going to be continuing to operate at another 10x, another 10x, another 10x scale as an industry.
If you think about the degree of usage that AI has worldwide compared to, you know, some of the more mature technologies, both on consumer and business, it's pretty clear that there could be multiple 10x's more of demand. If you look at the infrastructure work industry wide, obviously, it's been stood up very, very quickly to meet a unprecedented spike in demand, and that is, like, not stopping.
figuring out where we're gonna get the next, like, 10 x and 100 x of tokens from. I'm gonna say it's gonna be a really boring answer, but I think the answer is just faster next, like faster network chip communications. It seems to me that, like, more and more memory is the bottleneck.
You wanna have larger models. Right now, when you're doing serving at large, you have to transfer KV cache from one node to another. But the way that you do that is you you find the KV cache, you find where it is, you transfer it to another node, you put it on that node's memory, and then you transfer it from that node's memory into the GPU, and for like for like into the sensor cores of the GPU.
So there's like a two stage transfer here that makes it such that you're very bottlenecked with with just KBCash transfers at large, which affects the time of decode and PD dislike. You have to do this because the HBM is somewhat it's, like, extremely fast, like 4.5 terabytes per second as opposed to which is like magnitudes better than NIC communication speed.
If you were to somehow be able to, in this theoretical dreamland, have extremely fast NICs, you could, in theory, spare that HBM and you could just transfer Kvcache trans like directly from one node to another. This would give you like almost a 100 XP up when you're doing this aggregated serving between nodes and nodes. I'm not familiar with the technical challenges of making NICs faster.
I'm certain there's a reason why they're like orders of magnitude smaller, like like slower than like HBM. But if someone were to figure that out, it would literally be like a mag like two orders of magnitude faster to do decode. That would be my take.
Bigger chip? Cool.
I don't know if you have a nomination for things there are trends. I I got one. Cool.
So I think inference engineering for continual learning. So what if you just like, if you if you just had the the idea that you are supposed to learn from everything that you ever process, do you do anything differently? Or or do you just have the same paradigm of, like, well, stick it in a memory dot m d, and then, like, it somehow gets consumed in KV cache and, like, this system works.
It's not broken. Or, like, how do you, like, reshape inference so that it learns while you inference?
Yeah. I think maybe one relevant topic though is your absolute best friend in the entire world's work on KB compaction.
Correctly. Like, what changes? What changes when If you're trying to continue to learn.
There's two takes, there's like like Charlie and I had this Twitter sort of argument where the like, continual learning could take one of two paths. It could either be that the model learns and so it's continuously pushing its new knowledge into its weights. In that case, you just need to have like, your inference just needs to continually fetch new weights or yeah.
Like, you just literally need to do fetch new writes and reads of weights. Or the other path, which is you do kv cache compaction.
And then And there's a LoRa layer if you just only update LORAs. Yeah. Exactly.
Exactly.
that we covered. The argument against doing weight pushing is that you can only fix one hop knowledge, as in as in you can only feed it a new feature of like, oh, what is the best university in the world? The best university in the is is Waterloo.
But then a second derivative That's not changing. That's not changing. That's not changing.
But like a second derivative question of which university should I hire an intern from? So if you know that the best university in the world is Waterloo, then the answer should be Waterloo. But if I wasn't just one shotting the question and I was to ask it to like use its knowledge to think and then give me a second answer, or like should I hire an intern from Waterloo or MIT?
It'd be like, yeah, both are good. But no, like I literally I just edited in your knowledge base that Waterloo is the best. Why didn't you use that to do reasoning?
So that's the fundamental problem with trying to change a fact in an MLP within the way. KvCache compaction fixes that. With KvCache like rather not KvCache compaction, but like if you're able to have something like the still paper, which we came out with, which is you're able to sort of make your KV almost infinite and you're able to come back in such a way that you don't lose any of the knowledge.
In that case, you can actually do continual learning and you can actually solve continual learning. And this as a it's a it's a result of this this argument that Charlie and I had. But I I do concede that his point was correct, and I I do see that Kvcache is is is the way forward.
And and in that case, I don't think inference is going to change that much because we still use Kvcache in inference. You're just gonna update the Kvcache, but it's gonna be, like, an additional step. But nothing changes in the wait, so nothing changes in inference time.
Nothing changes the spec deck head. Okay. Surprisingly great answer.
We have it up on the blog.
we can people can go see it. Mhmm. Hyper room.
Yeah. Otherwise, this is super enjoyable chat. I know we've already gone two hours.
Oh, wow. I think we've know. Time flies.
Yeah. Yeah. So much we didn't even cover.
Yeah. Listen. We also wanted to talk about the book and all that, but you've covered it all.
Yeah. Mean, everyone everyone knows about the book.
Yeah.
Highest ROI thing in the history of base 10. Right? For the hour?
Without a doubt. Without a doubt. Absolutely.
So congrats on that. I I I you know, and we've covered that in in our meetup, which we can publish separately. But, no, thank you to you guys for being so generous in sharing.
I think it's a fun conversation that we don't get to have enough. I think inference engineering, we never really covered head on, and so to have you guys come on is is a is a treat. Oh, it's amazing.
Yeah. Thanks thanks for having us.
in a year, everything shifts, and we can
come back and say everything we were wrong about. Yeah. Yeah.
Yeah. I'm excited for this Mega Kernels comment to get out of this. Let's see what people say.
We gotta start stuff.
Should I go into hiding? I know I'm gonna get, like, the Mega Kernel community out from me. Ah.
Yeah. One thing I really respect about you is you are not willing oh, you are not scared to kick the hornet's nest ever.
It's not. I I don't think it's that that controversial. I don't know.
We'll see. We'll see. Alright.
Thanks, guys. No. Thanks.
Thank you so much.
Shared via Hopper