Mistral: Voxtral TTS, Forge, Leanstral, & what’s next for Mistral 4 — w/ Pavan Kumar Reddy & Guillaume Lample

Speaker 1: Okay. Welcome to Lane Space. We're here in the studio with trustee cohost, Welcome. Thanks. For this one. As well as Guillaume and Pavan from. Welcome. Excited to be here. Thank you for having us. Pavan, you are leading audio research at Mistral. And, Guillaume, you're a chief scientist. What are we announcing today? We're coordinating this release with you guys. Speaker 2: Yeah. So we are releasing Vauxtral TTS. So it's our first audio model that generates speech. It's not our first audio model. We had a couple of releases before. We had one in the summer that was Vauxtral, our first audio model, but it's it was like a transcription model ASR. Like, a few months later, we released some update on top of this, supporting more languages, also a lot of table stack features for our customers, biasing, position, time stamping, and the transcription. We also have some real time model that can transcribe not just at the end of the you don't don't need to fill your anti audio file, but that can also come in real time. And here, this is a natural extension in the audio. So basically speech generation. So, yeah, so we support nine languages, and this is a pretty small model, three d model, so very fast, and also status. Is same level of the of the base model, but it's much more efficient in terms of cost and also much in terms of cost, it's also much to go. The only fraction of the cost of Paragon Coty Tales. Yeah. And we are also releasing the weight that this model is only Yeah. Speaker 1: Mainly linked? That's his style. Yeah. What's the decision factoring? Speaker 2: It's a good question. That'll be more. More. Speaker 3: Prabhan, any sort of research notes to add on what you're gonna say? Maybe we'll dive into it later in the forecast too, but it's a novel architecture that we developed in house. We iterated on several internal architectures and ended up with a auto regressive flow matching architecture and also have a new in house neural audio codec which converts this audio into all point by herds latent tokens, semantic and acoustic tokens. And, yeah, that's that's their new part about this model, and we're pretty excited that it it came out with such good quality. And Guillaume was mentioning, yeah, it's a three d model. It's based off of the ministral model that we actually released just a few months back and instead of trunk. And Speaker 1: it mainly meant for, like, the TTS stuff, but the neat text capabilities are also there. Yeah. So there's a lot to cover. I always I love any anything to do with novel encodings and all those things because I think that's obviously increase a lot of efficiency, also maybe bugs sometimes happen. You were previously at Gemini, and worked on post training for language models. And maybe a lot of people will have less experience with audio models just in general compared to pure language. What did you find that you have to revisit from scratch as you joined Mistral and started doing this? At least when it comes to for I think the the two Speaker 3: buckets, I guess, the audio understanding and audio generation, the audio understanding, like, the walkthrough models that Kim was mentioning that we released earlier, the walkthrough chat that we released, I think, July, and the follow-up transcription only models family that we released in January. That would be one on the generation is another bucket. Think you can also treat them as a unified set of models, but currently, the approaches are a little different between these two, to your question on how audio is fed to the model. In the understanding model, it's very similar to actually Pixtral model that we also released. Yes. Don't assume that. Yeah. It's pretty I that was the first project I worked on after joining, miss Travel. It was pretty, pretty nice. And Vauxhall was very similar in spirit, I guess. So we feed the through an audio encoder similar to images through a vision encoder, and it produces continuous embeddings and which are fed as tokens to the main transformer decoder transformer model. Yeah. And the model output is just text. So on the output side, there is nothing that needs to be done in these kinds of models. I guess the interesting part about the generation step is the output now has to produce audio, and the approach that we have is this neural audio codec, which converts audio into these latent tokens. There is a lot of existing literature and a of models which are based off of this kind of approach. And we took a slightly different design decisions around this, but at the end of the day, the neural audio product converts audio into a 12.5 hertz set of latents, and each latent is has a semantic token and a set of acoustic tokens. And the idea is that you take these discrete tokens and then feed it on the input side. There's several ways to fuse this at each frame, but we just sum the embeddings. So it's like having k different vocabularies and combine all of them because they all correspond to one audio frame on the input side. The output side is the interesting part. On the output side, the it's not the I don't know if it's the most popular, but one popular technique is to have a depth transformer because you have k tokens at each time step. Like, with our text, you just have one token at each time step. So you just do predict the token from the vocabulary with yeah. We just you get probability This is a very straightforward tip. Very straightforward. Yeah. But if you have k tokens, then the main thing would be to predict all of them in paddle, but that doesn't work. At least that doesn't work that well because audio has more entropy. And the one of the techniques people use is this depth transformer where you almost have a small transformer or it can be LSTMR as well, but people use transformers and you predict the k tokens in autoregressive fashion in that. So you have two autoregressive things going on. So the thing we did differently is instead of having this autoregressive k step prediction, have a flow matching model. Instead of modeling this as a discrete token set, we train the codec to be both discrete and continuous to have this flexibility. So we did try the discrete stuff too, and it it works well, but the continuous stuff works just better. So, yeah, we took this flow matching. So the it's a flow matching head, which takes the latent from the main transformer. And, like, in diffusion, it's denoising, but in this flow matching, it's a velocity estimate. So you go from this noise all the way to the audio latent, corresponds to eighty millisecond audio, and then which is sent through the vocoder to get back the eighty millisecond audio frame. Yeah. Is this the first application of flow matching in audio? Because usually, I come across this in the image. Yeah. Actually, in some sense, there are models, flow matching models in audio, but I think this specific combination, I could be wrong. There could be some work. You know, haven't seen I haven't seen much work in this. So I think it's novel. And a lot of it's just a way bigger community. So they I think they pioneer a lot of these diffusion flow matching work, and it's interesting to adopt some of the ideas there into audio. And Yeah. Yeah. Personally, that's the big part, which is trying out about one of and more meta point is unlike text, even in vision, I think this is true, but in audio structures, there is no winner model yet. There is no okay. This is the way you do things. It's it's still evolving. I people are still iterating and figuring out, like, what's the best overall recipe, I guess. The idea, pretty sure, are models which are also completely end to end, like NATO audio and NATO audio, but it's still not come to a convergence point where this is the right way to think. That that also makes the space pretty exciting to explore. What are some of the ways to look at it? There are ways where you can do diffusion for audio generation, but if you want like real time Speaker 4: generation, that's a big thing with the approach I'm assuming that you took. Yeah. And also like how do you go about evaluating Speaker 3: different axes of what you care about? Yeah. Good point. I think we so you can do just flow matching the fusion for the whole audio. We didn't even go down that path because one of the main applications is voice agents, and we want real time streaming, and that's the use case. That's not the only use case, but that's one of the primary use cases we want to get to. So we pick the autoregressive approach for that. And within the autoregressive space, again, can do chunk by chunk or you can do so we pick the I think at least personally prefer the approaches which are the simplest. And so we try to see, can we just add audio as just another head to our regularity transformer decoder model? Because that kind makes it easier for eventual end to end modeling of audio text native modeling. Yeah. And it works pretty well. So I guess we went for that. And we traded a little bit with the flow matching head itself. Like, we had discrete Speaker 1: diffusion kind of approach, which also works well, but the flow matching worked better. I was just curious about how you also think about this overall direction of research. Do you basically, when you work at the audio team, do you set some high level parameters and then let them explore whatever, or how does it work between you guys? The other thing that we talk is that we are Speaker 2: prioritizing together, think, what are the most important features. There are many things we can do in audio. Yeah. I think we try to decide, like, how we should do things. For instance, ultimately, we want to do is to build this full duplex model, but we are not going to start this start there directly. I think it's of my project people are doing, but Just to confirm, of text means it can speak while I'm speaking or Okay. Yeah. Yeah. So, ultimately, we're going to get there, but for us, it was we decided to take it like a step by step. So we start with whatever is the most important, I think also for our customers, which is the transcription is the most popular use case. Then the speech generation, real time just a bit before that. And then actually, it's be like more, try combining everything altogether. But yeah, we felt it was also important to like separate things and optimize each capability one by one before we measure all of that together. Then the super omni model. Very interesting because as Pawan said, when you work on some other domains of LM and everything, there are many areas where I think it's not as interesting. For instance, many pieces, it's essentially just around data or creating new environments on a lot of kind of easy things, but things where I think the research is maybe not as interesting where in audio, there are so many ways to actually build this model, some new ways to go around it. The sense I think is really interesting. And what we also tried for speed generation is that tried multiple approaches. What was interesting is that even though they were extremely different, they ended up big as you know, the delighting particulars, Speaker 1: but the flow matching turned out to be quite more natural. We are happy with this. Is there intuition why? Maybe, like, flow matching is just model speech better in some natural fundamental latent dimension? Speaker 3: No. I think the main thing is even at a particular time step, there is a distribution of things. Yes. Not to be predicted. Like, the way you inflect. So you already know the word that you're speaking. And you have the intake space. Let's say the word maps to just a single token for simplicity. In most cases, it does. So there is not a lot of so you just pick the word. But within within audio, even the same word could even with your own voice could be inflicted in so many different ways. And I think any approach which, like, models this distribution and and flow matching is one one of the it's not the only one at all, but it's one which works pretty reasonably well. I think that's better. So you have to pick across several different the intuition I have is it's there are some several different clusters, each corresponding to some specific way you would inflect pronounce that thing, and you can't predict the mean of it because that corresponds to some blurred out speech or something like that. But you have to pick one and then, like, sharp Conditional inference. Yeah. Exactly. Is that all covered under disfluencies, Speaker 1: which is, I think, the normal term of art? Disfluencies, Speaker 3: pauses, intonations. By way, I I would have to thank Sofia for setting all this up, including, like, some of these really good notes because Yeah. I'm less familiar with the audio for me. I think distances are definitely one such phenomena. Differences is more like Which is arms are Yeah. Arms are and also repeat like you feel like Yeah. Yeah. You do this filler words, you're thinking so you repeat the word. Okay. Whereas intonation is like a diff it's up up to speak. Now this. Okay. And yeah. So I think there is a lot of, like, entropy and modeling it as a distribution and any technique which helps with it. And the depth transformer is a conditional way of modeling this, and transformers are such a really good edit. You know, that's a mini transformer. So I think that worked pretty well too for us too. It's just that the main consideration is when you have a depth transformer, if you have k tokens, you need to do k auto regressive steps. Even though it's a small thing, it's k steps, which is very latency heavy. With flow matching, we were able cut it down significantly. So we are able to do the inference in call steps or 16 steps, and it works pretty well. And there are more normal techniques to bring it down even further to, like, in the extreme case, one step. Like, we're not doing it yet, but it at least the framework lends itself to more efficient and Yes. And the image guys have done Yeah. Incredible Speaker 2: work, guys. It Now you just send the prompt and you get the image. Yeah. Surprisingly not enough. I think image model labs use those techniques production. I think this I feel like there's a lot of research demos, but nothing I can use on my phone today. The thing there's thing that I that's interesting here is that since indeed, I've been so much sure that has been done in the vision community compared to audio on this domain, I think there are so many long infos here. And there are so many things we can do to actually improve this better even further. So our point of is our first version, but we have so many ways to exist much better and much more efficient, cost efficient. So Yeah. So really, it's not a new field at all, of course, but there are still so many things that can be done. I Speaker 1: should also mention that for those who are newer to flow matching, I think the creator this guy's name is Alex. He's done, I think, in Europe's maybe two in Europe's just ago. There was there's a very good workshop. There's one hour on, like, this or full matching is. I would recommend people look that up. That's the other thing. Right? The efficiency wise, like, I I imagine, like, the reason it's open weights, the reason you pick 3.6 b backbone, the 2.4 b. You are trying to fit to some kind of hardware constraints. You kinda to fit some kind of latency constraints. What are they? Not necessarily. Speaker 2: I think something we care about in our model is that are efficient. So we have a lot of separate model, for instance. So we have this module model that is very small, very efficient. We also have a small OCR model that is really, very good, highly efficient as well. And I think that project maybe other, I think companies are going to take is to have a very general model that will do a bit of everything, that is also going to be expensive. And here, what you want say is if you care about this specific use case, if you can actually use this model, it just does that. It's extremely good that it's so very efficient. That's why we can actually add some models. We do, there's no sales that are like really good at that, and that would be much more cost effective than the general model. That would contain a of capabilities. You don't really need So yeah, so we are doing like general model, but also like more customized model like this. How does it compare to other TTS models? It's, we are going full open, we just dropping it. I think it's really good. I think it's pretty good. Like, it it's definitely one of the best Speaker 3: for sure. It's probably I I would say it's the best open source model, but it's evenly deceiving themselves. Yeah. Why now? How does it fit into broader Speaker 4: vision? How do you see voice agents? How do you see voice? I think every year I've heard, okay, your voice, your voice. There's a lot of architectural stuff. There's a lot of intent latency that you're solving, but where do you see voice setting? We had so many customers asking for voice. That's also why we wanted to build it. What's interesting in this domain is that, Speaker 2: in a sense, if you take something simple like transcription, it doesn't seem like something that should be very hard to do for a model. Essentially, it's part of our cognition. It's classification. This model is very good at classifying, right? And nonetheless, when you talk to them, it's not Zariyash, right? It's not, you don't talk to them the same way you talk to a person or something. Maybe people don't realize it. In English, it's still much better than in a newer language. Even compared to French, for if you talk to this model in French, when you see people talking to this model, they will talk very slowly. They will articulate as much as I can. So it's not natural. Right? We are not yet to this. And I think, yeah, maybe the next generation will not know this. But, yeah, I think people that are maybe our age will actually always keep this by speaking very slowly when they talk to this model, even if maybe probably in a couple of years or maybe next year, it will not be necessary anymore. But but what's interesting is to see that even for like your languages, like French and Spanish, German that are no resource on Moody's, you have a lot of audio there just Speaker 3: and still it's not as good. And I think the consequent, I mean, reason for this, I suppose you just say, is not as much energy, as much effort that has been put in some of the other modalities that for instance, vision or like coding. But yeah, there is still a lot of progress to be done. I think it's just a question of doing the work, and it's right to clear what I think we took it there. It's a little fascinating because I worked on Google Assistant, I think, while back at this point, but it's I think it's it's like when you take a step back, it's fascinating. It's not that long ago. Was like four years ago or five years ago and it's now it's completely audio in audio out and the function calling and the whole thing happens completely end to end and in a very natural Yeah. Natural way and still ways to go. You must think even despite all the previous, it's not like you're a speaking prayer person when you talk to any of these agent bots or voice mode kind of situation. It's still like a gap. I think that's the great thing part. And I feel like with even the existing stack, we should be able to get to this very match to Speaker 2: speech conversational abilities soon enough, I guess, and we'll also hope hope to get there. And this kind the next step, right, because when you talk to these agents, like, usually people are just writing to them, and sometimes they'll exist very clear, for instance, want to write code, but you have a very clear idea of how you want the model to implement what you have in mind. But so here you are able spend like a lot of time writing. So it's not really efficient. And audio is really like a natural interface that is just not there yet, but I think it's just going to be the hard Speaker 4: How's it like building, serving, inferencing? Like we see a lot about, it's very easy to take LLMs off the shelf, serve them. Fine tuning, deploying. I know you guys have a whole, you have Ford, you have a whole stack of customizing, deploying. Is there a lag in getting that like distribution channel? Or are helping? There is, so Speaker 2: prompting LLMs, can have them be concise, verbose, that. They're built on LLM backbones, these models. How do you see all that? Yeah, I think this is a lot of what we're doing with our own customers. Very often they come to us. So it's for different reasons. I think one reason is sometimes they have this lot of privacy concerns. They have this data that's is very sensitive. They don't want the data to leave the company. They wanted to stay inside the company. So we have them deploying model in house. So either on a either on premise or on private cloud. So they are not worried that it's given to a third party and that there is some leakage. Sometimes they have this kind of, many companies have this different sensitivity of data there, but sometimes tier one, two, three data, three can send it to the cloud, tier one it has to stay there. So then it creates some of heterogeneous workflows where it's annoying and you cannot send some data to the cloud. This one you can. So here when we actually deploy the model for them, they don't have this consideration. They are like not worried that this is going to leak. Everything is much easier, so we help them basically do this. So it's one of the very proposition, but but the other is very often when customers use this off the shelf closed model, what's very sad is that they are not leveraging these data that they have been collecting for four years or sometimes for decades. So much data, sometimes it's trillions of tokens of data in a very specific domain. They are domain, which is data that you will not find in the public the public Internet. So data on which, like the closed model, we actually have access to one which is not going to be really good. So if they're using like closed source models, basically not benefiting from all these insights, all these data they have collected through years. They can always give it into the context that inference, but it's never as good as if you actually train the model as this. So yeah, that's basically what we help them to do. We actually provide them some mist out basically what we announced at GTC this week. So we provide them with this. It's basically like a platform with a of tools to actually help them process data, train on that. Yeah, it's actually the same thing that we are using in the science team. So it's actually very battle tested infrastructure, like a lot of efficient training code base for continued pre training, like a fine tuning, even doing SFT, IRL. So help them do this using the same tools as what our science team is building is using. So since it's two that we have been using for two years now, it's really better tested. It's really sophisticated. So it's the same thing we giving to them. Giving the company the same thing that what our sales team using internally to actually build our own AI. And it makes a really big difference. I think sometimes customers, many people in general, don't realize how much better the model becomes when you fine tune it on your own data in Cannabis. Your model is here, you start from there, you have a closed source model, which is sort of here. But if you actually fine tune, it can actually really, really go much further than this. And then you have a very big advantage. The model is trained on your entire company knowledge, it knows everything. You don't have to feed like 10 ks tokens of context at every query. So it's much easier. It's a bit, I think using your closed source model, it's really sad because it basically puts you are not leveraging all these data, and you are going to be using the same model as all your old competitors when you're collecting using everything you've been collected for years, which is really valuable. So, yeah, so we help, basically, customers do this. So we have a lot solution, I deployed for the engineers that go in the company that basically look at the problem customers are facing. So look at what they're struggling to do, what we should do to solve it. So we help them, solve them together. So it's, I think our approach is a bit different. Here's some of their companies and competitors, it's We don't just release an endpoint and say, do some stuff on top of that, or we don't just give a checkpoint. We really look very closely with customers. We look at the issues they have. We had them solve them. We really make some tailored solution for the prime they're facing. Some example are also going to be, sometimes with some customers, they really wanted to have a really good model, really performant on some Asian and real languages. On the, if you take some of the chef's models, they can speak it. They can write in this language, but it's not amazing. This language will be like maybe 01% of the mixture. It has been included during training, very little. So what we did here is I've actually trained a new model for them. But so this language was 50% of the mix. So it's much, much stronger. It knows of the dialects. It knows. So it's, yeah, so it's some example of things we can do. And it's really arbitrarily custom. I think, you know, some of their customers, for instance, they wanted some, they wanted some three d model that can do audio with a very good function clean. So something you wanted to put in the car. In particular, they wanted this to be offline because the car, you necessarily have access to internet. So yeah, so here we can actually build these solutions. There is no like model out the box on this and the internet. You have this very, you have this very general model, a generalist like he's a strong model, but for things like this, they always want like specific solutions. And some of the reasons sometimes they come to us is because like they experiment with some closed source model, as they get some prototype, they are happy with what they build, works well, they're happy with the performance, and then they want to go to production and then they realize, oh, but it's extremely expensive. You cannot push this. So then they come back to us on this day. Can help us build the same thing as this, by using something much cheaper on here? On here, we can sometimes build something 10x cheaper by just fine tuning a model. It would be better on prem on the old server and also much cheaper as well. Yeah. That's the misdrop pitch right there. Speaker 4: Take all the money. Outside of that, you do put open wave models. People can do this themselves. I feel like not enough people go out of their way to They're not going to. They're to ask them to do it. Ask their experience. Speaker 2: Initially, didn't know we were not completely sure at the beginning of the company. Because think our strategy was not exactly the same as what it is today. But what we underestimated initially is the complexity of deploying this model and connecting them to everything to be sure it has access to the company knowledge. It was, yeah, we were seeing customers struggling with this, but it was that was two even, years ago, and no things are much more complicated because no, you don't just have text on SFT, on simple instruction following, no, have reasoning like agents, you have like a tools, you have multi modem audio. So it's much more complicated than before and even back then it was hard for customers. So they really need some support, and this is why we're actually providing, like, always some four d position as well as the devices. Speaker 1: I'm curious. Is there also Speaker 3: voice fine tuning that people do? So in this foge, we also had unified framework. And the hope is, like, the Voxter's speech to text that we released earlier this year and even the Voxter chat that we released last year. And I think a big people I think there's a big rich ecosystem of people fine tuning Whisper and people want the same thing with Warxer. It's much stronger than and, yeah, the the platform offers that kind of fine tuning. Yeah. Which could be any kind of fine tuning. Like, for instance, even sometimes people want to support new languages to this, which are tail languages which we hope to cover ourselves natively. But if there is language where you have data and you want to fine tune, I think this is a good use case. Or the other use cases, you it's the same language like even English, but it's in a very domain specific way with Yes. It's terminology, German, medical stuff. Exactly. And also there's specific acoustic conditions like there's a of or other. And the model will do decently in most conditions but you can always make it better and that those are some of the use cases where you can include even further and that's one good use case for this and for text to speech, we're just releasing it. So we'll have support for that soon too. I think it's similar use case. It's little different, the kind of things that you want to extend a to speech model to, which could be, voice personalization, voice adaptation for enterprises, and Speaker 1: many enterprises need very specific kind of tone, very specific kind of, like, personality for this kind of voice. And all of those are, like, good use cases for fine tuning. This one was gonna ask you. We never talked about cloning voice cloning here. How important is it? Right? Like, I can just clone a famous person's voice. Okay. But Speaker 3: The main use case would be, like, for enterprise personalization. Like, enterprises need, like, a lot of customization. You don't want the same voice for all the enterprises. Each enterprise want customized, a specialized, something which is representative of both their brand and also their, I guess, safety considerations and the use case. I think the kind of thing that you would deploy as empathetic a assistant in the context of a health care domain would be very different from the kind of thing that would be in a customer support bot and would be different from, like, more conversational aspects. I think those are the customizations you would expect from enterprise, Speaker 4: that's the main use case, at least from us. My my basic example is you don't wanna call to customer services and have the same exact voice. It's gonna be weird. But also on the technical side of this, so there's like a few things in Vauxhall that I thought were pretty interesting. He's a big fan of this paper. Said very good. But he said this is the best ASR paper he's ever read. Yeah. Yeah. I've hyped up this voice paper. Fair We covered it somewhere. But a big thing, so Whisper is known for thirty second generation, thirty second processing. You extended this to forty minutes. There was a lot of good detail in the paper about how this was done. Even little niches of how the padding is so it's very much needed. You need to have that padding in there. The synthetic data generation around this, I'm wondering if you can share the same about the new speech to text, Text to speech. So how do you generate Speaker 3: long form coherent? How how do you do that? And then any gems? Is there gonna be a paper? Yeah. Yeah. They would be a technical report. Yeah. I think it could have a lot of details. But me I think the summary of it actually, some of the considerations in this paper were because we started with the VSPA encoder as the starting point, and now we have in house encoders. Like, the real time model, for instance, which we released in January, we also released a technical report for that real time model as well, which is this dual stream architecture. It's an an interesting architecture. You should check it out. And there, we have a causal encoder. And I don't think there's any strong multilingual causal encoder out in the community, so we thought it's a good contribution. So that's one nice encoder if the other people want to adapt. That's a good encoder. And we trained it from scratch. I think our post stack is now mature enough that we're able to train super strong encoders. And some of these considerations like sparring and stuff is a function of the whisper encoder. And now that we train encoders in house, the design considerations are different. And for the question on text to speech, I think that also leans onto the original autoregressive decoder backbone. I think it's almost identical considerations. I think the long context in it's not even long context. So the model process audio at 12.5 hertz. So one second maps to, like, 12.5 tokens. So I think one minute is, like, seven twenty tokens. You can get, like, up to ten minutes in eight k context window and get half an hour in 30 k context window. So that's and 32 k context is something that we are very comfortable training on. We can extend it even much longer. One forty eight k. Okay. We can naturally see how it can extend to even our long generations. Yeah. We need the, like, data recipe and the whole algorithm to work coherently enough through such long context, but the techniques are, some way, very similar to the text long context modeling. And the key difference is it's just doing flow matching order aggressively instead of text open prediction. Okay. I think that was most most of the sort of voice questions that we had, perhaps. I have a big question on Speaker 2: Mistral Mistral small. Let's go. So Speaker 4: what is small? How do we define small? What is this? What is this? I remember the days of Mistral seven b on my laptop. Speaker 2: It's not fitting on my laptop. I could run out on the big laptop, but It's just a different question of terminology. Like, here, what we do by sport is enough active parameters, but it's true. We did. Let give it another name. But, yeah, we could have called it medium, but then I think. It's a model that we really mystery of experts. It's a model that combines different model. Before, what we were doing is that we had the one model, general model for industrial doing instruction following, we had like a separate model that was Depstrahl. So really good at Koning specified, specific to code. It was another model for reasoning Magistral. So these were separate artifacts built by different team at Mistral. And what we are doing is basically merging all of this. It was had Pixtral, third vision model we had was like a separate model. And the way we do things internally is that we have one team focused on one capability, build one model. And then when it's mature enough, we decide to merge this into the main picture. But here, this was the first time we basically merged all of this into one. There are some other things we did at time to merge it. I'm from Stanza, more capabilities or function coding, think would be, it's going be much better in this trial, small, proper one, but a much better model on the way working on. Also, the ultraversion of this. And yeah, key things is very sparse, 6B active, pretty efficient to serve. Two fifty six k context. Speaker 1: Yeah. I think what's interesting is just this general theory of developing individual capabilities in different teams and then merging them. Where is this gonna gonna end up? Speaker 4: Like, Speaker 1: we've seen the five things put together in this. Yeah. We'll do the next five team. I think, actually, OpenAI has gone away from the original four o vision of the Omni model. That's what they were selling all modalities in, all modalities out. But I feel like you might do it. I think there are some modalities where it's not competitive use. For instance, for audio. For audio here, if you want to do transcription, Speaker 2: I think it makes the sense to use a model that's large. If you just want to transcribe tech, it's it would be very inefficient. You want to do audio, you probably just want to be the one B or three d model. Performance would be essentially the same. It's going be incredibly cheaper. So here, that's why we want to have a separate, but it just does this. Yeah. I think the question is just, yeah, if you are talking to your model by speech and you're asking like a very, complex questions and how do do this on your web just to cascade things. Do want to put a duel in a model that has like a one key around it? It's like a not a competitive discussion, I think unaware if you're doing into the direction, but that's the right possible approach. But yeah, but I think for us the next capabilities we want to try to integrate into these models, now are going to be, yes, like more coding, more reasoning, but I think more capabilities that people don't talk too much about, but that's important. I think for our customers in different industries, for instance, things are unlikely to legal, know it's computer aided design, all of these things that is this method out of the box are unlikely to put at that because people really don't prioritize this. There is no, like, two new benchmarks on that. But it's not hard to make this, but it's more like good on this. Just have to do the work, sourcing some data processing it. Including it into some the expression. So so yeah. But we have other things we merge into this. I think for voice, yeah, the Speaker 1: key thing I think over maybe, like, the last year or so with VO and Grok Imagine and all these things is joining voice with video. Right? Which people don't understand spatial audio because, like, most TTS is just, oh, I'm speaking to a microphone in perfect studio quality. But when you have video, like, the voice moves around. Speaker 3: That's true. The consideration was a little different in the sense that there it's like a a standalone artifact where you get the whole thing and you consume it. But the conversational setting, it's a you need the extreme low latency. Yeah. Streaming Speaker 1: Yeah. Would be one of the primary concentrations. You can build a giant company just doing that. Right. So you don't need to do the voice. But I was just you know, on the theme of merging modalities, that is something I am like, wow. Like, I didn't everyone up till, let's say, mid last year was just doing these, like, pipelines of, okay, we'll stitch a TTS model with a voice thing and a lip sync thing and what have you. Nope. Speaker 4: What? Giant model? Yeah. I have a two part question. So one is it's still open. It seems like open source is still very core to what you guys do, and I just have to plug your paper. Jan twenty twenty four, this is the onyxtraol of experts. Like, very fundamental research on how to do good MOEs. Speaker 1: Paper comes out. Very good paper for anyone. That's just side tangent with No. This thing caused We bring that. Eight by '22 was, like, the Speaker 2: nuclear bomb for open source. I think that takes Speaker 1: seven b more. Seven b more. B. But this is a big part for seven b. Yeah. Yeah. I don't remember this. I remember I don't think it was January. Right? It was like New Year's. It was It dropped during New Year's. And everyone in Year's was December 2023, but I think, yeah, the model was did as well. I think there was a big just a little update probably. Yeah, no, but You have a point to make. No, you gotta check that. But then I just wanna hear more broadly on Speaker 4: open source for you guys. When you had asked earlier about what's next, what are the other side tapes working on, you put out lean straw. Speaker 2: This one is a surprise. I was like, I don't this doesn't fit my mental model, miss Straw. Yeah. First, for open source engineer, I think it's really something which in bills to the G A of the company. I think we started it around this. We have been open sourcing model since the beginning and even before this. So before this. So me Tim are at Meta. We released Lama. And I think what was really nice to see that before this, for most researchers, like universities, it was impossible to work on LLMs. There was no LLM outside. And if you look at many of the techniques that were developed after, for Lama was open source, all these post training approaches, like even DPOD, performance optimization, all of these were done by people that had access to this model and it would have been impossible to do without this model. So it's really making sense move faster. So we really want to contribute to this OpenTourse ecosystem. I think that the DeepSQL and also like very lot of impact. All these papers that are, I think, in the open source community are really helping the sense community as whole to move faster. So we want to contribute to this ecosystem. That's why we are releasing very detailed technical reports of Magistral and our first reasoning model. There's a lot of reasons, things that worked, things that did not work as well, starting helpful. And the yeah. So for the audio model, we also share a lot of details, share a of them for the real time model. And the yeah. So we really want continue this, basically belong to this community of people who share science. I think we really don't want to be living in a world where the smartest model, the best models are only behind clusters, only accessible to a few companies that we have the power to decide who can use them. And I think it's a scary future. We don't want to live in. We really want this model to be accessible to anyone. We want intelligence to be used and accessible by anyone that can use it. So, yeah, so that's why we are pushing this mission and sourcing model on the yeah. So not so, yeah. Move strategy. So it's open source, not the first model. Not the best. On the, yeah, LinStrial, I think, is also one step into this direction. So it's, yeah, a bit different than what we are usually releasing. But we have a small team internally working on formal proving, formal math. So I think a subject we care about in general. And we were working on reasoning. I think we started too early before LMs. Doing reasoning without LMD is very hard, especially when you work with formal systems because the amount of data you have is negligible. It's a very small community of people writing formal proofs. But the reason why we like it is because I think there is, if you look at what people are doing with reasoning, is there are the problems that you can use are usually going to be problems where you can verify the output. So for instance, all this AIME problem, the solution is an number between 100 like a thousand. So you can verify, compare this with the reference or it's an expression. You can actually compare the output expression, generate your model with the reference. But there are many, most of them have problem and most of the reason and problem, there is no lack of way to easily verify the solution. If the question is, shows that f is continuous, you cannot compare any reference. Right? If it's a proof that this is true or prove this property, there is no way to you cannot simply verify the correctness of your proof. It's hard to apply the there is no very favorable reward here. So what you could provide is, of course, like a judge, an unjudge that will look at your proof, but it's very hard and it's very You could assess on your reward hacking happening there. So it's difficult. But you could provide like a reference proof, but then there are also many ways to prove the same thing. So if the model says give a negative reward because it's a different proof, maybe it was still a digit proof just different. So it's not going to work well. What's nice with Lean and with formal proving is that you don't have to worry about this. We just They all function as as it compiles in LIN, it's functionally the same. Exactly. It's like a program if it compiles in print. It's correct. Yeah. It's very easy, and you can apply this on any kind of It's just way too small. So no human will actually go do it. Yeah. That's exactly it. The only people can do it. It's like a very small community of people doing a PhD on that. So it's super small. And it's sad because it's actually very useful on not just math, but also in software verification. So for instance, verification, today it's a tiny market. Very few industries work on this, and we need that. It's usually going to be like companies like building airplanes, aerobiotics, like, the things where they absolutely want to be sure life depend on this, but it's very rare that people formally verify the correctness of their software. But I think one reason for this is simply that it's just super hard to do. Are thinking of TLA plus? It's the language that some people do for software verification. No, I I with with that people use it in a in France. But but yeah, it's the reason I think why people don't use it more and why this industry is not as big as pulp is because it's very hard. But now with coding agents that are there, it's going to be very different that we are going to see much more of this. So I think, yes, industry there is going to be much larger in the future of those that we have these models. So, yeah, here also anticipating this a little bit. We wanted to work on that because it's proving like a math theory and a problem that unfortunately Speaker 1: is the same tools. Yeah. Yeah. One of my theories is that because the proofs take so long, it's actually just a proxy for long horizon reasoning and coherence and planning maybe. A lot of people will say, okay. It's for people who like math. It's for being okay. It's like a niche math language. Who cares? But actually, and you use this as part of your data mixture for post training and reasoning, actually, might spike everywhere else. Yeah. And I think that's on, like, underexplored or no one's, like, really put out a definitive paper on how this generalizes. Speaker 2: Yeah. Absolutely. And I think even that's what we are seeing already. For instance, if do some reasoning on math, then the American Institute do reason and code a role even. Yeah. Just code that in the in early stage. So it definitely there is some transfer, some sort of an emergence that happens. And I think some it's also interesting. It's not just, I think, the topic in general, but it's there is a of connection with this on encoding agents because sometimes the model can see, like, a theorem that it has to prove it's very complex, but then it can take the initiative to say, I'm going to prove this three lemma. I'm going to suggest three lemmas, and I'm going to, in parallel, prove each lemma. So three of them in parallel with sub agents, But I'm also going to prove the main theorem as you use the three lemmas at So you can do this sub agent approach. So pretty interesting. You can even if you fail to prove one the lemma, you can actually maybe succeed to pull the number one lemma too. So you get some reward here. So it's a bit less part than if you just get a zero one for the entire thing. So it's pretty interesting, I think we can actually unplug stuff together. Yeah. It's also an interesting case just for specialized models in general. Right? Like the cost thing you show is pretty interesting. Yeah. Similar score wise, Speaker 4: you're $30.70, $153,100 Speaker 1: bucks. I would pay a small dividend. I think cost is a bit unfair. Right? Because this one is at, like, inference cost. It's not as they're on top with their margins on top of it, but we don't know anything else. Right. Speaker 4: Gotta figure it out. Right. I did wanna actually push on that more, not on cost, but you mentioned about, okay, it's a great way to have verifiable long context reasoning. What are other frontiers that I'm sure you guys are working on internally? There's a lot of push of people pushing back on pre training, scaling RL, pushing compute towards having more than half of your training budget all on RL. Where are you guys seeing the frontier of research in that? You mean in Wizzle? Just in foundation model training in the next one thing that you guys do actually is you do fundamental research from the ground up, right? So you probably have a really good Speaker 2: look at where you can forecast this out. Yeah, I think for us, we're still working a lot on the pre training side. I think we're very far the situation of the pre training. I think ML4 pretraining will be like big step up compared to everything we have done before. So we are pretty excited about this. And I think on the health side, I think now we have more and more to think about this algorithm that will actually support this very long trajectories. I think when it was, for instance, gRPO, for doesn't really work with this tiny bit of policy, which was okay initially because you are solving math problems that can solved in like a few thousand tokens. So the model can actually generate them pretty quickly. So when you do your update, the model is never too far It's never too far off, but now when you are moving towards these kind of problems where sometimes it takes hours, like six hours to get a reward, then your model is completely uplifty. You have to be more accurate. The infrastructure that supports this, but also new algorithms. Now everything we doing internally, we're trying to build some infra that we actually anticipate is what we have in 06/2010, which is this extremely low scenarios on the hard price of the vehicle. I think when we started Mistral, part of me and maybe also Timothy, we wanted this very nice environment where people are there, so can do research alike with a of resources. So it was nice. Think things changed a lot when I think when ChurchillPT came out. I think after that, I think was very little on. These labs were on the other side in end. But, yeah, but it was nice. And I think we also want to work part of this creature before. Speaker 1: Coming to the end, we're just obviously, I think you guys are doing incredible work. You've they've a very impressive vision for open source and for voice. What are you hiring for? What's the what are you looking for that you're trying Speaker 2: join the company? Yeah. So we are hiring a of people. In our science team, we are hiring in all our offices. So we have HQs in France. In Paris, we have a small team in London, like a team in Palo Alto as well. Presently, we opened some offices in in Varso, in Poland, so one in Zurich. We also have, like, some presence in New York as well. Yeah. And soon one in San Francisco. So we are a bit everywhere, also, like, hiring people remotely. So we growing the team, trying to hire, like, very strong people. I think we want to stay so the team is not still a fairly small team, and think we want keep it that way because we we find it quite efficient for, like, a small team, very agile. So yeah. Okay. Let's focus on science and the for the point. We actually are strong believers in science. We started the our new science pod that focuses specifically on the air science. What areas do think are the most promising? What you are pretty excited about right now and something we have already started doing, or we'll probably be able share more about this in a couple of months, is that we are exploring AI for science. And there are a lot of areas where we think that you could get some extremely promising business If you have to apply AI in these domains, there are a lot of longing fruits. You just have to find these domains where actually AI has not been yet applied. And it's usually hard to do because the people working in those domains don't necessarily know the capability of these models. They don't know how well they are. We just had to pair them with, yeah, exactly. We researchers matching, which is actually hard to do. But this matching, are doing it naturally with our customers. So we have some company we're very closely with. So for instance, ISM Electric is one of our partners. So we are doing some research with them. There there like tons of extremely interesting problems, problems in physics, in science, science that they are essentially the only ones to work on because they are doing something no else is doing. And the, yeah, so there are many domains where AI can actually revolutionize things. It's just you have to think about it on eFamiliar with what can do more now to apply it. So, yeah, it's something we're more modeling with our partners, with our customers. So AI for sentences, but on big thing. Yeah. Okay. And then four deployed. What it makes a good four deployed engineer? What do they need? Where do people fail? Think it's usually you need people that are very familiar with the tech. Not necessarily with a lot of research expertise, but that are actually pretty good at using this model that can actually like, that know how to do fine tuning, know how like start some ARL pipeline. And it's it's not easy. It's something that must majority of companies will not be able to do this on their own. So here, I think we need people that are that like to solve problems that I set the blood actually. Some complex, very concrete problem. It's applied science, basically. And then, yeah, so I think it's not too different, I think, from the skills you need about in your research. Because essentially, you are trying to find solutions to problems that customers have not yet solved. Sometimes it's easy. Sometimes you really have to do the work. You have to, like, create synthetic data, find some edge case. It can be, yeah, depend on the problem. But yeah, you have to I think it is also a bit of patience on be creative. I think very similar skill is is urgent. The diversity of the work they do, it always appraises me. It's it's it goes all the way from the kind of stuff they encounter in industries. It's just very interesting, I think. Any fun, like, anecdotes? Yeah. It can be, really training this small model and edge that just we do one specific thing. It'll be, training some very large model that would have some specific languages as well, making models really good at some to do's, for instance, Speaker 3: computer ID design, these kind of things. Is that impairing with vision as well? Yeah. And defect detection for chips or like in in factories, identifying things. Like, it the diversity could be anything where you can deploy these foundation models. Yeah. The work to make it work in that specific setting, basically, Speaker 4: whatever it takes to make it, like, add value in that specific workflow. Yeah. And it goes across the stack. Right? Like, even just pulling up the website. Like, The truth is so broad. On Pute is so broad. We didn't even touch on MistralBybe. Have a live coding CLI tool. One thing you guys were actually like, I think the first two was Mistral agents. You're the agent builder. You can serve it via API and all that. I I'm guessing forward deploy people Yeah. Help build that out and stuff. It's also why we are so we are doing many things, but I think that's also Speaker 2: part of the value proposition that sometimes the customers are always very extremely careful about their data. And they don't want they don't like trusting so many partners, trusting one partner for code, dealing your data to another third party for, like, audios and another one. They they don't like this. Here, what they really like with their approach is that we can help them on anything. So they don't have to send that out to so many clouds. So yeah. I think that there can be many orders of magnitude more FTEs than research scientists, and they don't need your full experience, but they're still super valuable to customers. In practice, these two teams are still quite intertwined very often. So first of all, they are using the same tools, the same data pipeline and everything. And it's very helpful for the science team to get the feedback and the solution team because they can say, look where these customers are trying to do this. This is not working. Can we be sure in the next version? Yeah. This is basically a real world eval. Exactly. It's a real It's not something For instance, if you are just working in a lab, it's just ships model, but you don't do this work of deploying the model for customers. You have no idea whether your model is good at this edge case. For instance, even in your before this, there So is a very big between Speaker 3: the the public benchmarks that are fairly like academic on the the The rear cases are just very diverse. And in the specific concept of a customer, you can fine tune and make it like first evaluate, create a solid eval benchmark and then measure in the context of their the kind of audios, like, for instance, one use case is literally just there'll be a word for kids and they have to just say it out. It's a very specific thing. You're just saying one word and then you have you'll to you'll grade the kid whether they didn't write or not. It's like RL for a But so there's very diverse use cases and the idea is that the the applied scientist will go and make it better and then from the learnings we incorporate it into the base model itself. Speaker 4: It's just better out of box. It's a good full circle system. The foundation model evals are all just proxies of what you really need. You're never gonna have one that's it Speaker 1: doesn't make sense for there to be a one word transcription like that. It's not something you wanna fit on. Perfect. Everyone should go check out everything that Michelle has to offer and try the TTS model, which will link in the show notes. But thank you so much for coming. Thanks. Such a strange story, guys.