The Model Eats the Scaffolding: DeepMind's Logan Kilpatrick & Tulsee Doshi on 3.5 Flash, Omni & More

A lot. So a lot of exciting stuff. So let's see.

Let's start with some of the modeling side of things because that's really exciting. We have our 3.5 series coming out starting with 3.

5 Flash, at IO. We're really excited about 3.5 Flash because I think Flash does this really awesome job of being at the sweet spot of being really smart while also being really fast and really cost effective.

And so Flash is incredible. It is, like, three times faster than other of the large models. It's significantly cheaper for being able to still drive these, like, really awesome magentic encoding workflows, and we've been using it internally a lot, which has been really fun to kinda see that play out.

So that's one big piece, which is 3.5 Flash I'm really excited about. We're also releasing Omni, Gemini Omni Flash, which is a video generation and editing experience.

What's really exciting about Gemini Omni in general is it's our push towards being able to bring all modalities in and all modalities out. And the first way this is really manifesting is in this video editing context. So you're gonna be able to make really awesome, videos.

You're gonna be able to put your own avatar into the videos, which is gonna be awesome. I've been having a bunch of fun playing with that too. We're continuing to upgrade anti gravity and bring more into the developer experience.

So Logan can talk more about the developer experience overall, but 3.5 Flash and anti gravity are really gonna come together to build build something great there too.

which is really interesting. Yeah. And I think what's been really fun from the modeling standpoint there is one thing that we did with Gemini three that we're really continuing, I think, with the 3.

5 line is really bringing the model to every all of our products. So, you know, 3.5 Flash will be in Gemini app.

It will be in AI mode in search. It's also powering antigravity. It's also powering agentic experiences in AI Studio in, you know, Gemini Spark.

yeah, which is exciting. I think one other thing I'll say is, like, one thing that's cool about this I o two is what we're doing across modalities. So, you know, I think Logan said agent agent agent, which I think is true.

This this I o is really about bringing models to action in in in in that kind of real world sort of use cases. But I think what's also cool is we have the flash model, which is really about building these kinds of coding and agentic use cases. We have Omni, which is really about kind of what is this, like, multimodal vision look like.

And then also, actually, Gemini Live is getting an upgrade too. Gemini Live is getting faster. It's getting smarter.

The model is much better at detecting background noise. So it really does actually feel like a partner in a lot of ways, and I think it's kind of cool that we're also able to draw this through line across the different ways you might wanna interact with a model and the way different kinds of ways you might wanna consume content, which I think is also really cool.

it. One. This is my favorite question, so I'm I'm glad I'm glad we're meet you.

No. I think, you know, you're you're right that I think there is a slice of of users who are definitely willing to pay for a certain level of quality. And I think we really do believe that the pro model has been like really pushing that that quality.

But I also think for us we've seen so much value from the flash and the flashlight dimensions because we also see a extremely large number of users, especially if you're thinking about building, for example, consumer applications. Right? If you think about the Gemini app, if you think about search, when you're serving to that kind of scale, latency really matters.

Cost matters. Right? Because actually you find that users aren't willing to wait, right?

So we find that even when we tweak the model and you know hurt latency, we actually see that play out in our live experiments on search and the app even if the model is hugely better from a quality perspective because what you're asking users to do is wait. And so I think for us, like, part of the reason why we ended up introducing this flashlight SKU that wasn't necessarily part of the, you know, the original two point o series was because we really felt like there's actually a large scale demand for this depending on the types of use cases, especially when you're when you're talking at that scale.

but it hasn't been that, like, we haven't kept scaling up. So I think that it definitely has Yeah. There's almost been a conversation every time we scale up of, like, should we call it ultra?

Yeah. And and what does that brand mean? Because we we could, but there's sort of also a question of, yeah, how do we keep consistency for users also kind of series to series?

Yeah.

distillation to the midsize? I mean, we definitely use distillation as a way of kind of bringing down bringing down our sizes. So you you will see that, like, pro influences flash, influences flashlight.

We also do the reverse where we scale up. Right? So you you take the pro you take the flash recipe and and scale up to the pro recipe, for example.

And we do have I think what's been really especially over, like, seeing as we've used even antigravity into this point of Logan made with the harness, I think we've been seeing a lot of examples actually of leveraging pretty awesome models to drive progress internally. Actually, like, one thing Varun demos on stage on Tuesday is basically, like, being able to leverage a bunch of sub agents to go and complete a bunch of tasks and come back. And you can actually try that as, like, an early preview in antigravity today if you go to slash teamwork.

And, like, that's an example, I think, of something we've been using internally, which is an extremely smart model, and it leverages both the combinations of the best of Gemini 3.5 as well as inference techniques. And and, like, you're able to actually accomplish so much.

And I think that's that's the kind of direction I'm excited for us to go into more. So I think we're kind of pursuing all of these fronts. We're scaling up from from the pretraining and kind of frontier perspective, and I think that's been really continuing to show gains.

There's a bunch we're doing on the post training side, and then there's also just a bunch we're pushing on on the inference side. And then that plus, you know, trying to make sure we're we're working with the harnesses. I think we're gonna keep getting things that we're using internally that we even started to push out externally through previews.

Yeah. I think I think that's the right I think I fully agree. I think actually, like, maybe to double click on what Logan said originally.

Right? The benefit of the full stack that we have is we can hopefully build a really seamless experience. Right?

And you get the best of Gemini. You get it working in the most effective ways for you. You get it working in a way that is intuitive, is smart, is fast.

And so that also helps us then train the model to be better. Right? So this becomes this, like, flywheel that continues to power the model.

At the same time, I think we don't want it to only be the case that the model works in a single harness. Right? So we want any of our enterprise customers or a developer who's building their own use case to be able to leverage Gemini effectively.

And so it is important then from a model standpoint that we're training in such a way that we actually like, we we sort of call it, like, harness diversity. Right? We should be able to support a range of of different approaches to tooling, to different approaches to orchestration, etcetera.

But I think what's helpful about this this approach of, you know, kind of co training and and building that flywheel, it's easier to debug. It's easier to, you know, think about data collection. It's easier to eval.

You can just move at a faster pace. And I think we're seeing that across the industry. And so finding that balance is important, but I think it just helps build build make the model better.

Yeah. I think there's a good this is also a good pitch for, like, a harness bench. If that's not a benchmark that exists, let somebody somebody build harness bench.

I mean, we're already using Gemini, like, pretty deeply internally to improve Gemini. And so I think that is very much a theme for us, which is like how can Gemini actually be a part of the Gemini development process? And so that can include things.

I think that goes the full range from, you know, helping us be more productive. So that's obviously, like, the simplest part of this to actually, like, you know, submitting CL that would actually, like, run an eval that would actually, you know, suggest a research improvement that would actually drive improvements to Gemini itself. And I think there's a lot of ambitions we have to keep pushing in that research direction.

So I think very similar to the other other labs, I think this is very much an area of investment for us and an area we're super excited about. I think for me, what's I'm really excited about is, like, I think there's this really awesome research partner opportunity that we have with Gemini, right, for it to help us with creative ideas, for it to, like, help us test things faster. Actually, like, was it also one of my coworkers, Anka.

She's our lead for safety and alignment. And the other day, she I think maybe a couple days ago, she pinged me from her hot tub, and she was like, you know, I could run all of these ablations from my phone because I could, you know, kick off a bunch of things to, like, actually ablate Gemini to test for a bunch of these issues to see how, you know, some of our SIs differ or some data ablations differ. And here's my report.

And I could do all of this in the last hour. Right? And, like, that is amazing.

And that's the kind of thing that we can already do. Right?

how much that collaboration allows you to then focus on what is the interpretation of what you're seeing in the results, where do you really want this to go strategically. And so it changes a little bit of the role that the human can play, which I think is also really powerful for our teams.

on the product side, like, on the code side for any code that I was already submitting, I am mostly relying on anti gravity and doing, like, bits and pieces, more so bits and pieces myself. But it's also been really cool to, like, start having the model generate slide decks to start generating actual kind of content from my thoughts. Actually, in antigravity today, we introduced the Gemini mic.

So there's this, like, really awesome feature. Don't know if you've been playing with it internally where you basically, like, ramble at the model. So you, like, share a bunch of your your thoughts in whatever kind of loose form it is, and then the model actually leverages that to take action.

And for me, I've been finding that so much more powerful because I actually feel like I think a lot by talking. And so for me, like, it's it's actually like a very it's a it's like a very cool moment where I can be like, okay.

Yeah.

I think the other thing I would say, like, as far as, like, lessons learned is, like, there's really no substitute for being able to just experiment and iterate quickly. Right? So I think this goes to all of Logan's points about the foundation being strong, but I really think what has helped us is really being able to put in, for example, a new model, iterate really quickly with a product on, like, hey.

What are the right prompts that would, you know, actually make this model viable for a different situation? What is what are the ways to kind of prototype really quickly with this model? What are the ways to get it in the hands of even just internal users quickly, let alone external users?

And I think that is something that is now more and more possible with kind of sim like, layers that are consistent across the team. I think it's it's pretty amazing to see the the speed at which we can go from, you know, having a checkpoint that we're really excited about to putting it in the hands of internal developers to then seeing it come to life in a product. And then only when you see it come to life in the product do you really start finding its rough edges and to be able to, like, actually then kind of come to to terms with how you do that.

And so more and more then it becomes like, okay. How do you have the right ability to tune prompts quickly? How do you have the ability to run really good live experiments where you can get really good data and feedback quickly?

How can you build evals that help give you real signal? Those are the things that will speed up your progress of quality the most because it will give you the the ability to actually get to the kind of product that you love. And I think if you think about NotebookLM, I mean, that team really understands the model.

Like, they are they are just like I mean, you talk about a banger product, it comes from, a banger team. Like, they are really good at being able to, like, take the model and play with it quickly and, like, prototype quickly to get to something amazing. And I think that's you you see that actually play out in the product.

building Gemini within Google and having kind of all of these different product teams is, you know, there's always gonna be something that doesn't work for them. Right? Because there's always gonna be something that can get better in the model experience.

Right? So we're trying to build something in a product, and, like, the amazing moment is when you start trying to build it and it doesn't work. And so step one is you're like, okay, can I prompt my way out of it?

Like, what does that look like? And then you start figuring out, okay, what are the losses really? Like, where is the model falling down?

And then what we try to do as much as possible is keep these feedback loops with our product teams to say, okay, if this is where the model is falling down, how do we bring that feedback back to the model in terms of evals and data? What does that look like? So then that we can actually, in our next revision of the model, bring all of that feedback back in and iterate on it.

And I think that's how you've seen Gemini get better is really from from that feedback of where things aren't working. And so we try as much as possible to kind of have the structure be you know, we train a model. We hand that to kind of a wide range of teams.

Those teams implement the model in their structures. They do a bunch of things to Logan's point because it's extensible, but they also find all of these places where the model falls down, and we kind of cycle that back. And I think that's actually been part of, like, the fun part of the job, but also part of what makes, I think, Gemini work really well in some of these use cases.

Yeah. So I think we have to figure out what we want on the API side for this to look like in terms of I think maybe the heart of your question is, like, native video generation. That is so yeah.

So this what's what's exciting about Gemini Omni is it really is building on all of the magic of Gemini. So kinda like this whole nano banana for video. It's really about how do we bring in all of the world knowledge and the reasoning power of Gemini and actually be able to generate native video as a result of that.

And so I think we have to figure out then, like, how does this manifest in the context of the API from, like, a sampling standpoint? Kind of, like, similar to a lot of the decisions we've had to make about VO from a sampling standpoint. But I think right now, as of now, you'll be able to use it in the Gemini app, in Flow, and in YouTube.

And so those are all gonna be ways that we can start actually seeing how people experience the model, what, you know, what value are individuals getting. And I think similar to this nano banana for video, I think we're really excited for these types of things where you can say, okay. Take some of these images, take this scene, and, like, make these things all come together in in one video.

I think it's gonna be really awesome.

There's a lot of there's a lot of people, so I'm sure you have a lot of varying sort of perspectives. I mean, you know, like, to Logan's point, think, you know, even within GDM, you're gonna find a range of folks who will leverage Gemini differently. I think in terms of how we think about it, we do have a strong point of view on the kind of behavior we want Gemini to have.

So I think we do really, you know, want to be intentional about how Gemini manifests itself to internal and external people. But I do think it's really about how does Gemini help Googlers and how does Gemini help people within Google and outside. So I think it is really much about, like, how do we how do we create, like, good partnerships between Gemini and people, I think, is very much like the ethos of what we're trying to build.

And so how does Gemini become that partner? I think we use the word collaborator a lot. Like the word like, how can Gemini be your collaborator?

Both, like, in the code you're writing as well as in your, like, day to day life and what you're doing. And I think that's the ethos we're trying to bring in its behavior and persona as well as in the kind of products we're building around it, if that makes sense. Yeah.

generation of Gemini to the next? Yeah. It's interesting.

I I haven't thought about the phrase psychological distress, but what we do so I I think it really does matter how Gemini communicates with you as a partner or user of Gemini. I think that matters a lot. And so we have actually, like, pretty extensive safety evaluations in terms of things like how Gemini engages with you, in terms of things like sycophancy, in terms of things like, you know, role play, in terms of things like this kind of looping type behavior or rabbit holing type behavior.

So there's actually a lot of that that we look into for every one of our checkpoints because it really does matter, especially as we're starting to use Gemini more and more. If you're using Gemini for hours a day, it really does matter, like, that these attributes are well understood and well evaluated. So so, yeah, we we definitely and we look at them launch over launch, right, to say, okay.

Is that a bug? This is also one of Logan's favorite topics to discuss. Yeah.

No. I mean, I I think, updating the knowledge cutoff definitely important, and something that is on our radar. I think the the other part though is like how does deep research do so well or how can we use the model in search is because we also have the model search.

Right? So I think for us it is like really important actually that the model be able to know when when to leverage its parametric knowledge versus when to actually go out and get the information from the web. And especially because, you know, there is information that's as fresh as an hour ago or a minute ago, like, want the model to be as up to date as possible.

And so I think for us, we've been really leaning into how do we help the model search effectively. And that's a big part of what makes it successful in the context of search or the app, or even anti gravity actually for that matter.

I think people definitely do want lots of context, but I think what we've also found, if you look at even, like, personalization where you wanna access, like, all of your personal context or coding where you have, like, extremely large code bases, I think a lot of the the frontier here is gonna be actually on how you smartly use context. So thinking about, like, compaction and, like, what are the right ways to, like, find the right elements of the context and bring them into the model. And so I think that actually is, like, a huge opportunity.

It's, like, how do you leverage all of this information that the model might have access to? But, actually, a lot of it is frankly distracting for the model to actually do the right thing. And so how do you give the model the right amount of context in the right way to be most effective?

So I think that's actually really the direction that we wanna be pushing in, which actually then, you know, in actuality, the the amount of context that the model is is leveraging is actually much, much larger. But because we're being smart about how that's actually coming into the context window, you can actually fit it into smaller context windows. But I think also, you know, this goes back to my point about flashlight and flash, etcetera.

Like, larger and larger context windows also come with cost. And so what we're what we also saw with customers and we still see with customers is that a lot of customers want to use smaller context windows because of that, and they wanna be more intentional about what's going into the model.

it's been quiet on that front. Diffusion is is is awesome. It is super fast.

I think we are still testing and experimenting with it in a number of different ways trying to figure out, like, what is the best way to put this out into the world, where is it most useful. But I will say, like, actually, part of the reason why we've also been investing in Flashlight is, like, Flashlight is an incredibly fast model. And, actually, if you look at the 3.

5 Flash model we're releasing right now, on artificial analysis, it benchmarks at, like, I think 280 tokens per second, which is, like, crazy fast. In fact, it's actually so fast that, like, sometimes in antigravity, like, by the time I wanna cancel, like, it's too late. And so I think, like, we already are like, I think trying to figure out, like, where is where do you start getting to to Logan's point in a different answer, like, diminishing returns and, like, where do you see that that value proposition is I think part of the question too.

But we are continuing to push on diffusion research. Our researchers who are working on diffusion are doing some pretty awesome stuff. I was, like, in a meeting with them the other day about some results that they have.

I mean, I think they're, like, still pushing the frontier of kind of quality and speed in ways that are really, really cool.

yeah. One thing actually as far as speed is concerned, just another plug is actually an anti gravity right now. There's actually a faster version of 3.

5 flash. So it is is speedy actually, which is I think we're kind of excited to see how people will use that and, like, what the reception and reaction will be to that too. People want fast models.

Yeah. No doubt.

What are you asking? Let's see. I I mean, I think, you know, we talked a little bit about this, but, like, the one thing I will say is, like, I'm really excited about where audio is going also.

Like, that's one that I think we tend to talk less about. But if you, like, think about the Gemini mic example or you think about kind of, like, the the Gemini live experience, I'm, like, really excited about moving towards a paradigm where audio is just a bigger and bigger part of how we engage with these models and how they engage back.

in the works, which I'm excited about and and many more many more stories, many more podcast episodes so that we can get to all see us. You have to get up to seven. You have to to seven podcasts.

Change. But but, actually, like, legitimately, was in a room this morning where one of my team members was like, I know we're gonna be launching this a few weeks later, but I really need a vacation. And I was like, well I don't need vacation.

We're just we're just going. We're just moving. No rest for the weary in the AI era.

That's for sure.