⚡ [AIE CODE Preview] Inside Google Labs: Building The Gemini Coding Agent — Jed Borovik, Jules

Okay. Jed Borovic, welcome to Lanespace. Yeah.

Thanks for having me. So we're sitting here at f dot inc's beautiful podcast studios, and we're actually meeting at GitHub Universe. How's it been so far?

It's been great. I mean, yeah, the keynote today was awesome. It was fun to see Jules update a little bit.

We have a lot of folks from our team here. Jules is partnering with GitHub for the new agent HQ stuff, which we're excited about. Also And this is a this is an incredible podcast space.

So, I'm excited to Yeah. Do this here. I'm glad for them to to loan us a space.

You are also an emcee for AI Engineer Code.

being part of the New York Tech scene. I actually think it's great having big major conferences there.

I think the thing that's interesting about New York is it's such a big city with so much going on. Right? And so there's like, your tech is a huge part of it, but there's also so many major issues there, whether it's Fashion.

Media. Fashion, finance, like it's and so it's like, think that helps push the tech and do all kinds of stuff. But yeah, No.

The East Coast is a great city. Great you know, the the all the schools, there's, you know, all across East Coast, kind of great schools and great students doing all kinds of stuff. So, yeah.

You know, I went to school there. Hackathon scene there was amazing. Really fell in love with tech and and programming there.

in Stafford where like CalHacks and stuff? Yeah. So there's A tree hacks.

Yeah. There's one that was put on by it was a while ago, but we put on by NYU in Columbia. We do do a Hack and Why.

So there's there's a there's a bunch of events kind of that that we did together. It would bring, you know, people across New York City, students across New York City. And those were super fun.

Yeah. So it'd be at the Columbia one one time and then NYU the next, and we'd cycle back and forth. So a lot of cool stuff was made there.

Nice.

How'd you get into Joules? Like, what's the what's the AI journey? Yeah.

So, you know, this is gonna sound really cheesy, but I've told a story a couple of times to folks when they're like, oh, how'd you end up, you know, doing this? But it is actually very true. So I worked on search for a long time and specifically kind of like news and freshness.

And then, you know, when Stable Diffusion came out, that to me was the first like Gen AI moment. I know people talk about like ChatGPT is like the first thing, but for me, Stable Diffusion, you know, was a couple months before ChatGPT came out. It was a huge thing.

I was following it a ton online and there were two groups of creators having reactions to it. You know, there was one group that was, you know, this is stealing my art. This is stealing everything that's near and dear to me.

I hate this. This is ruining my life. And there was another group of artists and creators who were like, oh, this is a tool to create better art.

And so I was watching this With a new brush. Yeah, exactly. And right around then, was having conversations with a couple of people who would say things like, you know, if I had a kid in college, I wouldn't recommend they study computer science.

I was like, what? And this was your long before like Jensen Huang and people like he had been saying this kind of stuff. Was like, woah, why?

And it was like, oh, AI, like software engineering is gonna change. It's gonna be so, you know, who knows if there's be jobs. I was like, I loved being a software engineer.

I love programming like And I was like, wait, this is my stable diffusion moment. This is either it's gonna take my art, my craft, this is a tool to create better art. And I was like, I definitely know which path I'm taking.

So I got, you know, very into to building coding. So I was still working on search, but I spent a bunch of time, you know, making stuff for my own time and playing with things. And, you know, ultimately tried to find a role that would, you know, the most exciting role I could find to do this stuff.

And that was to join Google Labs and Jules where we were, you know, right around then we're starting to build these kind of coding agents at Google. And, yeah, the timing worked out well and I joined and, yeah, it's been it's been awesome.

Labs' mission is to build kind of new, like, innovative products that the rest of Google isn't well positioned for. Yeah. Which we've had like riser from Nopal GLM.

Exactly. Exactly. So Nopal GLM is maybe the most widely The mostly yeah.

And then it's called Nano Banana. I don't know if it's Yeah. So some of the so the thing about that's really exciting about labs is we work incredibly closely with DeepMind.

Yeah. Right? So all the stuff in terms of the, you know, the we're building a product, but we work so closely for the model.

And you know, one of the nice things about being at Google is you have this opportunity to really build an end to end AI product, right? From like pixels on the page, through the infrastructure, through, you know, the model and the training and all of that loops. So Ladd is here to build new products and we're really like a product org, but a true AI product org where we work incredibly closely with with, you know, DeepMind, but also, you know, other parts of Google, you know, as it makes sense.

Is that is that true? What can we say about it? Yeah.

So, you know, I think there are there are, you know, Google's published papers in this space for a while. And so, yeah, we have, you know, in Google, we built a lot of our own tools and, you know, CIDR, which, you know, folks maybe have heard of is our, you know, our internal IDE. And we've had all kinds of, you know, capabilities and tools there for a while.

So, you know, we certainly have had pretty good tools for a while, but they were for internal use. Yeah. Yeah.

like, you know, there must be some interesting data in there. Oh, yeah. I mean, and that's one of the things, you know, in building a lot of these internal systems, you know, the data is incredible.

Yeah. Especially when it's, you know, not only is the model and the training in house, but all the data around the usage and whatever.

what is that experience gonna be? And let's build for that future. Right?

And so when you think of a really powerful agent that can run for a really long time doing really complicated things, that's when like the products started to take shape for us. So for example, autonomous, you know, means like it has its own computer. Right?

So for Joules, it's, know, the end. Exactly. So tons of agents that run, you know, locally or in your workspace with you while you're coding.

But if you want something that's gonna run for hours or let's say days, you know, you might want it to have its own environment where it can do its own work. So that's just one of the pieces that's important for kind of this autonomous coding agent, but it's really like, think about this future where they're incredibly powerful. You can spin up tons of them, right?

They're autonomous, but also, we're thinking about what does it mean for it to be ambient, right? Like it's kind of when it has its own infrastructure, its own computer, its own ways to interact with it. How does that start to change what it can do?

For example, like we have an API. So people are using it for all kinds of things, triggering it from, you know, something happens and we saw an example where someone has they're triggering Julesford to do kind of all kinds of updates to their site, and then they have a GitHub action that is going to automatically merge Jules board requests. So it's just like all kinds of stuff is flowing.

We really have changing how people are able to do stuff. And is that CLI related? Just to close that loop.

Yeah. CLI says we also have a CLI, which is, you know, we wanna meet developers where they are. Right?

And so part of the, you know, an API is like, can trigger it from anywhere. But also, you know, when you're working locally, like, you wanna be able to trigger stuff. So we yep.

So we have we have the the JUUL CLI we launched a couple weeks ago, which lets you interact with it. By the time this podcast comes out, we'll be integrated with the Gemini CLI.

and be able to harness this power. Right? Because your developers work in all kinds of spots.

which, you know, was was a it was a major update a few years ago. Yeah. So, I mean, just just fill us in, like, what is your AI engineering journey?

Yeah. Totally. So I think one of the things that keeps coming up is like the model makes such a difference.

I mean, maybe it sounds obvious, but it's like the quality of the model really changes like what you're able to do and how you engineer around it. So for example, when we started, this was, you know, with relatively early models of Gemini, we had the agent scaffolding around it was incredibly complex. I think one of the things we've seen is scaffolds get simpler and simpler over time as the models get better.

And in some ways, this the scaffolding is almost a crutch for for things the model struggles with. For example, like, you know, really complicated sub agent systems. You know, we've we've played with that.

We've we've experimented with that. Can you give an example of like a kind of sub agent that you had to abandon? Yeah.

We're just basically like you have, you know, you give Jules a coding task to do, and it's going to have different agents for whether it's, you know, making a code at it or handling a sub problem or, you know, doing any kind of action with an integration or, you know, having like full sub agents for different parts of it, like a reviewer agent or even like people, sometimes people do these like different personas where you're like, you know, one of the things that cracked me up is like, you know, you're the product manager agent and then you have the code reviewer agent, the agent. We didn't go that far. But I think a lot of these things aren't as in favor now.

Mean, certainly people do, know, they like, I don't wanna say the agent harness isn't sophisticated, it certainly is. But as the models get better, like less is more, especially as it comes to like being able to improve through whether it's machine learning or just, you know, regular maintenance. I think certainly we found that, you know, we're finding that less is more.

I think that, know, that we were talking about a little bit before we started recording, like like Rag, right? Like, you know, Copies and mixing. And all that stuff.

And, know, it seems like, you know, not just for Jules, but kind of across the industry that like agent based search, right? Like it's maintaining embeddings is hard, but getting the chunking right is hard.

it's yeah. It's it's not the I feel like when we started, it was like Rag. It was like embedding based Rag.

Yeah. It was like the thing everyone did. And it's interesting to see how it's changed.

People ask me for like, where are the good code embedding models? And, you know, I pointed them to like a few like Chinese ones.

just experiments. Right? So like, you know, Nobel Gummis, we talked about It's not very serious.

This is incredibly successful product. We save money. That's really And for us, IL was kind of a little bit of a turning point.

So in May, when we announced Joules, you know, it was like great reception following IO. And that was a real moment of us to like turn this into, you know, a very much a real thing. Mean, we did something that we were, you know, we always intended to, it wasn't ever intended.

You know, did, you know, talk about my journey, like, was always a goal to build like a real product here. And, but for us that that was kind of a very key moment, very key milestone for us. And so yeah, now it's, you know, it's very much a real thing.

You know, we're as mentioning talking before, you know, Jules and the, you know, being talked about on the in the GitHub keynote. It's yeah. It's certainly here to stay.

What do you wanna know? Yeah. Yeah.

Yeah. Well, tell me. Why why would someone wanna go yeah.

To This is yeah. Let's turn it around.

you had to. Yeah. Of course.

So, you know, let's say I've watched I've watched the videos online, I kind of get a sense of respect people. What's happening between that for someone who hasn't been before? Like, what goes on other than the talks?

Like Oh.

AV. An attendee somewhere. Go.

I'm gonna sit Oh, yeah. Sorry. Ask.

So one out of 23 people who Yeah.

to meet and and have like high bandwidth conversations. Yeah. Are you and me gonna do the autonomous Coding Agent dinner?

and it seem like their margins are better, so why not? Yeah. Totally.

I mean, I think for us, it's certainly been you're the agent. Like as the models, you know, what was you talking about? What is let's build Jules for where things are going.

And as the models get better, think it just becomes clear and clear that agents are super powerful. You know, like we have you were talking about like before high context and management, so all that stuff's important. Like we have people we had this is a funny story.

We store some data for a session, but it only lasts we only store it for thirty days. And so after thirty days, your session becomes locked. And when the first user starts first started hitting that, they were upset.

We're like, there's no way anyone's gonna be using a single session for thirty days. Like, they would do a single track of work for thirty days. But just like how powerful that could be.

So And how do you compress context when you run into it? Yeah.

use up your 2,000,000 token context window. Is it is it 2,000,000? It is up to 2,000,000.

Yeah. Especially for coding agents because like Yeah. Know, like, you're reading files, like, it's so you're you're running commands with huge outputs.

Like, I think coding agents are a really interesting area, both product wise and the impact they're having, but also for research. They really push the limits of, you know, what other domains are you running an agent for thirty days? And what other domains are you queuing so much context and so many turns?

It's yeah. It's coding agents are, I think, kind of a special spot of like super interesting product, impact, research. Yeah.

a good pattern. I mean, also has this challenge is like how do you make sure enough stuff, your information is going back and forth, but that's the part, you know, just the summarization is a pattern, you know, like kind of externalizing some of that context, whether it's like writing it to, you know, like a note kind of thing is a common pattern.

methods compare better. Yeah. It's also interesting like, as models change, like, the answers change a little bit too.

Yeah. Yeah.

But also you're not the only consumer and they have other priorities than you. Yeah. Totally.

Totally. I mean, I think we're lucky in kinda how we're positioned to have very close relationships with with DeepMind. So we have and, you know, coding agents are an important area.

Like, let's be honest, right? Like for any kind of company building models, like you can see it in all the labs, like coding agents are important and coding capabilities are really important. Yeah.

Yeah. It's important from a kind of AGI perspective. It's important from a dollar's perspective.

It's important for all of it. So it's I think we're in a really lucky position where we have, we're able to have a lot of kind of good collaboration and but both ways, you know, like all kinds of capabilities that are being developed. And, you know, it's interesting, it's a whole host of things, right?

Because, you know, in terms of like AGI and the capabilities of things, it's also like computer use models and browser use models. And so it's just, know, models that output code, but it's also the whole suite of, you know, things that you want an intelligent agent to be able to do.

and, you I don't think of this as a zero something. I think this is like, really like there's this tide that's gonna lift all of our boats and it's, we're inventing a new way to do our art, right? And how to create good art as a software engineer.

And so what does that look like and how does that feel? What is that, you know, what is the experience we wanna create? I think as as people working AI, sometimes we don't do a good enough job describing this beautiful future we're creating.

I mean, I know, you know, like the CEOs and heads of these labs have started like, you know, writing their think pieces on this, but right? You know, for software engineers, like, is this beautiful future we're creating? And like, you know, I think that's like, one, it's it's inspiring.

It makes it, you know, maybe less scary for for people who are who are thinking about these tools. But also like, you know, if we can't articulate it and think about it, it's less likely we'll get there. Right?

So like, what is this, you know, great place we wanna create? Like writing software is so hard. Like in so many companies, it's such a especially big companies, it becomes so challenging to manage a code base and create.

And what can we do to make, you know, being a software engineer, absolutely incredible experience. What are these, you know, how do you want to interact with your model? How do you, how are you doing things locally versus in the cloud and how does that interop?

And so I think like as an industry, we're trying to like, you know, which is changed. Like we're inviting, in some ways, inventing and there's this movement to, you know, change how we do our art. And yeah, the more, know, the better we can create this experience, like we all win to some degree.

So, yeah, I think that'd be one thing where it's like yeah. Yeah.

And

I don't if you have any take on that stuff. Yeah. It's only so I mean, something we spend a lot I spend a lot of time talking and thinking about with with folks.

Know, I'll just spend time talking to people at companies and, you know, I think sometimes working on these tools, it's interesting to see it's not as like diffused. This technology isn't as diffused across software engineers as I sometimes expect. Right?

There's plenty of places that I think are not really using AI a ton. A lot of companies, lot of software engineers aren't. That being said, I'm very kind of excited about what the future of software engineers are.

Like, could you imagine going back to not having these tools? No. That sounds horrible, right?

Like that. And so that's one aspect of it. I also think, you know, I don't really buy this like, you know, that we're not gonna hire more software engineers story.

I think like for a few reasons, I mean, is an example that often comes up, but is it like kind of the elasticity of the demand for software? Yeah, Jevan's paradox. Exactly.

And you know, like a lot of the cases sometimes come up as you look at like farming, right? And so, you know, there was a time in America where like the vast, vast majority of Americans were farmers, right? And then technology happens and today it's like less than 1%.

Yeah. And that's one example. But the flip side of that is your electricity, which like, as that gets cheaper and cheaper, people just consume more and more and more electricity.

And with food, there's only so much food we're gonna eat. Right? There's there's a kind of a there's an inelastic demand for that.

Whereas, like, a very elastic demand. It seems like software, you know, software keeps getting better and better. Like the ability, like, creating more and more software from like, obviously, like punch cards to to where we are today is like remarkably different in terms of how you're able to create software.

So much more software is being made. And software just keeps becoming more and more of our GDP. Right?

Like it's it's a so I'm I'm bullish on kind of the the amount of software we'll be able to create, how it'll be created. I think there's also something here about, you know, as an engineer, being able to be more productive, like, encourages more investment in people building software, right? If it's, you know, the job of a software engineer can now, you know, they can do 50% more, a 100% more, 10 X more, like justifying investment dollars into projects, like dramatically changes, right?

And so, yeah, I'm bullish on this idea that it's actually gonna be great for softener, both for our ability to kind of do our craft or art, but also just what it means for the number of companies and the amount that's made and the quality of it and what we're able to do with it. So, yeah.

that we can actually trust. Yeah. What is it?

Well, they And it's the punch line right now. What is it?

and and then you correct it from there. Yeah. Mean, one thing we talked about, which very early on is what you're thinking is like, they're kinda like two problems as these things get better.

One is like, how do you specify what you want? And the other one is how do you verify that what you got is what you're thinking? Yeah.

So yeah, whether it's, you know, specifying through a spec or this like, you know, interactive plan or whatever it is, but then yeah. And then on the flip side with the vibe coding thing is you might specify, but you never come back and verify. Right?

You're like, you're it's more hands off the wheel. Like, maybe I'll click around the app a little bit and see how it works, but it's I'm not really engaged with the code. So how do you yeah.

How are you verifying and making sure that it's, you know To my knowledge, you guys don't emphasize tests that much. Right? It's not like you volunteer to write my tests.

Yeah. I mean, it depends it depends like we if there are tests in your code base.

But it's not like it's not like, you know, after everything, everything must have a matching test to the prompt that was mentioned, you know. No. That would the extreme of what we mentioned.

Don't know if people always want that. I mean, maybe it'd be helpful to do that to kind of show that was right. But let's say I don't write tests in my code base.

Like, I wanna merge that pull request that is introducing tests just for this one thing. Like, you know, I think in in some ways, the the engineer should be able to control what kind of outputs they want. Yeah.

If it helps and they want it, you know, absolutely.

Oh, totally. Totally. I mean AgentsMD.

Yeah. Agents, I mean, Spectrum development, think is in this this category. I think one of reasons is like multimodal, right?

Like, you know, if I'm going show you a bug on our website, like, do I want to come and like type it with words to describe it? Or am I going to point the picture? Yeah.

And so, you know, with Jules, you can upload images now, but, you know, kind of more, you know, we have certain ways we communicate as humans that are easier in certain situations and other inputs.

yeah. Definitely.

and speed and cost and like, you know, affordable cost. Yep. It's a lot.

It's Yeah. Know this is I mean, what kind of Crunches are fun. There's just so much to build.

There's so much, you know, think also as a software engineer working in this space, like, think one of the reasons we, you know, you see so many companies in this space is partly like, it's just so fun. Like there's so many things to build. There's so many tools that seem like, you know, fun sci fi.

Like there's it brings up a demo of what I've worked on.

What else should they also talk to you about? Like, you know, what what what do you what can you help with versus what are you looking for? Anyone should feel free to come up and talk to me, you know, at any point.

I you know, obviously very interested in anyone who's doing stuff with coding agents or someone who's using coding agents in an interesting way. So I'm always curious about, you know, workflows people have with their dating agents, whereas, you know, whether it's, you know, hey, I'm using this tool in this way and I've, you know, configured this crazy thing. Like, I always love hearing how people are using it.

I also love hearing people who are having bad times with it, where it's like, actually, don't, you know, maybe they're not coming to this conference, but, you know, Jeff Charlie's tools, and I don't like them, and I know he use them, and here's why. Yeah. So, you know, I'm totally open for any side of the all the way from, you know, full AI pill and coding AI lovers to people who hate it.

As far as what I'm looking for, you know, I think, you know, really just going to kind of connect and meet people. I think, you know, are always hiring. So like, you know, I'm anyone who's, you know, interested in working on this stuff, I'm always happy to talk.

But yeah, really just kind of, you know, meeting people, spending time, geeking out on this stuff. Yeah. There'll be lots of geeking out.

Yeah. Alright. Thanks for your time.

Looking forward. Yeah. Same.