DevDay 2025: Apps SDK, Agent Kit, MCP, Codex and why Prompting is More Important than Ever

and this is, like, the first time that's been, like, so well organized that we have our own little studio podcast studio in in the Dev Day venue. And it it's really nice to actually get a chance to sit down with you guys, so thanks for taking the time. Yeah.

I feel like we Dev Day's always a process, and, like, we only had three of them, and we try to improve it every time. And Yeah. I actually I I know for a fact that I think we have this podcast studio this time because the the podcast interviews and the interviews with folks like yourselves last time went really well.

And so you wanna lean into a little bit more and glad that we're able to have this studio for y'all. We we were kneeling on the ground interviewing, like, Michelle last year. I don't know what you're saying.

in my mind. I don't know about you. Yeah.

And we were at the OG Dev Day, which was the DALI hack night at OpenAI in 2022, and think Sam spoke to, like, 30 people. So I I think it's just crazy to see the Yeah. Honestly, I I think it's, like, it's kinda similar to this podcast studio, which is I think we've had a number of Dev Days now.

We're we honestly were, like, slowly figuring things out as a company over time as well and both from a product perspective and also from how we want to present ourselves at Dev Day. At this point, we've had a lot of feedback from people. I actually think a lot of attendees will get an email with a transfer feedback as well, and we actually do read those and we we act on those.

And, like, one of the things that we did this year that I really liked were all of those, like there was, like, some art installations and, like, the the little arcade games that we did, which was, you know, came up with an via, like, engaging with the feedback from the Yeah. The arcade games are so fun. I love, like, the theme of all the ASCII art throughout.

So the way that I think about it is I actually view OpenAI since the very beginning as a company that has really valued opening up our technology and bringing it out to the rest of the world. One thing we talk about a lot internally is our mission at OpenAI is to, one, build AGI, which we're trying to do, but two, you know, potentially, you know, just as important is to bring the benefits of that to the entire world. And one thing that we realized very early on is that we, as a company, it's very difficult for us to just bring it to every truly every corner of the world, and we really need to rely on developers, other third parties to be able to do this, which is, you know, Greg talked about the the the start of the API and, like, kind of how, you know, that was formulated.

But that was part of, you know, that mentality, which is we needed to we need to rely on developers, and we we need to open up our technology to to the rest of the world so that they can partake for us to really fulfill our mission. So the API, obviously, is a very natural, you know, way of doing that where we just literally expose API endpoints or expose tools for people to to build things. But now that we have, you know, ChatGPT with its, I don't know, like, 800,000,000 weekly active users, I forgot the stat that we Yeah.

Shared. I think it's, like, now the fifth or, like, sixth largest website in the world. And the number one and number two most downloaded on the Apple App Store.

Oh, yeah. With Sora. Yeah.

But that one, like, it moves around all the time, so it's kind of hard to celebrate and, you know, like You just screenshot it when it's good. Yeah. Yeah.

We definitely screenshot it and shared it earlier when it was when it was good. But kind of going back to my main point is, like, we've always kind of engaged with developers as a way for us to bring the benefits of AGI to the rest of the world, and so I view this as actually a natural extension of this. Candidly, we've actually been trying to do this, you know, a couple of times with last Dev Day with GPT, two Dev Days ago with, I'm sorry, two Devs ago with with GPTs and plug ins.

Plug ins, which was I think not tied to a Dev Day. So I view this as like, again, we we love to deploy things so iteratively, and I view it as like just a continuation of that process and also engaging deeply with developers and helping them benefit from some of the stuff that we have, which which in this case is ChatGPT distribution.

I think, like, in March is when we added in MCP to Agents SDK first and then soon after with kind of our other Yeah. I think there was, like, a tweet or something we did where it was, like, OpenAI, you know, is Yeah. There was definitely a moment.

I think there was a specific moment in a specific tweet. But what I will say, though, is, like and this is honestly, like, credit to the team at Anthropic that that kind of created MCPs. I really do think they treat it as an open protocol.

Like Yeah. We work very closely with, I think, like, David and and the folks on the, like, you know, consortium, and they are not, you know, really viewing it as this, like, thing that is specific to Anthropic. They really view it as this open protocol.

There is like, it is an open protocol. The way in which you make changes feels very open. We actually have a member of our team, Nick Cooper, who is sitting on kind of like that that steering committee for MCP as well.

And so I think they are really treating it as something that is easy for us and other companies and everyone else to to embrace, which I think they should because they they do want it to be something that is very embraced by all. And so because of that, I think it makes it a little bit easier for us to to embrace it. And, honestly, it's a great it's a great protocol.

Like, it's very general. It's already solved. Like, why would you make Yeah.

Yeah. It's very general. There's obviously still more to do with it, but it was very easy for us to, you know, integrate because of how how how how streamlined and how simple it was.

Yeah.

That is an experience I've never seen. Yeah. And I think that that's a kind of back to the iterative, like, learning that we've had.

That I think was because we've learned a lot from plug ins. So, like, when we launched plug ins, I remember one of the feedback that we got. I don't know if if, you know, people here really remember plug ins.

It was, like, March Oh, yeah. '23. Yep.

I'm like, one of the points of feedback was, oh, you can integrate we tell we told, like, you know, these companies that you can integrate these plug ins into ChatGPT, but they really didn't have that much control over how exactly it was used. It was really just like a tool that the model could call, and you were just, like, really bound by ChatGPT. And so I think, like, you can kind of see the evolution of our product with this.

And, like, this time, we realized how important it was for companies, for third party developers to really own and, like, steer the experience to make it feel like themselves, help them, you know, like, really preserve their own brand. And so and, you know, I actually don't think we would have gotten that learning had we not, you know, had all these other steps beforehand.

I don't know how much time I killed on that. Was extremely frustrated with the download screen. Was like, the UI bug is what, like, takes the demo down and be slow.

Yeah. Think it was a full screen, yeah, like, focus thing. But I heard the window wasn't in focus or something.

Yeah.

That that those are the main moving parts. Right? Yeah.

I think just a lot of flexibility around. The the one thing I'd say too is a lot of the demos that we showed today, I think were, like, you know, aired on the side of simplicity just so that the audience could kind of see it. But, like, if you talk to lot of these customers, like, they're building, like, pretty complex.

Like, you got to, like, zoom out on that canvas quite a bit to kind of, like, see the the full flow. And that and then for us, we, you know, we were kind of, like, working with a lot of customers who are doing this. And then, you know, if you turn that into, like, an actual agent's SDK, file, it's, like, pretty it's pretty long.

And so we saw a lot of, like, benefit from having the visual setup here, especially as the as the setup grows grows longer and longer. It would have been a little difficult to kind of showcase this, but even on, like, the minutes. Right.

Yeah. You can do it in eight minutes. But, like, even with some of the presets that we have on the Yeah.

I think we we've thought about it, especially around, I'd say, agents SDK. I would actually say maybe even, like, zooming out a bit more from from just this is, like, yeah, we we, like we're also sitting here and kinda, like, observing, like, things being made over and over again. Even, like, besides, like, agent workflows, we're kind of launching what the industry is trying to do with responses, like, we've done with responses API, like, stateful APIs.

And so, you know, obviously, we were the first one to launch responses API, but, like, couple of other other people have kind of adopted. I think I think Grock has it in their in their API. I think I saw LMSYS just did something recently in Wells, but not, you know, not everyone.

And so, unfortunately, I don't have a great answer today of, like, yes or no, but we are kind of, like, assessing everything and trying to see, like, hey. You know, there's a there has been a lot of value with MCP, with hopefully, our with our commerce protocol as well. ACP, yeah, that's the I definitely did not forget the name.

Mhmm. Where we're working with them, and then you can bring your OpenRouter setup. And then with that, you can actually you know, you write your Evals, using our datasets tool or use our dataset tool to create a bunch of Evals, and you'd actually be able to hit a bunch of different model providers, you know, take your pick from wherever, even, like, open source ones on together, and see the see the results in our product.

Yeah. That's awesome.

Yeah. I can't answer. Yeah.

I I was gonna say, so the I actually think agent evals is still a work in progress. So I think we've, like, made maybe 10% of the progress that we need here. For example, I think we could still do a lot more around multimodal Evals, but the main progress that we made this time was kind of allowing you to take traces, so the Agents SDK has really nice traces feature where if you run, if you define things, can have, like, a really long trace, allowing me to use that in the eval's product and be able to grade it in some way, shape, form over the the entire the entirety of what it's supposed to be doing.

I think this is step one. Like, I think it's good to to be able to do this, but I I think our road map from here on out is to, you know, really allow you to break down the different parts of the trace and and allow you to eval and, like, kind of, like, measure each of those and optimize each of those as well. A lot of times, this will involve human in the loop as well, which is why we have the the human in the loop component here too.

But if you kind of look at our Evolves product over the last year, it's been very simple. It's been much more geared towards this, like, simple prompt completion setup. But, obviously, as we see people doing these longer genetic traces, like, you know, how do you even evaluate a twenty minute task correctly?

And it was like, this is a really hard problem. We're trying to set up our Evalse product to move in that way to to help you not only evaluate the overall trajectory, but also individual parts of it. Yeah.

I mean, the magic keyword is rubrics. Right? Everyone Yep.

I actually think it's a really cool time right now in prompt optimization. I'm sure you guys are seeing this too. It's like, not only are a lot of products kind of, like, gearing around this, so, like, kind of what we're we're thinking about, but I also think, like, there's a lot of interesting research around this, like GEPA with, like, the Databricks folks are actually doing really cool stuff around.

This. We're obviously not doing any of the the cool GEPA optimization right now in in our product, but would love to would love to do that soon. And, also, it's just an active research area.

So, like, you know, whatever Matej and the Databricks folks, like, might think about next, what we might, you know, think about internally as well. Whatever new prompt optimization techniques come out, I think we'd we'd love to be able to have that in our in our product as well. And, yeah, and and it's interesting because it's coming at a time when people are realizing that prompt.

You know, like like, I feel like two years ago, people were like, oh, at some point, like, prompting's gonna be dead. No. Like Yeah.

You know? And it's like, you know It's gone up. Yeah.

Yeah. Yeah. And then if anything, it is, like, become more and more entrenched.

And I think that, you know, there's this interesting trend where, like, it's becoming more and more important, and then there's also interesting cool working done to, like, further entrench, like, prompt optimization. And so that that's why I just think it's, like, a very fascinating, you know, area to to follow right now and also is an area where I think a lot of us were wrong, two years ago because, if anything, it's only gotten more important. Yeah.

and, like, it is extremely difficult for us to run, you know, and serve, like, all of these different snapshots. Like, you know, Laura's great. MSL just, you know or sorry.

ThinkingLab just just published John Truman just had a cool blog post about this. But, like, man, it is, like, pretty difficult for us to, like, manage all of these different snapshots. And so if there is a way to, like, hill climb and, yeah, do this, like, zero gradient like optimization via prompts, like, yeah, I'm all for it.

And and I think developers should be all for it because you get all these gains without having to do any of the Yeah. You know, fancy fancy fine tuning work. Since you are part of the API you know, you lead the API team and since we you mentioned Thinky, I gotta throw a cheeky one in there.

So yeah. It's a good one. So it's actually funny when it when it launched, I actually DM'd John Schulman.

I was like Really? Wow. We finally launched it.

So the because you used to work with him. Yeah. Yeah.

So we it's it's actually funny. So at yeah. So right right when I joined OpenAI, like, this has actually been, I I I think, a a passion project of John's.

Like, he's been talking about doing something in this, like, in in this shape for a while, which is, like, a truly, like, low level research, like, fine tuning library. And so we actually talked about it quite a bit when he was at OpenAI as well. It's actually funny.

I talked to one of my friends who said that when he was at Anthropic, he also, you know, worked on this idea for a bit, and I think now He's a man on a mission. Yeah. I mean I mean, John's, like, so great in this regard.

He's he's, like, so purely just, like, interested in the impact of this because it's one, like, a really cool problem, and then, two, it also empowers builders and researchers. Like, you saw all the researchers who who, like, express all this love for Tinker because it is a really great great product.

It's actually funny. Yeah. I I I I heard this quote before, and there's definitely some truth to it.

I I don't know if our stateful APIs have become a database. I guess quite yet, but, like, who knows? Like, you know Yeah.

I mean, conversation Well, you charge for it. You charge for assistance.

like, transforming it all into this shape. It's like the It's useful. It's easy.

Best place to order. But yeah.

like, companies to kind of manage what their, like, developers have access to, managing kind of the configurations around it. And I think in terms of, like, first party versus third party, like, we wanna support both of those. We have some direct integrations, and then, I don't know, anyone can kind of create MCP servers.

I think we wanna make that a lot easier too, like, establish kind of private for for companies to use those internally. So I think, like, just really excited about that ecosystem growing. Yeah.

I I think one of the coolest things observed too is just I I actually think we we as an industry are still trying to figure out the ideal shape of connectors. So, I mean, part of why I think the one p connectors exist too, like, we we end up storing quite a bit of state. It's like a lot of work for us.

But, like, by having a lot of state on our side, we call them sync connectors, we can actually end up doing a lot more creative stuff on our side when you're chatting with IGBT and using these connectors to to kind of boost the quality of of how you're using it. Right? Like, if you have all the data there, you can do all this, like, re ranking.

You can, like, do we can put in a vector store if you want and put it anywhere else. Whereas and then so there there's some inherent trade offs here where, like, you you put in a lot of work to get these, like, one p connectors working. But because you have the data, you can do a lot more and get higher quality.

But then but then the question is, oh my god. There's, like, such a long tail of other things, which is where the MCP and, like, the third party connectors come in. But then you have the trade off of, like, you're beholden to, like, the API shape of the MCP creator.

It might actually work well, it might not work well with with the models. And then what happens if it doesn't work well? Then you kind of have to, like, you know, you're kind of, like, at the mercy of this.

And MCP, by by the way, is, like, really great because it already does some layer of standardization, but my sense is there's still gonna be more evolving here. And I think, you know, we wanna support both of them because we see value in both right now, especially working with working with developers who wanna have kind of, like, all options kind of on the table here, but it will be interesting to see how see how this evolves over time.

Yeah. It's interesting because so so direct answer is like no plans right now, of course. But I actually think we currently have some version of this, which is our partnership with Apple.

Because with Apple, you can actually sign in to your ChatGPT account, and some of that identity does carry with you into your iOS experience with Siri. Right? Like, if you if you I don't if you've actually used this the the Siri integration.

I I actually use it quite a bit. But if you sign into your TryGPPD account, the Siri integration will actually use your subscription status to decide what type of model to use when passes things over to ChatGPT. And so if you're, you know, just a free user, you get, you know, the free model.

directly there. Yeah. Mean, Sam's been talking about it for a while.

It's a very compelling vision. We obviously wanna be very thoughtful with I mean kinda how we do it. Know, now you have a social network.

You have a developer platform.

Yeah. But I think, yeah, it's definitely an interesting area too. Yeah.

It doesn't do it out of the box today, but, you know, developers have been asking about it for forever. Like, it's a really cool concept because then as a developer, you especially a new developer, you don't need to bear the burden of of inference. Yeah.

Yeah. I will say in general, like, if you kinda look at our road map, we we we engage a lot with developers. We kind of hear what is the are the pain points, and we try and build things that address it.

You know, ideally, we're prioritizing in a way that's that's helpful. But, yeah, we we've definitely heard from a good number of developers that, like, the cost is or, like, all of the, like, copy paste your key, like, solutions right now, which are, like, huge security ish like, hazards because developers don't wanna bear the burden of of inference. You know?

Hopefully, we make the cost cheaper. So It's a model skip getting cheaper. Yeah.

So, hopefully, you know, that that helps. But, but but what we realized is as we make it cheaper, you know, the demand for that goes up even more and you end up, you know, still spending quite a bit. But, yeah.

It is interesting too. Like, from a deployment perspective, it is like, it it has spanned both internal and external use cases. Right?

Like, kinda like these internal platforms, people use it for end like, data processing or something, which is an internal use case. But if you saw some of the demos today, like, there have been a huge number of companies that are trying to do this for external facing Mhmm. Use cases as well.

Customer service is one template. Customer service, the, like, ramp use case. We use this internally and externally.

Like, our customersupporthelp.open.com already powered on agent kit and then various other, like, internal Yeah.

Use cases as well. And one of the things that I actually think the team has done a really great job of so, like, Tyler, David, and Jiwon on the team, They built the the especially the the chat kit components. They built it to be, like, very consumer grade and and, like, very polished.

Like, you kinda look at the there's, like, a whole grid of, like, the different widgets and things that you could create there. Like, ideally, people see it and they're like they see it as, like, these very polished, like, consumer grade ready external facing things versus, like, you know, I think of internal tools and, like, the UI is always, you like, the last thing that people care about. But, like, you really, you know, push the team, and I think they did a really great job of making the ChatKit experience, like, really, really consumer grade.

And it should feel almost like ChatGPT or Yeah.

Yeah. And when working with customers, like, those have been the the most helpful customers for us to work with because, you know, when Ramp is thinking about, you know, how what what they want to publicly present to people, like, they have a pretty high bar, as they should, as as as well as, you know, all the other customers that have been iterating on it. And so that kind of feedback from our customers has really helped us uplevel the general product quality of of the launch that we had today as well.

Yeah.

Talked about it. Uh-huh. We've talked about it.

There are a bunch of trade offs.

Actually, don't know how we got that. But Yeah. Everything's in chakka.

No. But, like, I think especially on this stuff, I think there was, like, an arc of a a general, like, you know, platform that we did want to kinda build around, and it takes a while to to build these things. Obviously, Codecs helped speed it up quite a bit now, but it yeah.

I will say it does seem great to kinda, like, start start to have all the pieces start fitting together. Yeah. I mean, you saw we launched Evals, and we got the fine tuning API for a while, and and we laid all the groundwork for for some of the stuff over the last year.

And we're hoping that we can eventually, you know, make it into this this this full featured platform that that that's helpful for people. I think you have.

The so there's there's actually a a funny one that one of the new grads has has, I think, like, taught our team in general. And I think this is, a a point for, like, just how, like, new grads and and younger, you know, generation people are actually more AI native. So one of them is to, like, really lean in to, like, like, push yourself to, like, trust the model to do more and more.

So, like, I feel like the way that I was using Codecs, and so for me, it's most usually for my my personal projects, they they don't let me touch the code anymore. But you give it, small tasks. So you're, like, you're you're, like, not really trusting it.

Like, I I view it as, like, this, like, intern that I, like, I really don't trust. But what a lot of the, like so we had an intern class this year. What a lot of the interns would do is just, like, full YOLO mode, like, trust it to, like, write the whole feature.

And it, like, it doesn't work work for worse. It, like, doesn't work sometimes, but, like, I don't know, like, 40% of the time, it's just, like, one shots it. I actually haven't tried this with, like, codec g b d five codecs.

I bet it I bet it probably, like, one shots it even more. But one tip that I'm, like, starting to, like I feel like undo this like like, relearn things here is to, like, really lean into, like, the AGI component of it and just, like, really let the model rip and, like, kinda trust it. Yeah.

Because a lot of times, they can actually do stuff that surprises me, and then I have to, like, readjust my priors. Whereas before, feel like I was in this, like, safe space of, like, I'm just treating this I'm giving this thing like a tiny bit of rope. Yeah.

And and because of that, I was kinda limiting myself with how effective I could be. Like, sure, but okay.

You can now go into your usage or your settings account and see the health of your integration with our OpenAI API. This is scoped to your own org. Basically, if you have an integration that's running with us doing a bunch of tokens per minute or queries, it's now tracking each of those responses, looking at your token velocity, TPM that you're getting, the throughput, as well as the responses, the response codes.

And so you can see kind of like a real time personal SLO for your integration. The reason why I care a lot about this is obviously over the last year, we've spent a lot of time thinking about reliability. We had that really bad outage last December, you know, longest, like, three, four hours of my life, and then had to, you know, talk to a bunch of customers.

We haven't had one that bad since, you know, knock on wood. We've done a bunch of work. We have an Infrared team led by Venkat, and they've been working with Janna on our team, and they've just been doing so much good work to get reliability better.

And so is we actually again, knock on wood. We're we think we've got reliability in the spot where we're, like, comfortable kind of putting this out there and and kind of, like, letting people actually see their their SLO. And hopefully, you know, it's, you know, three, four, soon to be five nines.

But the reason why I cared a lot about it is because we spent so much time on it, and we feel confident enough to kinda have it behind the product now. Five nines is, like, two minutes of outage or something. Yeah.

Yeah. We're we're working we're working to get to to five nines. Yeah.

What is what is an extra nine take? It's it's exponentially more work. So, you know, and then but, like, we always we were you know, in the last couple weeks, we're talking about, like, hitting three nines and hitting three and a half nines and then hitting four nines.

But, yeah, it's it's exponentially more work. I could I could go for a while on the on the different different topics. But We'll have to do that in a in a follow-up.

I mean, mean, that's all that's the engineering side. Right? Yes.

Yes. Yes. Like, you're serving 6,000,000,000 tokens per minute.

We actually zoomed past that. Yeah. That's the that's the That's outdated.

Yeah. But, yeah, it's been crazy, though, the growth that we've seen.

We have the mic software. The thing I I I didn't want to say on the podcast was on the on the Tinker thing.