⚡️The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals & Human Data
Latent Space: The AI Engineer PodcastEpisode Description
Summary
This episode discusses OpenAI's decision to no longer evaluate models on SWE-Bench Verified, a key coding benchmark, due to its saturation and high contamination. Mia Glaese and Olivia Watkins explain the extensive human effort behind creating SWE-Bench Verified and the issues discovered through deeper analysis, such as overly narrow tests and model contamination. They advocate for moving towards harder, more diverse benchmarks like SWE-Bench Pro and developing new evaluations that measure complex, open-ended coding capabilities, design taste, and real-world impact within OpenAI's preparedness framework.
Chapters
Topics
People
Key Concepts (7)
References (11)
Transcript (47 segments)
Okay. Hi. We're here in the OpenAI studio with Mia and Olivia from the Frontier Evals team or however you want to introduce yourself.
Maybe Mia want to introduce, name what you do at OpenAI and we can get it started. Sure. Hi.
I'm Olivia. I'm on the Frontier Evals team. Are you sure to meet?
Hi. I'm Mia.
and my teams are the Codex team, the Schumedator team,
and the Alignment team, and we work a lot with Olivia's team on Frontiiva. Yeah. Very exciting.
And as by my understanding, you were part of the original team that worked on CVatch Verified as well. Yeah. Olivia's team, the Frontiiva team, and the Schumelator team collaborated on creating SuiteVench Verified.
So you've you've seen the evolution of coding benchmarks over time, and I I think it was roundabout to the mid to late twenty twenty four when we first published SuiteVench Verified. Things have evolved a lot since then. What's the blog post that you have worked on that you're that we're releasing today?
Like, what what is the sort of con what's the main thesis that you're pushing out? So the main thesis is that SuiteVench Verified has been one of the North Star coding benchmarks that the field has looked at to measure coding progress. But recently, we've seen that progress has kind of stalled.
And this realized that this is because the eval is effectively saturated and also highly contaminated.
So at this point, we think that it's not really measuring coding performance improvements well anymore, and we think that the field should move away from this towards other benchmarks. Like Superbench Pro. Like Superbench Pro.
Yeah. Amazing. Yeah.
and everyone just takes turns to increment, like, 0.1 on trucks. And then it's like, okay.
Well, you have the best coding model, I guess, because you're 0.1% higher, but it's not super convincing at this point. No.
Yeah. No. So cool.
I think the let's let's sort of reset on, like, what was your original work that you guys did for SuperVise Verified, which I think was pretty substantial. Like, it was, like, a very significant investment from OpenAI, which, like, people still don't appreciate. And then what were the dissatisfactions that we that we found over time?
Right? So, like, what what was SuiteBench Verified that should should that people should know about?
bench academic benchmark from a lab at Princeton called SuiteBench. And the agent is basically given a code base and a task that was sourced from a real world repository and GitHub issue and was asked to solve the task and is graded on whether some tests pass. And at the time, this was quickly became a popular benchmark because at the time, the field didn't really have good real world coding benchmarks.
But then when OpenAI took a look at the benchmark as part of one of the evals we wanted to track in our preparedness framework, folks started realizing that some of the cases where agents were failing were due to bad problem setups rather than just to models being dumb. So Voxo OpenAI did a pretty extensive human data campaign, hiring, like, almost a 100 real world software engineers to go through the problems and figure out, like, are the tasks well specified? Are the tests actually fair?
And kind of created a curated set of, like, 500 tasks that we thought were much better. It's just it's maybe it's hard to to overstate, like, the amount of effort that it took to, like, create that benchmark.
It was literally, like, many expert software engineers reviewing the problems, like, sequentially, multiple times, and to, you know, basically, like, see different experts independently decide of it. Yeah. You didn't have to do that.
You just tripled your cost for just I mean, we had to do it. We had to do it, actually, because it's quite a hard task to, like, look at something, like a a problem and and the patch, and then, like, it's not just the problem and the patch. Right?
You have to, like, understand it in the context of the code base that the human or or the models and to solve the task. So it's a very complex problem, and it was definitely needed to have three reviews.
a lot of effort to get there. Yeah. And there's there's more, but people can read the the blog post for that.
I will note that you guys had a trend in verifying benchmarks because I just recently saw I think Quinn had a HLE verified for Humanities License verified. Yeah. Which, like so now everyone's verifying everything, which is nice and good and, like, extra quality there.
Okay. So but the I think that the meat of it is that that this this was a lot of, like, well, here's the issue or problem statements, and then here's the here's the divs, here's the golden tests, and here's some regression tests. Right?
That's that's like the rough setup of these 500 problems. And there's some contamination always happens because all the Submitry Verify was fully open, I think. You you did have canaries, but, like, you know, stuff stuff leaks.
There's, like, multiple avenues that, like, the problems are sourced from open source repos. Yes.
when we usually publish evaluations, we publish evaluations and then we we add canary strings to ensure that, you know, they are easily filtered out at training time. Obviously, if you use sort of, like, data from, like, open Just quick GitHub.
You don't have actually, like, a cannery string in there. And you And these are also like, some of these are very popular repos, like the Django repository.
So you're gonna see, like, many intents that are being used kind of throughout Go. Yeah. And you just before recording, you were telling me that you found this in your own chain of thought with, like, g 5.
also seeing that, like, they had extra knowledge or something. Yes. So this was an example where the task asked the agent to influence something, but it wasn't told that there was this specific argument that the test was going to be looking for at using.
But in the GPT 5.2 chain of thought, we actually saw instances of the model reasoning, like, hey, I think that it's some language or version of this repository that implemented this particular argument. Maybe I should add it in.
Yeah. So this is an example of a test that, like, would be pretty impossible to pass without this contamination knowledge. Yeah.
other frontier models, like, in the market and, like, understanding how contaminated the benchmark is, like, across the industry.
What else did you find? I mean, let's I have to double click on this. So we when I say we, this is mostly we are from other folks on our tape, not I don't wanna take the wrong way.
Yes. But so we did some analysis on, first of all, are the tests actually fair? And so this happened by, first, taking all the problems that o three couldn't solve reliably, and then, again, getting a lot of humans to do basically another pass of kind of digging into, you know, what's wrong.
Is it the same exact analysis, or were they reading o three's output and going here's where o three went wrong? I think it was I mean, it was definitely, like, a scope to the set of problems that models failed. I believe they were able to look at, like, what the model solutions look like versus what the So this isn't the same work as the original set of It's not exactly the same work.
It was like a a deeper dive. It's like, okay. Which are the problems that we don't see any murderers solving?
It's like, if there's something fundamentally wrong with those problems, or is there something, you know, wrong with they are the murderers just not smart enough to solve the problems. So that's kind of, like, what we what we dug into. Yeah.
And you found some? Oh, yes. Like, in over half of the problems that were investigated in that deep dive, there was one problem or the other.
that the tests were looking for but wasn't specified in the problem description. So it wasn't fair to expect that model to make that particular design choice. Like, one pretty blatant example are cases where the task asks you to implement some feature, but the tests are looking for you naming that argument or that function with a particular name.
But if you chose another reasonable name, the test would fail. Yeah. And another set of types of bad tests are tests that are just looking for additional features that were never mentioned in problem description.
Wait. Wait. That's a significant effect.
That means that if you pass the test, actually, like, you probably did, like, a really good job. But just because you didn't pass the FET test doesn't mean that your implementation wasn't, like, a good one. Right?
So it was just like, we only accept, like, very narrow versions of solutions and, like, not the whole space Yeah.
and sort of, like, good solutions to the problem. Yeah. I think it's important that you're doing this because it in some way, it is you, in twenty twenty five, six, going back in time and correcting your own work.
Right? Because you could have caught all this in in the original verified work. I think so.
find a problem in the abstract than when you're looking at a very smart agent's best effort solution and try to compare it. Is harder or harder or else It's much easier when you have Much easier. Exactly.
Sorry. I think I think also, like, at the time when three bench verified was published, I think it was, like, a very strong benchmark. It's not like we're we're, oh, this is not this wasn't like a strong benchmark at the time.
I think this is something that a lot of benchmarks go through like as an evolution. Right? Like when they start to become like popular and like viable because they measure something like important and models maybe do, like, 20% correct on them, sometimes even less.
And sort of, like, people have something to hold on and and improve models on on these benchmarks. And by the time that you hit, like, very high performance on the benchmarks, like, additional like point 1% improvements, it's become sort of like meaningless. And so like at the time, I think, you know, that benchmark was like super valuable and it it taught like us and like the industry a lot.
not necessarily like what we want to measure, which is like coding capability of our agents, but like the agent's ability to, like, correctly guess how to name a specific function. Yeah. And and that isn't really what we are, like, want to measure at this point.
Yeah. I I think that's fair. Is there I mean, if I if I asked you to ballpark it, like, most models are most frontier models are now, like, 80 something.
Is there, like what's the actual, like, number on Superbench Verify that you did you guess as, like, the ceiling?
I guess that's really hard to say. Like, I when Gypsy 5.2 came out, folks took a look and found that it was solving, like, 31 problems that were in the set of should be very hard to solve without contamination problems.
So I think it's quite possible that that number is already something that we've hit if you didn't have contamination at all. Fair enough. Hard to say, though.
Yeah. Cool. We're gonna stop reporting CBench Verified.
Right? And then CBench Pro will will be sort of the next one, which is an effort from scale. What's your sort of comparison analysis?
What's what attracts you to CBench Pro? The first one, I think, is just that it's harder. SueBench Verified, I think that's something like 90% of the problems are things that were estimated to take, like, an expert software engineer, like, less than an hour.
They're very well specified, very self contained. And the SueBench Pro problems are just bigger and harder, and there's much more headroom on the eval because it's not saturated. Yeah.
And, like, categories of, like, one to four hours and four plus. So Yeah. And it's more diverse.
Lots of re repositories, multiple languages, qualitatively more different types of problems. So all that's great. On the contamination side, we also think it's better there.
So the way we were measuring for contamination, first we went to Verified, was with this little, like, contamination auditor agent, which is given the description of the task and the patch and the task ID and told to go take this target model and kind of with an open ended, like, set of questions, try to find questions that will manage to kind of reveal what contamination might be lurking in that model. And in SuiteBench verified, we found many instances of contamination across OpenEye models, across like Quad, Opus 4.5, Gemini Flash.
And in all of these, we saw things like regurgitating the ground truth solutions, things like, in some cases, giving, like, the task IDs and other things that are pretty clear evidence of at minimum familiarity with the repositories. Yeah.
So we might have what task ID that that Yeah.
It was super strong. On the other hand, we don't see this.
but it's very different than Subint verified. So less contamination is good. I think there also, like, we should expect that at some point, like, that that's not going to be, like, the right benchmark anymore.
problems that we can match our capabilities on. Awesome. So let's go into that.
I think that there are a lot of I think we also prepped this in the in the pre chat was well, people feel a qualitative difference when they're using 5.1 to 5.2 to 5.
3, and it's not super expressed in the these benchmarks because they they are on a number of these things. What capabilities do you really want to benchmark in a in a ideal coding benchmark? You know, I guess, like, agentic coding benchmark, whatever you call it.
I mean, one thing is kind of open ended design decisions, places where the problem maybe is a little bit under underspecified, and seeing if the model can make reasonable design decisions. What's a reasonable prompt for that? Like, this five code me a b to b sass and make no mistakes?
Or you know? Like, that's that's the meme, but, like, okay. What's, like, what's, like, an actual usable open ended problem like that?
Like Sure. I mean, maybe an example could be finding a way to speed up a particular part of a code base, but there might be multiple different ways to speed There are dedicated performance benchmarks. I think you guys have one.
So efficiency or is that is that oh, no. I think that's of your horses group. But, yeah.
Yeah. I mean, that that that is a good one.
like, value about working with with software engineering agents. They think three band three bands verified, obviously measured like some they measured like some important capability which is like, given like a description of a GitHub issue, can you produce like a patch that solves that issue, you know, satisfactorily? And like, obviously, there's like some issues with the with the benchmark that means that now that we're at like 80%, we don't really trust like further improvements on it.
But like it does match on something that is like a real real capability of models. But I think as a field, we're like moving beyond sort of, you know, can my coding agent like solve a small like GitHub issue for me. Right?
And so we are starting to look at, like, much more longer term tasks. Right? Like, that don't take, like, fifteen minutes, but maybe, like, hour, sometimes days.
And then beyond sort of, like, what kind of tasks can my agent solve, like, there might be things that are kind of a bit harder to grasp. Right? Like, Olivia talked about sort of, like, does it have, like, design taste?
Right? Like, does it solve the problem the way that, you know, my team likes to solve problems? Okay.
The is the code nice? Right? Like, is it is it well written?
Is it sort of, like, clean code? Right? Like, people care about this.
Is it maintainable in the future?
things that that are still, like, super meaningful for people that are working with coding agents. Yeah. So, I mean, these are all qualities that are obviously the no longer the low hanging fruit.
Like, we have no idea how to eat all this. I think the the the simple question, maybe the the the there's sort of two forks in the road. One is the sort of very human intensive, money intensive path, which is hire a bunch of contractors and try to annotate this.
The other is use an LLM to to proxy it and try to align the LLM so that it can give you a reasonable proxy. Which of those would you want? Tai, what you wanna do both?
I think, like, maybe you should talk about GDP law as, an example.
Sure. So GDP law is an eval that was, again, produced by a collaboration between human data team and the front of evals team. And it's trying to measure whether agents can do kind of a variety of like real world white collar work.
That was an eval where grading is very hard. Requires kind of a lot of kind of, like, domain knowledge on exactly what are you looking for in each different context. Yeah.
professions, like, that take up a significant part of GDP, which is great at High level professions and then a lot of, like, different granular sub professions. I have said, like, I'm a big fan. It's it it is this is the eval for for AGI, basically.
it required so much kind of, like, domain knowledge that the human data team hired, like, a lot of people from these professions to be very involved in creating tasks and creating the gold solutions and trying to help create rubrics and so forth so they can grow it lively.
the GDP val, which is a generalist thing, take that same approach to apply it to code, and you roughly have, like, a a rough road I think that's an interesting
Yeah. Solution, I think. What you're pointing out is an important problem, which is sort of this this, like, how realistic is it?
And, like, do you know, what what we want to do is, like, coding agents should write code that, you know, we think is good. And so it's like asking human, that's actually like a a good way to ensure that. It's also kind of a slower, like complex way to do that.
And so part of why I think, you know, SuiteVenture Verified ended up being super popular and where we are seeing like ABAP benchmarks like this being super popular. It's like, it's very easy. It could even be easier, but like validating that a solution passes all the tests is like fairly trivial once you can like run the tests in in like your on your computer or wherever you're running them.
And you can kind of like, okay, is it correct or is it not correct? And you can kind of aggregate that and that it's super simple. But it doesn't tell you it's like, you know, did the model solve the problem?
Like, wow. Like, you know, ugly like, what if actually, like, an open source maintainer of that project have, like, marched that PR? Like, that it doesn't tell you.
But there is a lot of value in having benchmarks that are both, like, easy to compare across the industry and also that can be sort of run really fast without human involvement.
Yeah. Amazing. Your teams also put out other kinds of evals that are related, like the I think there's an RL paper bench, and then the the sort of, like, the more sort of recursive self improvement type evals.
How much should that figure into mainstream coding evals? You know? Like, is there is there some way in which those things join together?
So we were asking, like, should we build should we also be building Evals for the self improvement Evals? Are you saying do coding Evals currently cover that?
I I I just think, like, those are some of the most advanced Evals that we have, and we're not using them in the normal path. And it's just it's an interesting split between, well, here's Evals for coding normal things, and then here's the one for machine learning. That is, like, completely different.
Right? Yeah. I think you you get what I mean.
That's mostly, like, a safety argument, I guess.
the model is really good at, like, AI code, basically. Yeah. Oh, yeah.
Like, my guess is that part of the reason that a lot of benchmarks so far haven't focused as much on the AI coding is just a question of, like, what data sets are easy to gather. Yeah. Because a lot of the, like, you know, state of the art AI code bases are proprietary.
So if we make evals for that, like, we're probably not gonna release them. And it's harder for people in the field to make evals that kind of measure, like, is this a realistic research coding workflow? I do think that it's good for the field to try to measure these skills in a public way, I think it's just harder to make it realistic.
Got it.
maybe we we we denominate in dollars. Right? So you had freelancer and all that.
Other people are doing, like, vending bench, whatever. Any any alpha in those, or are they are they you you still want, like, a traditional academic benchmark?
I think in a way, like, there's, like, different ways to measure the same thing. Right? If we're like, oh, this is, like, how much money it produces.
Yeah. It's a fairly similar thing to saying, like, oh, this problem would take, like, a human, you know, two hours to solve or something like that. Usually, they they're, fairly, like, correlated.
Right? Like, however, you know, much it would take like a human to solve that problem kind of determines the value that we ascribe, like a solution. And so I I do think that is like an important thing.
It's, like, how complex and how sort of long running are the tasks that we are, like, able to entrust our agents with. Yeah. And so I think that that's, like, an important piece.
they all kind of, like, try to capture, like, a similar thing. Yeah. Okay.
So they're they're all proxies for some amount of increasing capacity that we wanna measure. I think that's a good thing. I think the only other sort of major player in this field is METER, which has done the sort of long grass, and congrats.
You guys have completely destroyed the curve for that. Any takes on that? Obviously, you come up really well, so, like, it looks good, but I don't know if, like, that approach is something that you wanna incorporate in your own work, making it else.
This is the long autonomy test for your Yeah. No.
we we work with with MITRE on these evaluations, so, like, we we do appreciate them. I think then they're using time. Right?
They're not using money. So I think, like, that was your question.
getting to. K. Com complexity is the abstract thing, and then it projects down the time, projects down to story points, whatever, dollars.
Great. One last question on just, like, just the overall preparedness framework is that, know, I was actually kinda looking at people mentioned the preparedness framework lot. I don't think it's well explained to a lot of people, and you actually have a nice website where it's like I think it's like test and, like, inform and teach something.
And I I feel like you you actually do a lot of work there, and and I don't know if you wanna talk about how the preparedness framework applies.
So the preparedness framework is open eyes kinda like public framework for how we track frontier risks. So these are kind of capabilities that are typically dual use. Like, you can use them for good things or bad things, but we wanna at least keep an eye out for the bad things to make sure that we both we as a company and, like, the broader society are kind of prepared to handle the potential downsides.
And so at the moment, we kind of track three different categories. One is kind of bio risk. Another is cyber security, and a third is kind of research automation and model autonomy.
And that's kind of what ties most into the SuiteBench where coding is not all of automating research, but it is one very important key component. And so we initially created SuiteVenture Verified as part of, like, building out evals for that model autonomy work stream. And now, I think for like, we have to move beyond that towards looking more at, like, can models actually start to actually automate research on workflows.
Yeah. Amazing. Okay.
how evals and human data and alignment all work together at that?
thing that I would say is that we really appreciate we we work really hard to build these events and and so we that's where we published Zubench verified and that's where we're, like, sharing GDPR, these sorts of things. We also deeply appreciate, like, other people and the entire field to kind of build evals and and share them and reuse them. Like, three bench program, like, yes, but that's a better eval now.
We should use them. So would really encourage people to find more ways to create and share events that we can we and the entire field can use to measure, like, progress Yeah.
including including encoding, because it's important to understand sort of where we are. We had to leave, but we're we're just kind of talking a little bit about, like, the the future directions that we want you files to go. Mhmm.
And I I think here here we can dive in on, like, give us work good work on these these these things, and we'll talk to you. You know? Like, here's your platform to make a call for what you're looking for.
I think a few things that would be useful, I'd say, first of all, really, really hard task.
top notch engineers months or teams weeks would be quite good, especially if grading is reliable and grading is like, you know, you have, for example, like rubrics that have been sourced and validated by many people in the field. I think that'd be quite valuable. I think also benchmarks on kind of creating products end to end.
I think if people are by putting more, that would be quite useful. I think a third thing that I'd say that is maybe not quite an eval, but I think it's still relevant to the kind of overall mission of, like, we as a field and as a world should be tracking, like, where are these capabilities going? I'd like to see more metrics tracking, like, real world usage, like how much is AI actually being used in the field, how much is it, you know, replacing people's jobs, how much is it, you know, augmenting people, speeding people up, just like real world metrics.
Yeah. Yeah.
that's how it is. Yourself, like, you know, I think in terms of the frontier evals that that OpenAI is really going to excited to push. Like, you you put out really good work every single time.
What should people expect from from OpenAI itself? I'm not sure I could say what we're gonna General directions. Soon.
I mean, general directions, I think, looking at real world impact, like real world real GDVLs who yeah. Whatever. That kind of stuff.
Yeah. Yeah. Amazing.
Okay. Well, I'm excited for more of your world impact. I I think you guys have, you know, really made a lot of progress, and I think taking a lot of industry leadership for CBench Verified and and now moving on to CBench Pro.
So thank you for doing this. Thank you for being so transparent, and I think people will respond in kind. Yeah.
Thanks for your time. Thank you.
Shared via Hopper