[State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang

the project is, like, one and a half years old? Yeah. Yeah.

So I think one and a half years old in terms of when it was actually useful. Yeah. They were put it out October 2223, and then people didn't really touch it too much.

And then, of course, like, Cognition came on the scene, and Devin was an amazing release. And I think after that, it kinda kicked off the arms right you beforehand? They just showed up.

It was you know, I got an email about, like, two weeks ago. I think it was from I think it was from Walden. He was like, hey.

You know, we have a good number on it. I was like, wow. Congrats.

You know, thanks for using it. And then the release was like mind blowing. I was like, wow.

These guys did an excellent job. Yeah. Amazing.

And then SuiteBench verified was, like, maybe last year. That's right. Yeah.

Like Catch us up this year. Like, you have other languages.

Sweet Bench Pros, Sweet Bench Live.

I think I think we're we're we're we're okay with it. When we came out, we were like, oh, cool. Interesting.

It would've been, you know, fun to be part of it. But, you know, I mean, congrats to them. That's a great benchmark.

Yeah. Alright. Yeah.

But yeah. Multimodal? Yeah.

We did multimodal and multilingual, and I think like those have Multilingual seems to Is be it like JavaScript? What else? Yeah.

Multilingual is like it's like nine languages across like 40 repos. But, you got them.

a lot of the newer benchmarks, like, really try to diversify the repos. Like, in the two follow ups we did with multimodal and multilingual, we made it a point to do that. So I think But you can also just put out SuiteBench '20 25 and just That is true.

And do a new distribution. Yeah. Yeah.

So it's been cool to see the follow ups. I think quietly, and and it's an open question for me. I'm excited to see how people curate the next sets.

Like, it's kinda interesting to see in the literature or in their blog posts, like, how they're justifying why they're creating their separate split. The easier ones were like, oh, more languages, more repos. And then I think now people are like, well, ours is more difficult because of this curation technique.

So, basically, the idea is I don't like unit tests as a form of verification, and I also think there's an issue with Suitebench where all of the task instances are independent of each other. So the moment you have the model kinda submit it, oh, it's done, you know, and and that's the end of the story, end of the episode. You know?

So with CodeClash, what we're thinking is, let's try to really evaluate, like, long horizon development and development on a code base that is consequential and conditioned upon what a model did, you know, before to that code base. And so the general idea is you have two or more language models, and they play a programming tournament. And what that means is each model maintains their own code base.

And each round of the tournament, first, they get to, like, edit and improve their code base however they see fit, very self determined. And then in the competition phase, those two code bases are are pitted against each other. So the code bases are run, and there's generally an arena.

You know, we have a lot of diverse arenas, but the arena is determined, like, code base a is better than code base b. And then you kinda repeat that across multiple. As determined by an element judge.

Yeah. Yeah. So element judge is definitely one of the mechanisms.

We started with some pretty, like, simple programming games. So one of the cooler ones is, like, Highlight, which Michael Oh, yeah. I played it for Jane Street.

Yes. That's right. Yeah.

That's right. You know, that's awesome. Yeah.

Highlight one, two, three, like, Michael Troll of Cursor wrote this game. Two Sigma on Jane Street? Yes.

Yes. Oh oh, Two Sigma. Two sig it's Two Sigma.

I worked at Two Sigma.

The current ongoing effort is, you know, to build economically valuable arenas. That's, you know, the popular word these days. So Yeah.

Sweet Lancer is a big one this year. Yeah. G d GDP BAL awesome.

Yeah. Just I mean, I think the big selling point of Terminal Bench and Sweet Bench and these EVALs is that it was really close to real world utility. And so I think it's resolvable for CodeClash, and that's what we're working on.

Yeah. Okay. Yeah.

So you're part of Ophir's group? Yes.

Suificiency, I really like in the line of Yeah. Performance What's the TLDR on that one? Yeah.

For sure. So Suificiency was wrote by this PhD student called Jeffrey Ma, who happened to be my high school classmate. And the idea there was, like, you take a code base and you just want to, you know, do modifications that will literally make the code run faster.

So I think it's just, like, parallelization, SIMD operations, stuff like that. Yeah. So so no no behavior change, just faster?

Exactly. Okay. Keep the unit test passing, but I want better runtime.

Okay. Yeah. Yeah.

And then and then there there's algo tune that is kind of in line with that. And then there's also kinda pushing along, like, the scientific coding domain. Cycode?

Yeah. Exactly. Which is like zero code.

Cycode two is awesome.

Yeah. Just just just complete. Exactly.

Like, you know, you can do well on those first and then sort of graduate to the multi turn expensive stuff. Yeah. Yeah.

number. Yeah. They like the x axis being sort of the runtime and their or, yeah, y axis being the completion.

You know? Like, we can do more long running staging tasks. Yeah.

I think the projections are are quite interesting, and I definitely appreciate them kinda using sweep bench verified to to sort of proxy a lot of these things. But, yeah, they're great. Yeah.

Okay. Yeah. Any other work that, like, caught your eye that you're gonna Yeah.

I mean, I I think within the okay. Terminal bench sweep bench. Yeah.

Critical point was kinda cool. Critical Point? Yeah.

The it's like a very new benchmark that Ophir did, and I think it's kinda related to physics. There's this one called SecBench kinda related to cybersecurity. Exactly.

SREBench, which I I think is affiliated with LOD. Like, it's just cool to kind of see people really dive into different coding domains. And then stepping a little bit outside of coding, I'm personally, I think it's quite interesting to think about the user simulator stuff.

So, like, TauBench bench? TauBench too. Yeah.

And and VendingBench.

To me, I think it's super cool to see companies like you know, I'm sure Merkor and stuff are focusing on building environments, like, for code, beyond code. And so I think it it might be interesting to have, like, work gym style stuff. This is stuff that my adviser, Dee Young, at Stanford thinks about a lot.

So yeah. Yeah. I just realized we we're talking about terminal bendy.

Yes. We're fighting a lot of other folks. Yeah.

Yeah. You know, really really really good work just overall.

Yeah. I don't know if you're up to speed on that. I'm a little It's a little spicy.

Yeah. It's a bit spicy. I think I saw so I, you know, for like, I worked with Shinyun and Karthik back in Princeton very closely.

Think Karthik, I just saw, posted a tweet kind of Defending Yeah. Like, rebutting some of these claims. Yeah.

you've been cheating. Yeah. Oh, interesting.

It's that would be that would be cool. Yeah. I mean, yeah, you'll have to ask the TauBench authors, but yeah.

No. That that's that's fun. Yeah.

I I think there was Impossible Bench was a recent benchmark maybe from, was it from Anthropic? I don't know. But they basically took SweetBench verified, and they changed the issues to make them impossible, and they checked, like, how often the models would be like, I actually just can't do this.

I don't know what's going on. Oh, like for refusals? Yes, yes, yes.

So Oh, how do they do? I thought that was interesting. I think they're all the models are all kind of attempting and saying like, oh, I did it, you know?

So maybe not great on this That's cool.

How does coding eval evolve next year? Wow. That's a great question.

I mean, honestly, I think I think it's people will make more sweep benches. I think terminal bench has really got something going where you you ask people to you know, a sweep bench, you're you're confined in some sense to the domain of issues and PRs that already exist, which I think has its benefits of being close to reality and natural. But I think with Terminal Bench, there's a lot of creativity that you can infuse into that.

So I would personally be really excited. Like, the two point o job was really excellent, and I'd be super excited to see, you know, three point o, four point o. Because of, like, the environments?

Yeah. I mean, the environments, you know, bringing more people into the fold. You know, I think correct me if I'm wrong, Mike, but early on, you had PhD students, very smart CS people who were adding tasks.

And, you know, what does that look like when you fold more coding environments for noncoding tasks, noncoding environments in general, and ask people to make stuff there? So that's pretty cool. And then, of course, for myself, I think just, like, this long running kind of thing just feels very compelling.

I think the vision of, like, hey. I tell it a goal. I don't have to be super specific about my task.

I have, like, a decent verifier that proxies what I want. Something literally like a code base that makes the most money in this, like, setting. You know?

Like, that's my verifier. Verifier. You know?

And I walk away for five hours. The thing is just running. I'm hanging out with you, talking to my friends.

I come back, and it gives me, like, literally a soda code base on on on that, you know, task. I think that would be super cool. Okay.

I'll push back. We're part time in Cognition. Yes.

to me, I think there's there's potentially room for growth, so I I would actually agree with your take here. I mean, with my lab at Stanford with Dee, like, there's a you know, her emphasis is on human AI collaboration. And so I I definitely don't believe in this idea of just kinda getting rid of the human.

But, yeah, maybe just, like, finding the balance of, like you know, just because the developer ecosystem is so diverse and there's so many participants in there who want different things out of it, like, just enabling different levels of abstraction and, you know, depends on the task. Like, there's settings where you wanna be, you know, more involved and more sort of hands on, and so you wanna use Windsurf for that. But then maybe there's kind of this general data processing thing.

It's just a lot of JSON parsing you don't really care about, and that's the one I kinda wanna walk away from and just let it figure it out. Yeah. So, yeah, I I would agree with you, generally.

Yeah. Amazing. Any call to action?

What what do you want help on?

I guess, like, find more of your work? Definitely. For the call to action, super jealous of all the great data that Cognition and, you know, Cursor would get.

Like, that user entered action data is, like, really fascinating. From an academic standpoint, it feels like there's two difficult approaches to resolving that. Either you build, like, a really compelling product like LM Arena that people have people use consistently, which is, I mean, really tricky in and of itself, or you build, like, really good user simulators that try to mimic sort of these settings, but that is also, like, nontrivial.

I don't think it's as simple as, hey. Check GPT. Act like a human.

Right? Yeah. So it would be really cool to sort of get inspiration of, like, what exactly does that data look like?

Or or between the two, like, what's the best way to scale up sort of evaluating human AI interaction? And then I think for visibility for my own, we're pushing more arenas. Like, I think for for CodeClash, what I'm excited about is the current framing is really long running SWE agents, but, you know, you could have multi agents, like two agents work together on the code base.

And what happens? You have a human and an agent work on the code base versus just AIs. What happens there?

You know? Like, when the models improve, and hopefully, hill climb and they become better at digesting logs and iterating on analysis, you know, how does how does human AI interaction, like, change with model capability?

have some manually curated answers, and then, you know, post trivia questions. That's very easy to saturate, so I don't know. I'll ask you.

I think I think Silas tweeted a while ago, like, sort of, like, the the wiki, the code wiki, etcetera.