[State of Evals] LMArena’s $1.7B Vision — Anastasios Angelopoulos, LMArena

Yeah. Has changed the the the the the feel of it. I don't understand how you feel.

Yeah. I don't know. I mean, the reason we kept the LM at the beginning is because we were, like we started as LMSYS.

Right?

He does. He's been incredible to us. You you wanna shout out some stuff, Ethan?

Yeah. Absolutely. You know, the way the company started was as an incubation by Anj.

Yeah. So what he did is he kinda, like, found us at Berkeley and picked us out of the basement, and was like, hey, these guys seem like they're onto something, and started working with us really early. Gave, you know, gave us some grants.

He was not you know, a 16 was not the only one to do this. We also had a great grant from Sequoia, but Anj was, in particular, quite quite supportive of us, and, you know, gave us some resources in order to continue building out Arena before we even were committed to to starting a business. And in that capacity, he sort of, like, you know, formed an entity for us and this and that.

And, you know, he was like, hey, you guys can walk away at any time if you guys don't wanna start a business. I mean, was really incredible. Very aggressive investment move by him.

Right? Because, of course, any money that he spent business. Yeah.

At the end of the day, like, we could walk away and give leave him with nothing. But I think he knew wisely that the right thing for Arena was to start a company out of it. It was the only way that we could scale and that, you know, Wey Lin Yan and I would would ultimately see that and be excited about doing it ourselves, which is which which ended up being the case.

Was there a moment for you where you're I'm sure you were debating it yourself. You had other opportunities.

the only way to scale what we were building was to build a company out of it. That the world really needed something like Arena. Arena being really a place to sort of measure, understand, and and advance the frontier AI capabilities in on real world users, on real world usage based on organic feedback.

And that in order to achieve the scale and, you know, distribution necessary and the quality, of course, of the platform necessary to to do this effectively, we would need to start a company out of it. You know, we considered other options. Are we gonna keep doing this as an academic project?

Are we gonna do it as a nonprofit? Blah blah blah. But ultimately, under those constructs, we didn't feel like we'd have the resources necessary to accomplish our mission.

So you raised a 100,000,000 or 80? $100,100. 100.

$100,000,000. That's a lot of resources. It's great.

What Yeah. What's it for? Well, you know, obviously That's good on behalf of, like, everyone who's Yeah.

Like Everybody. Dude, it's an arena. How are you gonna spend your money?

Yeah. Yeah. It's great.

Yeah. Please tell. First of all, we don't necessarily need to spend all that money.

Right? The the purpose of money at a company is to give you cards to flip. It's to say, hey, you you have enough resources necessary, and so that if your first bet fails, you can make another bet and another bet.

Of course. So that's not to say that we're gonna spend all of it. Of course, you wanna spend things responsibly.

Having said that, the platform's actually quite expensive to run. We fund all of the inference on the platform. You know, the way the way that it works is the platform you pay market rates.

They don't give you discounts. No. No.

We get discounts, but they're but they are standard enterprise discounts. Okay. The same that would be given to any other customer.

again, this is off the cuff, but I can safely say we have more than 5,000,000 now. Five, six million. We have probably 250,000,000 conversations that happen over the course of the platform.

We're on the order of, you know, mid tens of millions of conversations every month that are having on the platform. It's actually quite a large, you know it's one of the largest consumer platforms for LLMs. Of course, nothing is really comparing to ChatGPT, but, you know But no.

Still. I mean, I think the the largest skilled ones I The benefit of this is that like, it's actually quite a diverse population. So 25% of the people on our platform, for example, do software for a living.

Still at this scale. How do you know? Because we like do all sorts of we we either survey them, or we'll like analyze the prompt distribution that's coming to platform.

I'm happy also to share more. We've done something called Expert Arena, which is like trying to understand the distribution of experts that are coming to the platform. A lot of that can be like unauthenticated or whatever usage.

It is. But about half of our users now are logged in. Yeah.

And so we have some ability to understand them, and and we also have surveys that we run on the platform that tell us a bit more about who are the actual users. Yeah. Of course, there's always, like, response bias in surveys, so you have to take it with a grain of salt, but nonetheless.

And if you don't know Anastasia's background, like, he's he's the guy to correct for response bias. Yeah. No.

Okay. And there's a lot of there's a lot of guys like that and girls. So Like, guys and girls.

You're not the only player. There there are this artificial analysis started in Arena. Yep.

AI. Yep. It's like some crypto people that started this.

Yep. Yep. I don't know if you have a conversation with them on like, well, hey, this is our thing or, like, let's work together on something.

I I I don't know if you've what the industry No. You know, so I've talked to artificial I've actually talked to both groups. Yeah.

Both seem, you know Am I missing any major players? It's just those No. I think those I don't know.

Yeah. I don't think so. Okay.

I think those are like some, of course, large depending on how you define the term. Artificial analysis obviously has like huge, like, market mindshare Yeah. Around the analysis of of different AI systems.

Yeah. They told me they were just like, we we are gonna be gardener of AI. Yeah.

That's that's kind of their goal. And I think they're going after that consulting market and so on. Artificial analysis, from what I understand, is like a team of consultants that are doing this.

Yeah. Seem like really, really nice guys, going after that that particular market. It's different from our platform in the sense that their analysis is based on sort of like aggregating public benchmarks and turning those into analytics.

Independently rerunning. And independently rerunning, which yeah. Yeah.

Yeah. Which matters. Yes.

And and and and using those in order to compile sort of reports and so on that that educate the field on the performance of all these different models. But they also have arenas. They have arenas, but the arenas are not based on organic usage.

Like, the the thing that distinguishes our platform versus theirs is that the users are actually inputting their own use case. They're actually asking their own question. And that gives a a level of realism Yeah.

That platform doesn't have. Of course, they they specialize in a slightly different thing, but I see those platforms kind of diverging in that sense. Yeah.

Yeah. And sometimes it's like, it's the only way to do this.

Do you actually care about like other people's videos? To form your own intuition. Maybe.

Yeah. Maybe you're like interested in comparing Like, I'm a shitty prompter. Right?

Are you? So I don't believe that. I'm a terrible prompter.

No. No. No.

No. I learned by example. Don't denigrate yourself.

Don't denigrate yourself.

Oh, yeah. Yes. We well, listen.

Gradyo, incredible platform. Gradyo scaled us to a million mal. Yeah.

That's incredible. And, of course Can you tell the Hugging Face ones that? Of course.

Yeah. We're really, really grateful to Gradyo for taking us so far. Eventually, you know, it became time for us to move off of that and go to a React sort I'm sure Hugging Face would have loved you to stay on.

They would have. I'm sure. Yeah.

Was there a technical, like, reason you you you just couldn't get the performance? Yeah. It just became hard to develop, and there were all these tools that we wanted in React.

And, you know, to do all the fancy things that you can do in React became kind of like One example of that. I don't know. What what's a feature that you really wanted?

Let's say we wanted to create, like, our own custom, like, loading icons for video with notifications. Okay. How are we gonna do that in in React?

It's hard. Yeah. I mean, we'll make a Sure.

There's a custom component. I'm sure the high end guys are gonna come in and say like, hey, you can do that in Gradio, which maybe you can. Of course they could.

Also fewer developers know. How are we gonna hire for that? Do we have to reskill them?

People are less familiar with that stack. Yeah. You know?

So anyway It's full React Nx. Js, all this. Yeah.

Yeah. Yeah. All that.

Okay. Cool. Other use of funds that that that might be interesting?

Like, you know, basically No. That's basically deployed resources. Okay.

Yeah. It's it's it's on primarily inference. That funds the free usage of the platform, and and then also hiring, of course, headcount.

Yeah. We have an office. You know?

That's us.

So Leaderboard Illusion's a paper that critiques Alan Marina. And the main Pretty, like Yeah. Brutally.

Well, you know, I would say unscientifically. Yeah.

was, like, 74. It's all good. You know, it's actually not it's a it's a respectable place that they had on the leaderboard.

I don't even think it was really Cohere people like the Cohere model developers doing this. It was more their research side. But in any case, what is the what is the leaderboard illusion say?

It says that Alan Marina was what their the claim is that Alan Marina was doing this undisclosed, quote unquote, private testing on our platform, that model providers will send us pre release models, and we'll expose them, and so on and so forth. And that this creates so called inequities in the leaderboard, you know, due to that prerelease testing. For example, they will they cited that Meta, at some point, tested Lava Board.

Amount of models with us. Yeah. Of course, we can't disclose all of the details of how all that was done, but that is that is the main claim of the paper.

Now our response to that paper, it's online. You can find it on response to Leaderboard Illusion. And our response to that paper is essentially pointing out a series of factual mistakes in the paper that that question the validity of the claims.

So you can go look at the first version of the paper on archive yourself, and you'll see the claims. I think most scientists would view that as Oh, they've corrected it. They've of course.

Oh. Because we I mean, but they didn't correct everything. They just corrected some aspects that were just blatantly unscientific and false.

But, you know, they're for example, said that we were that we only sampled, like, 9% open source models and, like, you know, 60%, like, closed source models, and this created a gap between open and closed source. But in reality, we're actually really supportive of open source models, and, it was more like sixty forty. And, so that was, you know, one of the examples of of a of an air an error in the claims.

Another example is that they were claiming that there was some sort of bias introduced by this pre release testing, and that it was undisclosed. And reality is, you probably know, we've been doing this pre release testing for a long time. Our community loves it.

They love basically getting like The secret code names. Yeah. The secret code is NanoBanana.

Yeah. All that. So NanoBanana, by the way.

It started on you. It started on us. Yeah.

Yeah. Right? And people loved it.

It went like global sensation. Like, non non zero fraction of the global population using nano. Talk to Nina about naming it banana?

Or was that No. Decision? It was it was their decision, decision, I I believe.

Believe, But but it it was was sort of this randomly generated thing. And it just went No. No.

So apparently, Naina, who's a PM Yeah. Yeah. Is named after her because her nickname is Nana.

Oh. Oh. That's sweet.

I didn't know that. It was Nana putting banana on it. Yeah.

Oh, that's sweet. And I Yeah. Didn't know that.

Didn't know the origin story. Yeah. I mean, to us, it just looked like sort of a random thing.

But it went clearly, heads and shoulders above. Huge. Which, like, before that, there was Reeve Image.

Remember? Yeah. I do.

Of course. Yeah. And also of BFL and all those mean, all those models are also great.

Yeah. And I think those teams are also improving quite quickly. Yeah.

But Nano Banana was a sensation. I mean, moment alone changed Google's, like Roadmap. Yeah.

Market share. Yeah. Seriously.

I mean, Google stock. Billions of dollars Yeah. Are moving because of Nano.

And now there's like an OpenAI code red and everything. I I don't I don't know about that, but yes. The information reported this.

I'm such a huge Nana and Banana Pro show. Yeah. I totally agree.

I was also kinda wrong about this. I didn't see the positive benefits. But actually, I think that these, like, multimodal models are gonna become some of the most economically valuable aspects of AI.

Yeah.

Yeah. Soon, we're not gonna be even making the fraud Yeah.

and center the use cases of real users. Foreground those so that people know what to target. The goal is to create a benchmark that is constantly fresh, that does not suffer overfitting, because of the fact that we constantly have new data points coming in.

That tracks the, you know, all the different new models, all the different new use cases of AI, and gives the whole world sort of ground truth for how real users are using these models, and how how good they are on those use cases. We continue to do quite a few open source day releases. We've probably released more data than basically anybody on the real world use cases of AI.

Millions and millions of conversations, real world conversations from real users that the community is using to study and and improve on. Yeah.

So let me first talk about things that I'll never do. The platform integrity comes first to the platform. The basically, the public leaderboard that we show on Ellen Marina, I think of as a charity.

It's a loss leader for us. We don't really make money on the public leaderboard. You can't pay to get on the public leaderboard.

It's not like a Gartner in that sense. Mhmm. It's not like any of these, like, you know, pay play systems.

Never going to be like that. Models are gonna be listed on the leaderboard whether or not the providers pay, and whether or not they're getting a good score. They can't pay to take it off either.

So what that means important. That that's very important. And so what that means is that the leaderboard has a certain integrity that will never be compromised.

Of course. But but not all preview models will make it onto them. No.

But that's okay. Those preview models have never been released. Yeah.

Yeah. Right? Who who cares about putting unreleased models on a on the leaderboard?

The point is that for every released model, the score that you see on the leaderboard is statistically sound. It reflects the real world capabilities of the model. Yeah.

Why? Because millions of people from around the world have voted for it, and that's where that where that number comes from. All we do to con compute that number is millions of people are voting.

We take those votes. We turn them into a number. That's always going to remain, you know, a transparent and fair reflection of model performance.

Where are we going? Lots of different new categories. I don't know if you recently saw we expressed we we exposed occupational and expert categories.

So now, single digit percentage of our user base, we're millions millions to tens of millions of users. Right? So single digit percentages means a lot.

Single digit percentage of our user base are in medicine, in legal, in business, you know, finance, accounting, creative marketing, stuff like this. And we're able to show the performance of these models in all these different verticals, because we have all these users in our in our user base. And we're we're working more towards multimodal.

We've thought about it. Yeah. I think it's a it's a Yeah.

What what what are the counterarguments? Like, why why not? Well, there's obviously a need for an API.

The question is more of focus of our company. Just because we're a startup, and so we really should be doing one thing well. Arenas.

Yeah. Arenas. So I'm not I'm not sure that we how far we wanna sort of splay out and on what timeline we'd wanna do that.

Yeah.

Well, so first of I wanna give a shout out to our community manager, Greg, who is doing an awesome job managing our community, whether that's on Discord or on Ella Marina. He's really incredible. So I would say hire Greg.

But don't hire Greg. Hire Greg. Don't hire Greg.

Find a Greg. He's ours. Find a Greg.

Find a Greg. But in general, you know, the question of how do you get to so many users? That is a tough question.

And keep. Retain them. That is a tough question.

Because consumer is one of the hardest markets in the world. There's a lot of websites in the world that people can go to. You know, why should they go to yours?

And the reality is if you want to create a really dominant product, you have to provide people value. And to be frank, I don't think we're all the way there yet. It's not like I have the solution and answer for how to build a a great consumer product.

If I did, we wouldn't be at 10 tens of millions of users. We'd be at hundreds or, you know, we'd be at a billion users. We'd be like Is there a world you like are bigger than Chachi PT?

I don't know. I don't I don't know that we need to be. Yeah.

And I don't know that we ever will be, because that's a that's an extraordinary generational product that they built. Right? It took a lot of time, and then to some extent, it also involved luck.

There's a lot of lightning in the bottle moments, like Nano Binnel was for us, where our user base just like goes up by a lot. But when those users come, they can just as easily leave. So the way I think about it is, every user is earned.

You have to earn them every single day. They can leave at any moment. They're fickle.

And so all the time, you have to be thinking about how do I provide this person value? Learning. How are they using my website?

What more could I give them? And how do I build in all the retention mechanisms so that they stay, and then they're also bringing their friends?

Yeah. Cool. Do you want from people?

What what are you looking for help on? Like, any call to action? Yeah.

I we are always looking for people to come and join us. If you are one of the best people in the world in your area, whether that's consumer product, whether that is machine learning, whether that is, you know, b to b, go to market, marketing, all these things, we need you at Arena. We're building like a high performance team of real experts in everything that they do.

Yeah. So, I mean, we, of course, partner with all of the major model labs. Yeah.

And that's that's just straightforward. Like, hey, we have a new model. Here here you Yep.

Exactly. So I think the the most straightforward thing would be for someone like Cognition. It's like, let's evaluate that then.

Agent. Yeah. But we should be continuing to shape our well, Code Arena is an agent evaluation.

True. That's true.

So But that maybe should change. Maybe we should be evolving towards that direction. And I think the Code Arena is a good example of an arena that would support a full featured harness like a Devon.

Yeah. And so in my view, I'm if if I'm talking to a cognition, I'm saying, hey. Let's get Devon on the arena and figure out how to, you know, loop together the Devon harness so that we can I'm sure I'm sure that there's something that could be really valuable there, especially given Devin.

Last week, people were talking about Devin dead. Did you see that? Yeah.

People were saying Devin's gone. Devin's not gone. It's not gone.

Devin's everywhere. Say it very well. But people so can we highlight that for people and show them, hey, Devin is actually the best or one of the best in the world at doing what it does?

Al Marina can actually do that. And Yeah. Our our our place as a central evaluation platform allows allows that to happen.

Yep. Love it. Alright.

Thank you for owning the State of View Vals. Thank you so much. Congrats on a wonderful year.

Appreciate it. Congrats to you too. Thank you.

Congrats on all the growing, you know Yeah. Momentum in your podcast and in your career. Thank you.

It's really impressive to see.