Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith

This is kind of a full circle moment for us in a way. Because Yeah.

important. Start off Get get go straight into it. Let's start Start off with the spicy take.

Okay. How do I pay you?

Well, very happy to talk about that. So it's been a, like, big journey the last couple of years. Artificial analysis is gonna be two years old in January 2026, which is pretty soon now.

We first run, like, the website for free, obviously, and give away a ton of data to help developers and companies navigate AI and make decisions about models, providers, technologies across the AI stack for building stuff. We're very committed to doing that and tend to keep doing that. We have, along the way, built a business that is working out pretty sustainably.

We've got just over 20 people now. And two main customer groups. So we wanna be who enterprise look to for data and insights on AI.

So we want to help them with their decisions about models and technologies for building stuff. And then on the other side, we do private benchmarking for companies throughout the AI stack who build AI stuff. So no one pays to be on the website.

We've been very clear about that from the very start because there's no use doing what we do unless it's independent AI benchmarking. Yeah. But turns out a bunch of our stuff can be pretty useful to companies building AI stuff.

story kind of begins, like, in 2022, 2023. Like, both George and I have been into AI stuff for quite a while. In 2023, specifically, I was trying to build a legal AI research assistant.

So it actually worked pretty well for for for its era, I would say, But was finding that the more you go into building something using LLMs, the more each bit of what you're doing ends up being a benchmarking problem. So had, like, this multistage algorithm thing, trying to figure out what the minimum viable model for each bit was, trying to optimize every bit of it. As you build that out, right, like you're trying to think about accuracy, bunch of other metrics, and performance and cost, and mostly just no one was doing anything to independently evaluate all the models and certainly not to look at the trade offs for speed and cost.

So we basically set out just to build a thing that developers could look at to see the trade offs between all of those things measured independently across all the models and providers. Honestly, it was probably meant to be a side project when we first started doing it. Like, we didn't, like, get together and say, like, hey.

Like, we're gonna stop working on all the stuff, like, this is be our main thing. When first called you, I think you you hadn't decided on starting company yet. That's actually true.

I don't even think it would pause, like, like, George, didn't quit his job. I hadn't quit working my legal AI thing. Like, it it was genuinely a side project.

Yeah.

eval framework harness. Yep. Yep.

That was some cool stuff. At the end of the day, right, running these evals, it's like if it's a simple q and a eval, all you're doing is asking a list of questions and checking if the answers are right, which shouldn't be that crazy. But it turns out there are an enormous number of things that you've got to control for.

And I mean, back when we started the website, like one of the reasons why we realized that we had to run the evals ourselves and couldn't just take results from the labs was just that they would all prompt the models differently. And when you're competing over a few points, then you can You can put the answer into the model. It's a problem.

Yeah. That in the extreme. And like, you get crazy cases like back when Googled a Gemini one point zero Ultra and needed a number that would say it was better than GPT-four and like constructed I think never published like chain of thought examples, 32 of them in every topic in MMLU to run it to get the score.

Like, are so many things that you They never shipped Ultra, right?

I mean, I'm sure it existed. But yeah. So we were pretty sure that we needed to run them ourselves and just run them in the same way across all the models.

Yeah. And we were we were also dead certain from the start that you couldn't look at those in isolation. You needed to look at them alongside the cost and performance stuff.

Yeah. Okay. A couple technical questions.

I mean, so obviously, I also thought about this and I didn't do it because cost. Did you did you not worry about costs? Were you funded already?

Clearly not, but, know No. We we well, we definitely weren't at the start. So, like, I mean, we're paying for it personally at the start.

There's a lot of money here. Well, the numbers weren't nearly as bad a couple of years ago. So we, like, certainly, like, incurred some costs, but we were probably in the order of, like, hundreds of dollars of spend across all the benchmarking that we're doing.

Okay. So nothing. Yeah.

It was, like, kind of fine. Yeah. Yeah.

Days, that's gone up an enormous amount for a bunch of reasons that we can talk about. But, yeah, it it wasn't that bad because you can also remember that, like, the number of models that we were dealing with was hardly any, and the complexity of the stuff that we wanted to do to evaluate them was a lot less. Like, we were just asking some q and a type questions.

trying to see whether or not it can solve a particular type of reasoning problem, and you don't want to test it on its ability to do answer formatting at the same time, then you might want to use an LMS answer extractor approach to make sure that you get the answer out no matter how it answered.

so dark magic. You've also got like the different degrees of variance and different benchmarks. Right?

So if you run four question multi choice on a modern reasoning model at the temperature suggested by the labs for their own models, the variance that you can see on a four question multi choice eval is pretty enormous if you only do a single run of it and it has a small number of questions especially. So, like, one of the things that we do is run an enormous number of all of our evals when we're developing new ones and doing upgrades to our intelligence index to bring in new things so that we can dial in the right number of repeats so that we can get to the 95% confident confidence intervals that we're comfortable with so that when we pull that together, we can be confident in intelligence index to at least as tight as, like, a plus or minus one at a 95% confidence. Yep.

And, again, that just adds a straight multiple to the cost. And then Oh, yes. Yeah.

Yeah. Right? So there's one of many reasons that cost has gone up a lot more than linearly over the last last couple of years.

laser focus, like, on everything we do on having the best independent metrics and making sure that no one can manipulate them in any way. There are quite a lot of processes we've developed over the last couple of years to make that true for, like, the one you bring up, like, right here, of the fact that if we're working with a lab, if they're giving us a private endpoint to evaluate a model, that it is totally possible that what's sitting behind that black box is not the same as they serve on a public endpoint. We're very aware of that.

We have what we call a mystery shopper policy, and so and we're totally transparent with all the labs we work with about this, that we will register accounts not on our own domain and run both intelligence evals and performance benchmarks Yeah. That's the job. Without them being unidentified, and no one's ever had a problem with that.

k. The the the the biggest one, like, that I'll bring up, like, is more of a conceptual one actually than, like, direct shenanigans. It's that the things that get measured become things that get targeted by the platform they're trying to build.

Right? Exactly. So that doesn't mean anything that we should really call shenanigans.

Like, I'm not talking about training on test set. But if you know that you're gonna be great at another particular thing, if you're a researcher, there are a whole bunch of things that you can do to try to get better at that thing that preferably are going to be helpful for a wide range of how actual users wanna use the thing that you're building, but will not necessarily do that. So for instance, the models are exceptional now at answering competition maths problems.

There is some relevance of that type of reasoning, that type of work, to, like, how we might use modern coding agents and stuff, but it's clearly not one for one. So the thing that we have to be aware of is that once an eval becomes the thing that everyone's looking at, the scores can get better on it without there being a reflection of overall generalized intelligence of these models getting better. That has been true for the last couple of years.

It'll be true for the next couple of years.

It was really great to get Ned and Daniel on board. Obviously, they've done a whole lot of great work in the space with a lot of leading companies, and we're extremely aligned with the mission of what we were trying to do.

That's an interesting question.

people in AI grants who are obviously at the frontier. Yeah. To to some to some extent.

But then so a lot of what the AI grant companies are doing is taking capabilities coming out of the labs and trying to push the limits of what they can do across the entire stack for building great applications, which actually makes some of them pretty archetypical power users of artificial analysis. Some of people with the strongest opinions about what we're doing well and what we're not doing well and what they are wanna see next from us. Because when you're building any kind of AI application now, chances are you're using a whole bunch of different models.

You're maybe switching reasonably frequently for different models and different parts of your application to optimize what you're able to to do with them at an accuracy level and to get better speed and cost characteristics. So for many of them no. They're, like, not commercial customers of ours.

Like, we don't charge for all that data on the website, but they are absolutely some of our power users. So let's talk about just the the the evals as well. Right?

Like, you see start out from the the general, like, MMLU and and GPQA stuff.

first, just, like, background, like, we're talking about the artificial analysis intelligence index, which is our synthesis metric that we pull together currently from 10 different eval datasets to give what we're pretty confident is the best single number to look at for how smart the models are. Obviously, doesn't tell the whole story. That's why we publish the whole website of all the charts to dive into every part of it and look at the trade offs, but best single number.

So right now, it's gotten a bunch of q and a type data sets that have been very important to the industry, like a couple that you just mentioned. It's also got a couple of agentic datasets. It's got our own long context reasoning dataset and some other use case focused stuff.

As time goes on, the things that we're most interested in that are gonna be important to the capabilities that are becoming more important for AI, what developers are caring about, are gonna be first around agentic capabilities. So surprise, surprise, we're all loving our coding agents and how the model is gonna perform like that, and then do similar things for different types of work are really important to us. The linking to use cases, to economically valuable use cases are extremely important to us.

And then we've got some of these things that the model still struggle with, like working really well over long contexts that are not gonna go away as specific capabilities and use cases that we need to keep evaluating.

Like, how like, how we've changed the index to where we are. Yeah. I think that reflects on the well, the change in the industry.

Right? Yep. So that's a nice way to tell that story.

Well, v one would be completely saturated right now by almost every model coming out because doing things like writing the Python functions in human eval is now pretty trivial. It's easy to forget actually, I think, how much progress has been made in the last two years. Like, we we obviously play the game constantly of, like, the today's version versus last week's version and the week before, and all of the small changes in the horse race between the current frontier and the who has the best, like, smaller than 10 b model, like, right now this week.

Right? And that's very important to a lot of developers and people in especially in this particular city of San Francisco. But when you zoom out a couple of years ago, literally, most of what we were doing to evaluate the models then would all be 100% solved by even pretty small models today.

And that's been one of the key things, by way, that's driven down the cost of intelligence at every tier of intelligence. We can talk about more in a bit. So v one, v two, v three, we made things harder.

Or we could just go right into showing people the benchmark and, like, looking around and ask asking questions about it. Yeah. Let's do it.

Okay. Yeah. This would be a pretty good way to chat about a few of the new things we've launched recently.

Yeah. And I think a little bit about the direction that we wanna take it, and we wanna push benchmarking.

almost every talk that George or I give at conferences and stuff, like, we always put this one up first to just talk about situating where we are in this moment in history. This, I think, is the the visual version of what I was saying before about the zooming out and remembering how much progress there's been. If we go back to just over a year ago, before o one, before Claude Sonnet 3.

5, we didn't have reasoning models or coding agents as a thing, and the game was very, very different. If we go back even a little bit before then, we're in the era where when you look at this chart, like, OpenAI was an untouchable for well over a year. And, I mean, you would remember that time period well of, like, there being very open questions about whether or not AI was going to be competitive.

Mhmm. Like, full stop. Whether or not OpenAI would just run away with it, whether we would have a few frontier labs and no one else would really be able to do anything other than consume their APIs.

I am quite happy overall that the world that we have ended up in is one where Multimodal. Yeah. Absolutely.

last year, actually. Yeah. Yeah.

Agreed. Yeah. Well, couple of weeks.

It was it was Boxing Day in New Zealand when when DeepSeek v three came out. And I like, we'd been tracking DeepSeek and a bunch of the other global players that were less known over, like, the 2024 and had run evals on the earlier ones and stuff. I I very distinctly remember Boxing Day in New Zealand.

I because I was with family Christmas and stuff, running evals and getting back result by result on DeepSeg v three. So this was, like, the the first of their v three architecture, the six seven one b MOE. And we were very, very impressed.

Like, that was the moment where we were sure that DeepSeek was no longer just one of many players that had jumped up to be a thing. The world really noticed when they followed that up with the RL working on top of e three and r one succeeding, like, a few weeks later. But the groundwork that absolutely was laid with, like, just extremely strong base model, completely open weights, that we had as the best open weights model on Boxing Day last year.

Yep. Boxing Day is the the day after Christmas for for those not to know. True.

Maybe so we can hit past all the, like we've, like we have lots and lots of videos and stuff. The interesting ones to talk about today that'd be great to bring up are, like, a few of our recent things, I think, that probably not many people will be familiar with yet. So first one of those is our omniscience index.

So this one is a little bit different to most of the intelligence evals that we run. We built it specifically to look at the embedded knowledge in the models and to test hallucination by looking at when the model doesn't know the answer, so I'm not able to get it correct, what's its probability of saying I don't know or giving an incorrect answer. So the metric that we use for omniscience goes from negative a 100 to positive a 100 because we're simply taking off a point if you give an incorrect answer to the question.

We're pretty convinced that this is an example of where it makes most sense to do that because it's strictly more helpful to say I don't know instead of giving a wrong answer to factual knowledge question. And one of our goals is to shift the incentive that evals create for models and the labs creating them to get higher scores. And almost every eval across all of AI up until this point, it's been graded by simple percentage correct as the main metric, the main thing that gets hyped, and so you should take a shot at everything.

There's no incentive to say, I don't know. So we did that for this one here. I think there's a general field of calibration as well, like the confidence in your answer versus the Yeah.

Brightness of the answer. Yeah. We completely agree.

Yep. Yeah.

slash lack of confidence calibration thing. And so, hey, this is one of them. And I mean, like anything that we do, it's not a perfect metric or the whole story of everything that you think about as hallucination.

Mhmm. But, yeah, it's pretty useful and has some interesting results. Like, one of the things that we saw in the hallucination rate is that anthropics Claude models at the the very left hand side here with the lowest hallucination rates out of the models that we've evaluated Omnissus on, that is an interesting fact.

I think it probably correlates with a lot of the previously not really measured vibe stuff that people like about some of the Claude models. Is the dataset public? Or what's is it is there a held out There's a held out set for this one.

So it we we have published a public test set, but we we've only published 10% of it. The reason is that for this one here specifically, it would be very, very easy to, like, have data contamination because it is just factual knowledge questions. We will update it over time to also prevent that, but we've, yeah, kept most of it held out so that we can keep it reliable for a long time.

It leads us to a bunch of really cool things, including breakdown quite granularly by topic. And so we've got some of that disclosed on the website publicly right now, and there's lots more coming in terms of our ability to break out very specific topics. Yeah.

I would be interested. Let's let's dwell a little bit on this hallucination one. I noticed that Haiku hallucinate hallucinate is less than sonnet, hallucinate is less than opus.

driven this. Yeah. You can partially blame us on how we define intelligence, having until now not defined hallucination as negative in the way that we think about intelligence.

I mean, I think I think for hallucinations specifically, there are a bunch of different things that you might care about reasonably and that you'd measure quite differently. Like, we've called this AA amnesian hallucination rate, not trying to declare it like it's Humanity's last hallucination. You could you could have some interesting naming conventions and all this stuff.

The biggest picture to that, something that I actually wanted to mention just as Josh was explaining, critical point as well, is so as we go forward, we are building evals internally. We're partnering with academia and partnering with AI companies to build great evals. We have pretty strong views on in various ways for different parts of the AI stack, where there are things that are not being measured well or things that developers care about that should be measured more and better, and we intend to be doing that.

We're not obsessed necessarily with that everything we do, we have to do entirely within our own team. Critical Point is a cool example of where we were a launch partner for it working with academia. We've got some partnerships coming up with a couple of leading companies.

Those ones, obviously, we have to be careful with on some of the independent stuff, but with the right disclosure, like, we're completely comfortable with that. A lot of the labs have released great datasets in the past that we've used to create success independently. And so it's between all those techniques we're gonna be releasing more stuff in the future.

Cool. Let's cover the the last couple, and then we'll why don't we talk about your trends analysis stuff? You know?

Totally. Before that, actually, I have one, like, little factoid on If you go back up to accuracy on Odysseus, an interesting thing about this accuracy metric is that it tracks more closely than anything else that we measure the total parameter count of models. Makes a lot of sense intuitively, right, because this is a knowledge eval, this is the pure knowledge metric.

We're not looking at the index and the hallucination rate stuff that we think is much more about how the models are trained.

the leading frontier models right now are quite a lot bigger than the 1,000,000,000,000 parameters that the open weights models cap out at and the ones that we're looking at here. There's an interesting extra data point that Elon Musk revealed recently about x AI that Grok three and four 3,000,000,000,000 parameters for Grok three and four, 6,000,000,000,000 for Grok five, but that's not out yet. Take those together.

Have a look. You might reasonably form a view that there's a pretty good chance that Gemini three Pro is bigger than that, that it could be in the five to ten trillion parameter range.

Yeah. Great. I mean, obviously, if you're a developer or a company using these things, none of exactly as you say, it doesn't matter.

You should be looking at all the different ways that we measure intelligence. You should be looking at our cost to run index number and the different ways of thinking about token efficiency and cost efficiency based on the list prices because that's all it matters. It's not as good for the content creator rumor mill where I can say, oh, GPT four is this small circle.

Look at GPT five is this big circle. And that and then that used to be a thing for a while. Yeah.

I mean, but that that that is like a on on its own actually very interesting one. Right? That Is it?

Well, just purely that chances are the last couple of years haven't seen a dramatic scaling up in the total size of these models. And so there's a lot of room to go up, probably, in total size of the models, especially with the upcoming hardware generations.

Yep. Yeah. Each of the tasks have a whole bunch of detailed instructions, some input files for a lot of them.

It's I with within the 44 is divided into, like, 220, two to five maybe subtasks that are the the level of that we run through the agenticarnas. And, yeah, they're really interesting. I will say that it doesn't necessarily capture, like, all the stuff that people do at work.

No eval is perfect. There's always gonna be more things to look at, largely because in order to make the tasks well enough to find that you can run them, they need to only have a handful of input files and very specific instructions for that task.

I think the the the way that we're that we're thinking about the places where it makes sense to use an LLM as judge approach now, like, quite different to some of the early LLM as judge stuff a couple of years ago. Because some of that and MTV, which was a great project, was a good example of some of this a while ago, was about judging conversations and, like, a lot of style type stuff. Here, we've got the task that grader and grading model is doing is quite different to the task of taking the test.

When you're taking the test, you've got all of the agentic tools. You're working with the code interpreter on web search, the file system to go through many, many turns to try to create the documents. Then on the other side, when we're greening it, we're running it through a pipeline to extract visual and text versions of the files and be able to provide that to Gemini, and we're providing the criteria for the task and getting it to pick which one more effectively meets the criteria of the task out of two potential outcomes.

It turns out that we proved that it's just very, very good at getting that right, matched with human preference a lot of the time, because it's I think it's got the raw intelligence, but it's combined with the correct representation of the outputs, the fact that the outputs were created with an agentic task that is quite different to the way the grading model works, and we're comparing it against criteria, not just kind of zero shot trying to ask the model to pick which one is better. Got it. Why is this an ELO and not a percentage like GDP val?

Like, model has to go find clips on the Internet and try to put it together. The models are not that good at doing that one for now, to be clear. It's pretty it's pretty hard to do that with a code adapter, and the computer yourself doesn't work quite well enough and so on so on.

Yeah. So we, like, haven't grounded this score in that exactly. I agree that it can be helpful, but we wanted to generalize this to a very large number of models.

It's one of the reasons that presenting a ZLO is quite helpful and allows us to add models, and it'll stay relevant for quite a long time. I also think it it it can be tricky looking at these exact tasks compared to the human performance, because the way that you would go about it as a human is quite different to how the models would go about it. Yeah.

I also like that you included Llama four Maverick in there. Is that, like, just one last, like Well, no. No.

No. No. No.

No. It is the it is the best model released by Meta, and so it makes it into the homepage default set still for now.

Oh, my backwards explanation for that would be that, well, it's meant for consumer use cases, here you're pushing it for something. The constraints are different and the amount of freedom you can give the model is different. Also, you, like, have a cost goal.

We yeah. Let the models work as long as they want, basically. Yeah.

Do you copy paste manually into the chatbot? Yep. Yep.

That's That was how we got the chatbot reference. Once. Yeah.

We we're not gonna be keeping those updated at, like, quite the same scale as putting it on the hundreds of models on the house. So and, you know, I don't know. Talk to browser based.

They'll they'll automate it for you, you know, like True. Yep. Yeah.

We should.

in themselves. Yes. Right?

Yep. And that's grown a huge amount over the last year. Right?

Like, the tools that are available have actually diverged in my opinion a fair bit across the major chatbot apps, and the amount of data sources that you can connect them to have gone up a lot, meaning that your experience and the way you're using the model is Yeah. More different than ever. What tools and what data connections come to mind when you say what's interesting?

What what what what's notable work that people have done? Oh, okay. So my favorite example on this is that until very recently, I would argue that it was basically impossible to get an LLM to draft an email for me in any useful way because most times that you're sending an email, you're not just writing something for the sake of writing it.

Chances are context required is a whole bunch of historical emails. Maybe it's notes that you've made. Maybe it's meeting notes.

Maybe it's, pulling something from your, any of, like, wherever you work store stuff. So for me, like, Google Drive, OneDrive, in our super base databases if we need to do some analysis or some data or something. Preferably, model can be plugged into all of those things and can go do some useful work.

Based on it, The things that, like, I find most impressive currently that I am somewhat surprised work really well in late twenty twenty five are that I can have models use super base MCP to Query. Read only, of course, run a whole bunch of SQL queries to do pretty significant data analysis and make charts and stuff and can read my Gmail and my Notion. And okay.

You actually use that. That's good. That's that's that's good.

Is that a cloud thing? To various degrees of water on both JGPD and Claude Yeah. Right now, I would say that this stuff, like, barely works in fairness right now.

Okay. Okay.

yeah. I I think it is true that I have never actually sent anyone an email drafted by a chatbot yet.

Anything else? I mean, we can equip it with more tools, but, like, by default, yeah, that's it. We we give it for a g d p eval tool to view an image specifically because the models, you know, can just use a terminal to pull stuff in text form into context, But to pull visual stuff into context, we had to give them a custom tool.

Yeah. But yeah. Exactly.

Yeah. You you can explain an expert. No.

you know, base for, you know, generalist building a generalist agent. It is kind of cool. For more specific tasks.

I'd say the best way to use it is Git clone and then have your favorite coding agent make changes to it to do whatever you want because it's not that many lines of code and the coding agents can work with it super well. Well, that's nice for the the community to explore and share and hack on it.

numbers, I guess. I don't know how you so classify all these. Yeah.

Or call call let's call it the last of, like, the the three new things that we're talking about from, like, the last few weeks. Because yeah. I mean, there's a we do a mix of stuff that where we're using open source, where we open source on what we do, and proprietary stuff that we don't always open source.

Like, long context reasoning dataset last year, we did open source, and then all of the work on performance benchmarks across the site. Some of them we're looking to open source, but some of them, like, we're constantly iterating on and so and so and so on. So there there is a huge mix, I would say, just of, like, stuff that is open source and not across the site.

This here is call it like a new way to think about how open models are. We, for a long time, have tracked where the models are open weights and what the licenses on them are, and that's like pretty useful. That tells you what you're allowed to do with the weights of a model.

But there is this whole other dimension to how open models are that is pretty important that we haven't tracked until now, and that's how much is disclosed about how it was made. So transparency about data, pre training data and post training data, and whether you're allowed to use that data Mhmm. And transparency about methodology and training code.

Mhmm. So, basically, those are the components.

trying to understand the holistic picture of the models and what you can do with all the stuff the company is contributing, this gives you that picture. And so we are gonna keep it up to date alongside all the models that we do in Telesatix on on the site, and it's just an extra view to understand. Can you scroll down to the the the trade offs chart?

The the yeah. Yeah. That one.

Yeah. This this really matters. Right?

Obviously. Yeah. Because he can be super open, but dumb.

I mean, this one obviously goes the wrong way here. Right?

Is the access to to hill climb. Yeah. Unfortunately, it might be fundamentally true that the the slum will always go this direction because once you open something up, then everyone else can get to the level of what you have now.

Well, so let me let me tweak your point system. Right?

It is hard to wait for the materiality of the contribution to to open source. Like, it's we we we tried to make it so that it is quite well defined and no one can disagree about, like, which, category things should be in. So we're not saying, like, this was a big contribution or a small contribution in terms of, impact on the industry or anything.

It's just, like, how much of your data did you release.

Totally. But I mean, but it it is true that NVIDIA have actually put an enormous amount of effort over the last year especially into the Nematron models. Yeah.

And so many people actually use it for, like, synthetic data and stuff.

I mean, it's it's in your interest for there to be more AI.

users. We have like a detailed breakdown of that in the openness index, and that was actually one of the initial questions, like took us down the route of wanting to do this. Because yeah.

The simplest thing that, like, our opinion is is that there's a lot of advantage to having like an official OSI license like MIT or Apache two because then the box is just checked. You don't even need to read it because it's just Apache two, and you can do whatever you want, and it's fine. There are often very good reasons that companies don't want to release language models with those completely open licenses.

If you get So the top category, that's one of those licenses, you're totally good. And then we've got some lower categories for when attribution is required, and then when commercial use is not allowed.

One, that the cost of intelligence has been falling dramatically over the last couple of years. The best way to think about that is that the cost for each terror of intelligence Sure. Has been dropping.

The like, one fact on that is that you can get intelligence at the level of g p d four for over a 100 times cheaper than g p d four was at launch right now. I think my number is a thousand, actually.

statement is normally like A 100 is because you're out. Yeah. But in fairness, this slide, like I we were actually saying before the podcast, right, is like maybe six months old now, and it's conceptually still correct, but, like, could actually probably do a tweak on the exact numbers because, like, the market's moving so quickly.

Feel free to kick it off. I I mean, we'll have this right.

There are two trends that seem to not make sense together, both of which we talk a lot about at artificial analysis and are very important to developers building stuff in AI. The first is that the cost of intelligence for each level of intelligence has been dropping dramatically over the last couple of years. We track the cost to run artificial analysis intelligence index for each bucket of intelligence index scores, and each bucket, you just see the line go down really, really quickly, and actually go down more quickly to each new level of intelligence that's been achieved over the last couple of years.

So the rate of that cost plan has actually been going up. So we've got that being true, and yet it is clearly possible to spend quite a lot more on AI inference now than it was a couple of years ago. NVIDIA stock go up.

It's going it's going really up. I just heard from a friend's startup that just went to the age of zero. They're spending $5,000 per employee on coding agents spent alone.

That's ridiculous. That's an impressive number. We need to get our numbers up.

We're we're we're not gonna be any hitting Well, I was it's so high that I'm like, are you doing something wrong? Yeah. Because there are some efficiency questions along the way.

But, like, you can make AI inference useful to that level in a bunch of ways that I can imagine. Right? Yeah.

I I don't think that nuts. But basically, the the reason we made this slide to answer the question, right, is to show that the crazy thing is that it is actually true. We've had this 100 x to a thousand x decline in the cost of GPT four level intelligence on the left hand side, and yet on the right hand side, because the multipliers are so big for the fact that even though small models can do g p d four level now, we still wanna use big models and probably bigger than ever models to, do frontier level intelligence.

We've got reasoning models using tokens, and then we're throwing them these them in these agentic workflows where they're consuming enormous numbers of input tokens and making enormous numbers of output tokens working for a really long time.

or it's more of like a power story rather than like a a share sort of compute tokens efficiency story. But, yeah, what what's going on in in hardware? Okay.

So the the the the odds, unfortunately, is it depends, and it just depends massively on, like, so many things across a bunch of different types of workloads and ways to think about it. So one of the simplest ways to think about this is to take single relevant model, to think about serving it at speeds that are realistic for what you actually might wanna hit and can afford to hit, and then think about the throughput per GPU that you can achieve serving the model at those speeds. One of the reasons that's important is that there's a trade off between the throughput per GPU that you can achieve and the per user speed that you can achieve.

And as in it costs more to serve stuff fast to to users. When you run all of that, for especially big sparse models, you can get a lot better than two or three x gain going from Hopper to Blackwell generation NVIDIA. I am this shouldn't be too controversial to say, but, like, I'm pretty confident that Black Widow has delivered pretty enormous gains and that the next couple of years of NVIDIA's roadmap are going to continue to deliver quite enormous gains, and that those will actually come through as lower total cost per token to the companies that are running models on them, and will allow bigger models, will allow way more tokens to be made for lower cost, and that that's gonna continue.

These things also stack on all of the software and model improvements being made. So basically, like my prediction across, like, both sides of that, like, smile chart that we're gonna see the left hand side continue to be true and probably, like, for another order of magnitude and the right hand side continue to be true for another order of magnitude.

There there must be a lot somewhere. Right? Yeah.

Exactly. But we've got numbers in the wild that are quite a lot lower than that right now.

But that that goes to why this is now such a complex thing. So earlier this year, and probably when you and George last spoke for the AI Engineers World's Fair, we had this great slide that was super easy where we would show that the average reasoning model is using 10 times the number of tokens per query in our intelligence index as the average non reasoning model. And there was this moment where that was a pretty clear distinction and extremely useful to look at it just like that.

Definitely no longer the case, not least because you can think about reasoning strength for a bunch of these different models, but particularly because different models have wildly different token efficiency now, more than an order of magnitude in difference, that means that the way that you probably need to think about cost for any application is to use something like our cost to run intelligence index metric as the starting point for what it's gonna look like for these different models, different reasoning strengths, and this continuous spectrum from non reasoning to reasoning. That's basically, like, where we're at. So we will still show reasoning and non reasoning and define reasoning as when there is that separated chain of thought that you're getting at a different parameter in an API normally, but it doesn't necessarily anymore mean that that model is actually going to have longer end to end latency that is going to use more tokens than something that is branded on a non reasoning model for the same task.

Mhmm. That's true. I think 5.

extra thing. Let's say let's say let's say that we've got that that's a really important extra thing there. Right?

That you've got not just the average number of tokens being used by the model, which we cover really well right now, but the behavior that you want in the model is it to use more tokens when it needs more tokens and not to use more tokens when it doesn't need more tokens. So that's what OpenAI were basically claiming that 5.1 codecs is better at.

We don't actually publish anything on this right now, but have tracked it a bunch internally in our internal analytics on evals across the models that we run, where we look at the difficulty of the questions and the correlation between token usage and difficulty. And net net, surprise surprise, like models have got better at doing that over the course of this year. I think going into next year, that's gonna be really important, especially as you multiply it by the number of steps in an agentic workflow that a model has to take to get to an answer.

We are going to care a lot about token efficiency and number of turns efficiency for getting to what we want. Which would you rather have? Token efficiency or number of turns efficiency?

Or like which is more important to work on? It depends on the application and both are gonna be really important. Yeah.

benchmarks have been single turn, but I wouldn't say they need to be at all into the future. Right? Like, we have a couple of agentic benchmarks in the index right now and GDPR that we were talking about.

We let the models do up to a 100 turns and, our stirrup agentic harness to do that eval, and we're gonna build similar stuff like that in the future. It definitely is hard and you've got whole kinds of infrastructure problems to run that, and exactly as you say, parallelize it because we need to run that on hundreds of models, and we wanna do that really fast when new models come out and when labs want us to run it on their models. But you can do it.

We're putting in the work to build that stuff, and it's gonna be great. Okay. So we've covered I mean, there's a lot more to cover.

And you can see it as a good good to go bad things depending on what your view is, but it means that we have a quite active creative direction approach to trying to understand what creative professionals and users want to do with those image and video models, and so that we can be directing the arenas in our categories toward gathering votes on what people care about. One call out actually to listeners, like, if you are using our arenas, is that you can submit requests to us for things that we should cover. I didn't know that.

Yeah. Understudied categories, areas that you think the models are bad at and the labs don't focus on enough. Like, if you want something solved, one of the levers that you have is send us a couple of prompts on it.

We might be able to get a category going on it. And this thing that we were talking about earlier, right, that once things get measured, they can get targeted, you can make that work for you. For me, as a content creator, infographics.

on most of our charts, the lines go in a particular direction, and our overall prediction is the lines are gonna keep going in that direction. We're we're gonna do a lot and do a lot to be as useful as possible to developers and companies to measure what's important on every one of those and along those lines. But I think we're gonna talk about similar stuff.

It's just that we're gonna have continued on this trajectory for another year, and things are gonna feel pretty different because of that happening. I know that's the boring answer to that question. No.

No.

in March. Why don't we break it now? How soon is the podcast gonna come out?

Whenever you want. Okay. So we're we're at v three right now.

So the so the version that we that's going inside is is is v three. V four is what we're gonna call the next, you know, major update. Surprise.

Surprise. We're going to be adding several of the things that we've actually talked about today that we've launched over the last few weeks. So that's not going to be wildly shocking, but some of the things that are most exciting is that adding GDP val is going to give us this general agentic performance in a really strong way in intelligence index.

And in Critical Point, the Physics eval George was talking about, similar to frontier math, that gives us completely new view with a brand new dataset of very, very hard research problems. We are going to be using omniscience and we are going to be using hallucination rate. The exact ways that all of those are gonna come together The waiting is gonna be hard because the numbers are different.

Yeah. Yeah. We're we're we're gonna make sure that we don't do anything to cause odd distortions and stuff that could be misleading.

But every time you version it, you have a one time reset of them. Exactly. You know?

Yeah. Yep. That's exactly how we think about it.

We will make sure that within each version number that there's no drift in any of the scores so that people can rely on them and reference them. Yeah. You just have to watch out for that version number.

Once it's v 4.1, those numbers won't be compatible with v four. Of course.

Potentially, for the earlier versions, Tile two bench telecom Yeah. We're pretty convinced is pretty good. If anything, the only issue there is that models have got very good at doing it.

And so, like, anything TALT three. Yeah. Yeah.

On we go. Yeah. On we go.

Okay. Well, thank you so much for providing such a great service to the industry. I'm I'm glad to at least know you guys as before you got famous, and now you now you are famous.

So Oh, look. Our pleasure. And we really appreciate your support along the way.

Like, I I wasn't kidding at the start. Right? That it was quite material moment for us, like when artificial analysis was covered on latent space.

Some random guy in San Francisco mentions you in. I was a fan of latent space for like a year before you mentioned us. So I've been I've been listening.

I don't think I was, like, familiar with, like, you personally yet at that point, but, like, I was I listened to your voice probably for many, many hours. And so once, like, mentioned you it, then, like, got to get to know you and, like, met you for first time nearly a couple of years ago. Like, it was really cool, honestly.