The Benchmark With No Instructions — ARC-AGI-3 (winning team!)
Machine Learning Street Talk (MLST)Episode Description
Summary
Tim Scarfe interviews the Tufa Labs ARC-AGI-3 team to dissect their winning approach on the ARC-AGI-3 benchmark, focusing on how their system discovers goals and balances exploration with action efficiency. The episode explores the challenges of goal acquisition, the role of language models in reasoning and planning, and the evolving design of ARC-AGI benchmarks to better test general intelligence.
Chapters
Topics
People
Key Concepts (18)
References (6)
Transcript (122 segments)
Are all batteries the same? That's like asking if all soccer players are the same. Take Messi, the most decorated player ever.
Is there any other player who has achieved that? No. Just him.
Now take Duracell. Is there any other battery with power boost ingredients inside? No.
Just Duracell. Remember, goats only trust goats because they are built different and messy only trusts Duracell.
This episode is brought to you by Google Chrome. You think you know a browser, but Gemini in Chrome? That's new.
It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50 page restoration block or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense?
There's no place like Chrome. Check responses, setup required, compatibility, availability varies 18 plus.
But we have the benefit of millions of years of evolution. Right? And so it's almost a bit unfair that we're expecting AI algorithms to be able to do all of that.
Exactly.
I think it boils down to one of the big open questions in the field, like is language critical to intelligence? I will say, let's say, at least when I'm playing the games myself, and I think that goes for all of us, I'm using language. My name's Jerome Cotard.
I have a background in physics and mathematics, I've been working in the industry for about fifteen years, with the last five years specializing more towards AI. It was immediate to him what the goal was. He recognized the pattern, and I showed him the game.
Within three seconds, he finished the first level. Often the agents start thinking that reducing the energy bar to the minimum is the goal or that stepping 10 times in a region is the goal, which for a human is kind of clear that that it's not the actual goal. My name is Stefano.
I study computer science machine learning. I did some research in reinforcement learning at EPFL.
I guess the million dollar question though is, do you think it's possible in principle to do really well on ArcoGI three and be no closer to edge UI?
Yes. I do think it is possible.
I don't know. I guess the main idea is a bunch of bright people in the room and do good research together.
they can do a very good job of pretending essentially. That is, in a sense, indistinguishable.
36% might be misleading as a number if you don't look behind it. So what it really measures is action efficiency. My name is Dries Smith.
I studied electrical and electronic engineering.
I specialized in reinforcement learning for the past five or six years now. And I've been doing on the research side, ARC AGI two and three, and also some quant work in between. Yes, perhaps that tells something about the benchmark.
There's some leakage of human Hello.
I'm Michael. I am my I I do masters in data science and I'm super excited about ARC and I love solving it. In the example that we saw, we have a maze.
Right?
native primitive in a newborn that says this is a maze. Okay. So this is one of the official Arc AGI games.
This is one of the first games that was released. It's called Alice 20. And here you can see a 64 by 64 frame, which is typically how the game observations is presented to the agent.
So you get an initial frame and each of these pixels can take one of 16 colors. And now you need to figure out what the goal is and even how the player moves, whether it's a specific player or object that's moving around or some other more abstract game.
You need to figure this out. So very often our harnesses will break here. So they actually got the idea right.
You need to move to the exit with the two things aligned. But then but they they they think they should win here, but actually you need to move up one more move. But it is quite common actually that they fail to beat the level from this point because they cannot think that you they think, no.
But that's that's outside the maze, you can't move there. Yeah. And the human would just, whatever, just give it a go and see what happens.
Yeah. But somehow, we can't get our harnesses to also do that. The the best recipe we have today to build the intelligent systems is scaling up these large language models.
I think Sam said you should definitely not be trying to to train LLMs yourself. Nah. I don't recommend it.
Why not? I think the easiest is just go work at Anthropic. That's the easiest.
I have a have a sense. I cannot prove anything that this is the time things start to get weird. Yeah.
I think he's right about that. Most fundamentally is I want to understand intelligence, how the brain works, and how to build intelligent machines. It's a big mystery.
I guess when I was young, I was excited about both computers and brains. Computers, I feel now we have a clue how it works. And I did computer science studies, so that mystery is kind of gone.
But the mystery of about brains is very much alive. I have a strong sense that most of society is bottlenecked on cognitive labor and insight and basically intelligence.
that would be a big unlock for many things. You won the preview competition and I was using a different method. Yes.
And then you found that that didn't work very well when the main competition came out and you moved towards this. Yes.
agent preview competition last year, I actually tried something completely different. So the goal of that competition was basically to test whether there is obvious solutions that breaks the mold, breaks what they're actually trying to accomplish. And there was, and the solution was basically just to brute force actions.
So what that solution did was it basically searches through the large action space for ArcGIS three. We have more than 4,000 actions, which makes this difficult problem to do, you but could theoretically search for all that or just randomly search for actions. What I did for Stochastic Goose was basically to search for a large number of possibilities, but I tried to only search through actions that results in a frame change.
So initially, if you took an action that did not do anything in the game, nothing would change, not even the timer bar. That allows you to model that behavior to see, okay, does this action change the time bar? If it does not, we downvote that action in the future for that given frame.
There was also only three games, so you have to be selective of how you improve things. You can't pre train because you will just overfit. Exactly what I did was I just used a action model that learns which frames are valid for a given transition.
And then it was more about the engineering of being able to learn within 100,000 actions, because that was kind of the max action limit we could get out in the time limit we had. So that seemed to have worked well. It solved, I think it got 18 levels completed out of the three games, which was, I think it solved two of the games as well in the time limit that was provided to us.
But they then hardened the competition specifically against that. So the new competition, the games are much harder and the timer bar moves even if you use an action that's valid, but doesn't really change anything in the game. And more importantly, they introduced action efficiency.
So this makes it very difficult to brute force. You have to be very direct of how you explore. And that's where LLMs come in.
So you could do, even though it's slower, just being able to somehow guide the exploration helps a lot. It's like the possible combinations is too many to just brute force and obviously your score goes to zero quite quickly. So they've made it so that if you go just let's say 2X or 3X above the human baseline, you're already close to zero.
And yep, even though it's slow, it just helps. Kind of untractable for us. We tried other methods such as directly predicting like more of a transductive method where you just have the input frames as context into a long sequence model, and you just predict the actions, but it doesn't generalize well.
And it doesn't really make intuitive sense a lot because if you play the game, for example, the first game here, the maze level, you would intuitively play one or two actions and think about your path and it will go to the end position. But if you have to think at every step with the same computational capability or same budget, then yeah, you might be misrepresenting where you should go at the start, but at the end for straight lines, for example, you don't have to think that much, you can just batch those actions. So that's yeah, where the coding agent idea came from.
And we also had a lot of two good literature results where they scaled using Opus models. One was ArgenTica and the other one was the RGB agent, which also showed good results given no compute constraints. You could use closed source models.
And we took that as inspiration.
Yes. You mentioned transductive as well, which is quite interesting because roughly speaking, I think of transduction as you're making a prediction about this specific test instance. It's quite an interesting discussion whether or not this is transduction because even though it's chain of thoughts, it it looks like a form of induction in the sense that it's a rationale that could be cross applied in the future.
So you could use the memory in the agent. You could do some kind of library transfer and make it inductive. But at the moment, if it's only for the sole purpose of this particular problem, would you call it a transductive method?
as you mentioned. You can actually read the reasoning trace and understand when it's understanding the game and making progress and when it's not. Yeah, because it has this reasoning chain of thought, which is in English and you could reasonably understand it.
So yeah, I would actually say it's more inductive. Like previous attempts, as we mentioned, was where the agent actually just directly predicts the actions. That will be more transductive and that doesn't seem to work at the moment.
Obviously there's a lot of ideas to try and I'm sure the community will come up with something interesting to make that work. But for now, seems to be the way for us.
this problem really interesting because there's so little time to learn and really, if you wanna get the 100%, which is the which is the grand prize, then you really you you have to outcompete a human, really can take a lot of time just to think. And time is not a in question for the solutions. So I think that's really interesting and the reflections and looking back at the past, at your past experiences makes makes LLMs very flexible model for this solution.
Yeah. And interestingly enough, also a lot of even though there shouldn't be any priors, there's already a lot of the game priors are already included in the LLMs. So for example, here we're looking at a maze.
And maze is something that every LLM, even the small ones, will know and will recognize from beat images or ASCII grids.
goose solution wouldn't have that encoded anywhere in it, that the mace is a thing. And that really helps to direct the reasoning and the actions of the model. So the one of the main constraints was basically two weeks.
I joined the competition late, two weeks to to do it. So there is a lot of things that can be improved, but I think it was a good initial solution. So the main constraint was we only had three games, three public games and three private games to evaluate on.
So we had to make sure or that to make sure that whatever the design you don't assume too much about the environments and the games because on the private leaderboard or like the public leaderboard beforehand, could actually see that I wasn't even close to the top because it was super easy to overfit. So the basic idea was we can't pre train. We don't have enough environments.
Let's try to do test time training like we did for ArcGI two. But how do you do that? Like you can try reinforcement learning, which I have a lot of experience with, but it's not that obvious how you do reinforcement learning because every time, so you get a new game at this time, only real reward you have is a level transition.
But once you transition a level, you never go back. So it's not like you're trying various routes to try and optimize for passing level, you just have to pass it once. So it didn't really make sense to use pure reward based RL.
I did a lot of research in the past in curiosity of unsupervised RL. And I thought perhaps that's a better approach where we don't have explicit rewards. We're just optimizing for curiosity, like exploring new areas that the agent hasn't explored before.
So I tried a bit of like world modeling. So the basic idea is the games are deterministic, so you can use a world model to perhaps have the input frame, then the action and try to predict the next frame. And then if you can't predict it well, you have the policy explore that more and that's the reward you're optimizing.
So that was a promising approach. I couldn't get in it working in two weeks. So what I just then assumed is that any frame change is interesting.
You wanna explore things where the state changes, you don't wanna explore things where the state stays the same. So that was the basic derivation of the, or like path I followed towards getting the action model implementation for Stochastic Goose. And then it was down to basic reinforcement learning things like how do you get it to learn within a thousand time steps?
Because you have a 100,000 time steps, you need to start taking useful actions in like a thousand time steps. So it was like using a replay buffer, hashing experience so it doesn't go over the memory a bit. You have to do some prioritized experience replay and some basic engineering stuff to get a work of that competition.
Yeah, you mentioned exploration as well. Guess there's a bit of an elephant in the room, which is that Cholet is talking about the acquisition and synthesis of abstractions. And when reinforcement learning folks talk about exploration, it seems to be in quite a surface superficial way, so in terms of like entropy or things changing.
And do you think is that in any way against this idea that we can acquire deep abstractions about the domain? Yes. Definitely for sarcastic goose.
It was not at all what I guess what Fontra at the end wants as a good solution for Arcadia three, but I can't speak to what he actually thinks is a good solution, but that was purely to see if I can benchmark, maximize this competition within two weeks. I think the duck and things we are doing on that side is perhaps more in line with what that initial vision is. For example, you have this reasoning tokens, which has like abstract representation of objects, which we then manifest or write down in like English language textual tokens.
So I would say that's more abstract, you identify objects. You've tried to find out what the mechanics is, dynamics of the game, what the goal is. That might be more abstract.
And it's also kind of like a neural guided search and also the programs, the Python programs it creates is like executable Python programs to abstract and also build some sort of simplified world model and search over that world model using actual algorithms like breadth first search. There is understanding debt, right? That you're building this really, really complex thing.
And after a while, have you noticed in Claude code, they don't even show you the code anymore? Right? You can expand it, but by default, a lot of people don't even look at the code.
So do you think that you need to be familiar with the deep abstractions in your code base in order to kind of build mental models and evolve it and extend it? Do you get lost in no man's land? Yes, this is a, yeah, if you will.
This is an active discussion with an R team. We don't have a discrete answer yet, but I think what we've found is like you need deep understanding of some important parts. Some other parts, like let's say a web viewer, you can vibe code more easily because that's not really If it breaks, it's fine.
But the core logic for, let's say, the implementation and also evaluation is especially important. We have to make sure it's working, but it is a case that gradually we're understanding less and less of our own code base. And we're struggling even with reviewing some of the changes is you might use a codex to help review some of it because it's such a broad change or we need to split it up.
But things are moving so fast that you can't just manually, yeah, if you just manually write everything, you can't keep up with the rest of the team. So yeah, I guess this is an active discussion. It's difficult.
As for me, my career has been going on a bit longer and let's say for most of my career, there were no coding agents, which means I also have a bit of an opportunity to still make use of patterns that have been useful in the past. I think one important one that as a team, we are more and more learning to use properly is requirements based engineering. So we will formally write requirements, let's say, really following the, let's say, detailed prescripts like numbered requirements and to specify how they are tested.
And I mean, we might still have the coding agents helping there, but mostly handwritten. We will review that as a team. And from there, the coding agents do, we can much more confidently hand it to a coding agent to implement than if it's just a Viper single prompt.
Yeah, I love all of that, but I suppose there's always been this thing, right? That in software engineering, we have specifications, we have documentation, we have user stories, we have requirements and all these things. And there was waterfall versus agile.
For me, the difference between waterfall and agile is this admission that there is epistemic uncertainty. We need, like reality pushes back, we discover contours in reality that we didn't see before, and we need to adapt the requirements over time. And my worry is that when you crystallize the process, I guess what I'm saying is there's more to tacit knowledge in developer teams than the requirements.
There's the generative process that created the requirements in the first place. And when you only write software for the purpose of satisfying the requirements, then the ability to evolve the requirements gets kind of deteriorated?
Yeah, so what is always the case, and that was the case in classical, let's say, precoding agent development as well as it is now, is that it is not your right requirements, and those requirements are frozen and carved in stone, and then you write code. It is an interplay. So when you start implementing, might realize, wait a minute, this combination of requirements is not possible.
You don't then just implement something that approximates the requirements. You go back, and we also kind of tell our coding agents to do this. You go back and say, Hey, these requirements don't work.
They need to be adapted, and you make a conscious choice there. So we will also very often kind of go back with the coding agent, Hey, can you now very carefully check that every requirement is satisfied? In fact, have it write specific text per requirement, this is how it is being satisfied in the code.
We try to also really write the tests around the requirements rather than just having kind of random tests for every line of code. And then you get that classical interplay that you also have in classical engineering. You can do with your coding agents.
This is a kind of AI human co creativity.
So humans have the taste, they have the intention, they have something that the coding agents don't have. Because, you know, the bull case, wouldn't it be amazing if you could stick the requirements in and the agents themselves would kind of understand, oh, I see what they meant and they could evolve the requirements? So I think in the short term, I can't say for the long term, at least what we're trying to do is find the areas where we can the model still failing, like high level conceptual ideas about the problem.
And then we try to make sure when they say we're reviewing code or creating code that we ask specifically around that areas and make sure it's correct and understands us well. But yeah, I guess it's always gonna be a game of cat and mouse where we're gonna perhaps hand over more to the AI assistants, but in some areas it might just totally fail and we need to take responsibility there and make sure we understand what it's doing and it understands what we want. But yeah, longer term, I guess if you're AGI pulled or a super intelligent pulled, I guess it's, yeah, it's up for debate with.
because something that would just sit in your scratch pad, you can just paste it in a prompt and see if it sticks. So you can definitely experiment overnight with a new idea. I think the really the added value that I see of the lab right now is and what the AIs are missing, what the what the coding agents are missing right now is like literally the the the attention to detail.
So this task is very long. Like, if we have a long context, the execution of one game might take an hour just for it to solve one level. And just seeing and if you could in principle point the auto research at this and say, hey, just improve the score, right?
But that doesn't work because it doesn't really it cannot really it we have tried that and that has not really led to improvements because it doesn't really understand where the agent fails. It's not simply as Andreas Carpathi's other research challenges where you just minimize a loss and there's just parameters. You need more insight to this task.
like, it it came from reading deep into the program, into the logs, and understanding what the issues are and improving those one by one rather than just like, vibe coding another 2,000,000 lines of code. With coding agent trying to optimize this task, we observe kind of similar failure mode that what we observe when they play games. Like, for instance, if you give them an auto research task, they might overfocus on details and try kind of not see the big picture and not kind of zoom out and see things from Fireball.
And they might, I don't know, start optimizing some hyperparameters or, like, at 0.01% improvement while there is, like, some other big improvement that could be done, which is kind of in another obstruction level. And we see very similar failure mode in games where if the agent at the beginning of the gameplay, they start they get kind of locked in on the wrong hypothesis, it's extremely hard that they escape that.
Like, they they kind of convince themselves that that's the right path and the right hypothesis to go, and they're not able to escape that.
zoom in in the details and then seeing the higher level picture at the same time. I just wondered, like, how do you guys think about the abstraction mountain? Because for Cholet, it is bottom up.
So you start with the core knowledge, you know, like agents, spatial knowledge, stuff like that, and and you kind of synthesize upwards. Whereas LLMs are interesting because they learn these fractured and tangled representations that are quite high level, but also quite generalizing. And then you can stitch them together and you can repurpose them and cannibalize them in different contexts.
But it's always a little bit lossy. So if you're doing this library learning and transfer, you always have this issue of like, well, yeah, there's this abstraction over here and it kind of works, but it's not quite what I want, but I can still use it anyway. I mean, kind of reasoning something that should be first class or do you think it could just be implicit in the harness if we had a memory system?
were just describing how LLM's will have indeed maybe let's say, time to call knowledge priors as something somewhat vague, but not fully testable, but still usable. The thing is, if you replace LLM by human there, I think it's still true. We also don't have formal core knowledge priors that are, let's say, fully exact and that we apply to the world and describe the world fully exactly.
We have kind of vague, trained over our life understandings that evolve over time. So, at least if you want human intelligence, that does seem to be the paradigm you want there. And I think the LLMs get kind of the bonus that they also can reason very well and much more implicitly than the human scanning code.
So if they want to get exact, they can.
vibe understanding, basically. I'm guilty of always like, you know, suggesting what Charle thinks. I don't really know what he thinks.
But, you know, I've I've got a pretty good simulation of Charle in my mind. But I I I think he's he's influenced by this kind of, nativism psychology type thing. And and he thinks that a lot of the reasoning we do do is almost platonistic.
Somehow the laws of nature imputes these primitives into our mind, and we compose these primitives together for certain classes of problems. So we can do abstract system two reasoning. For those types of problem, we do compose these things together.
But you're absolutely right. There's so many things in the world are actually really complicated, like navigating relationships or even pathfinding in a complex environment, you know, on the tube network or something like that. So we do a little bit of both, but there's there's at least a pocket of pure reasoning.
I think that's what Charlotte thinks.
LLMs pretrained on the Internet? And then if you apply it to a new pattern using, like, in context learning, is that a form of okay, let's say with reasoning as well, would that not be a form of like perhaps skill acquisition on the fly because it learns obviously, there's a lot of core knowledge as well, but it's also a different part where it has to adapt on the fly. So you can give some example that might not be on the internet and it might adapt, sometimes it fails.
It is still a distribution, but at least you get some adaptability. And then for example, when you started prompting it, think of like a chain of thought prompting, it started improving because it had most time to reason and adapt to what you're asking. And then if you train it using reasoning, you can actually do way more because now it has more time to actually reason and figure out what you're actually asking and form new extractions for solving a specific problem case, as opposed to just regurgitating what it's seen on the internet.
LLNs trained to do reasoning, they are intelligent because intelligence is just adaptation. So it's, I know this now and I need to combine what I know in several steps to solve a particular task. In the partial knowledge regime where we have a verifiable function and we can do hill climbing.
But what we see though is that they combine together fractured and tangled representation. They get the right answer but for the wrong reasons. So they create a spaghetti monster.
But what we do is we take a more valid path and we can acquire and reuse abstractions. The models themselves, they don't seem to work bottom up. Work, they have an understanding, they have representations, but they're very fractured and entangled.
And that's still useful, but that's a kind of statistical low level intelligence.
but we're kind of in statistical land. It's performance, not competence. I don't have strong views on this, but I have a feeling that we are doing the same with our minds.
Like if you look at the evolution process that derived us, we're kind of where we are at at the moment is also we've been evolved to work well with spatial information with, like that's one of the core priors in our brain and some level of reasoning that might also be why intuitively understanding quantum physics is so difficult is because we kind of overfit to our world and how we've been evolved. So yeah, I'm still not sure whether the human brain is also like just this core priors and like pure reasoning. There is some guided search to find answers based on what we found through evolutionary process to actually work well to actually survive in the world and yeah, propagate.
It's the million dollar question, isn't it? Because our brains think symbolically, but our brains are actually just a bunch of neurons. So is it a post hoc confabulation or is it actually like a causal basis function of our cognition?
And if it is just kind of confabulation and it's not the basis, then maybe AI doesn't need it to be the basis either.
using neural networks as guidance. It's a tricky one, because for me, coming mainly, let's say, from a more classical machine learning Bayesian perspective, it feels like this should be the right way to go. And certainly when I learned about these things, and kind of very broadly also kind of sweeping stuff like DreamCoder into that regime, although it's not quite the same.
The thing is, it definitely feels like it's more natural to think of it in that way and perhaps more natural to the way we reason. At the same time, it is not competitive at this point with LLM based approaches. Who knows what happens in this competition?
But we don't see those approaches necessarily being the best on ARC AGI one and ARC AGI two. And we really don't see how we could use them on ARC AGI three. That could also simply be because they haven't had the same attention, the same resources brought into them.
yeah, there is that mismatch. Why don't they work if they match our intelligence so well? And I would even I'm trying not to use the word symbolic AI in general, because I think the more abstract way to think about this is constraints.
And Charle's been using the language of constraints, we wrote an article about creativity. And think that creation, whether it's intelligence, so in a known domain or whether it's creativity in a transformative unknown domain, is all about just respecting constraints and acquiring constraints of a problem domain. And this is why harnesses and tools and code work so well because you're increasingly imbuing domain constraints into the learning and acquisition environment.
And so we find ourselves in this kind of halfway regime where we have a constrained, coherent execution environment, but it's only really steered and legible in the sense of these functional descriptions that we're talking about. So rather than having some low level bottom up competence, we have a kind of intermediate competence, which is as described in a similar way to the Blind Men and the Elephant.
knowledge. What I find really interesting about the core knowledge priors, and what you said, like we have these representations and they're very fractured, but sometimes somehow they work. I think there's two levels of abstraction.
So what what two levels of intelligence that human has, like there's the coin or surprise, and then there's what we learn through life. And the the games that have been tested with humans have been tested with humans that were not infants for our challenge. And there's a lot more that we accumulate over generations through education and so on, and those abstractions are way more powerful.
In the example that we saw, we have a maze, right? There's no native primitive in a newborn that says this is a maze. But this has been over generations of problem solving through humans very much established concept.
And if we can inject this into an LLM, we can skip many levels of abstraction. And I would say that's why when we play these games, some of them for the first time, they're also very difficult for humans because if this if this abstract if this is done well, and there's nothing like a maze in a game, and it really core boils down to the core knowledge priors, then there's it's very difficult even for a grown up adult to to synthesize the correct rules. However, most often, and that may be a failure of of Arc a little bit in its design, is that these games usually have some sort of, oh, you're shooting a ball or you're solving a mace, you're you have an enemy.
Right? And these these are like as as pure as they're trying to be, they're never going to be pure because they're made by humans. And that's why we can do so well with LLMs, I think.
Because a lot of these, like, the mace is the canonical example or I think like those can be really well done with LLMs.
fractionated understanding. But isn't it interesting that when humans use LLMs, we can make them act as if they understand. Because we can say, well, you can put a framing in the prompt.
You can say, think about this problem like it's a maze, or think about this problem like it's tic tac toe, or something like that. And what you're doing in the prompt is you're basically setting constraints on its generation. And then, you know, like the more you work on a software engineering project, you know, like it doesn't quite understand, you put more constraints and you update the prompt and it's like it converges.
And after a while, you don't need to repeat yourself. It understands. And so it's almost as if when you get LLMs in the mood, when you make them track this perspective where it respects the constraints, then it understands.
But like the million dollar question is, how can we just make it understand autonomously? How can we give it a novel domain and do what we do, which is like saying, ah, this is basically a maze. This is how you need to think about this and get it get it into the understanding.
Good way to look at this is to look at the function that they learned during training. Because like LLM becomes very good at understanding languages and operating on anything that is kind of language related.
And then all of the work that we do in the harness is trying to put this novel domain into language. So, like, trying to kind of represent the game or whatever domain you're trying to make the LLM act on in a in a way that is kind of language related, so it's closer to its pretraining prior, and it allows to act on it.
domain compared to what was trained on. Yeah. I also want to comment a bit.
So you mentioned indeed that what you often, I think is easiest to do with coding agents, your analogy where you kind of have to say, yeah, but you need to prompt it a few times before it really gets what you want. But I don't see, let's say having worked a lot also with instructing human teams of software engineers, for example, I don't really see a huge difference there in the sense that you cannot expect to just tell someone the 2¢, I want something like this, and then get the right thing. That's a failure on your part not to specify what you want, and we see, indeed, this requirements engineering, requirements based engineering, which takes a lot of thought.
We might spend days getting the right requirements. But then it is, I would say, reasonably well capable of doing the right thing to begin with, basically. But yeah, it's also our job to specify it, right?
I call it perspectival blindness.
So you hire a new person and you build an internal wiki and you specify all of your learnings and you think, oh, I've distilled this, I've crystallized it. The new starter will just be up to speed within a couple of days. It doesn't work like that.
Experience is this weird tacit, almost like a causal graph that has a provenance. And it's only when the new starter plays with all of your tools and tries different things and has counterfactual representations and whatnot. So yes, a very human thing as well.
Yeah. I mean, the general thesis is that knowledge is quite path dependent. It's not about where you end up.
It's not about these abstract functional descriptions. Sometimes it is if they're so incredibly idealized like calculus or something, they're the basis of evolution on their own.
the path of how you got there in order for you to continue the path. Yeah, I think it's really, especially in Ark Challenge, I think it's really interesting to consider and would be really nice to see what the when they were testing the game, like what when the humans failed, how did they fail? Cause at first the scoring was based on the second best human attempt for efficiency.
And that is very skewed towards like towards a good solution, right? And then it was changed to a median, I think.
Was second best per level as well, so it was almost impossible even for the best human to solve a game.
Yeah. Yeah. Yeah.
So it was so it was based based on every level, and it was really difficult for a single even a single human would, like educated human would not score 100 on this benchmark, right? So, we don't know much what the failures are, so we have played these games, we have tested them ourselves, let's say, and sometimes we don't get them. So it's not like I struggled with some of them, eventually I solved them, but sometimes I would spend like on some of them I would spend an hour to solve them.
So you cannot really say this is something that's immediate. It's also difficult for humans.
actually humans fail on these tasks rather than, oh, human can solve it if you give it 10 attempts or something. Really interesting point, yeah, because so this is the human difficulty calibration. And on ARC V2, for example, you know, the test was let in if any two humans could pass the task.
And this is a bit of an ideological thing because I think Charley believes in this notion of general intelligence. So in his mind, someone with general intelligence, because they have the core knowledge, they could, inside, let's say, a certain number of steps in the combinational closure of these core knowledge primitives, any human with a certain amount of general intelligence could solve any of these The counter to that is that, as we've just been describing, maybe human intelligence is quite high level and fractionated and pattern based and it's quite specialized.
those fractured representations together to come to an answer. But maybe they're just in possession of those representations and another human isn't. I have a friend that is a professional esports player.
I showed him one of the games, and he completed the first level without spending an extra move that he didn't have to. It was immediate to him what the goal was, he recognized the pattern, and I showed him the game within three seconds he finished the first level with sub superhuman performance, let's say. And okay, he has been playing professionally games for five years.
Right? But and he's been trained on this. But like exactly as you say, like, there's something we can bootstrap on and we do bootstrap on, and it does work.
is enough. Yeah. I think gaming is a really good example.
Mean, good friend of mine is called DDK, and he does commentary on Counter Strike and stuff like that. And it's when I when I watch him do commentary, it seems super situational. So like in in Quake three, you know, there's you gotta time the the mega health, and then there's the red armor, and then there's a there's a position over here.
And what you get is, like, the emergence of these complex situational phenomena in the game. And this is this doesn't seem anything like what Charley is talking about, you know, because there's two worlds. Right?
There's the sort of the the emergent complexity world, and then there's the reductionist core knowledge world. I think in this world, we can still use intelligence and we can still acquire abstractions and descriptions for these high level phenomena. What do we do?
Do we analogize them in terms of low level knowledge we already have? Or are they something new?
It seems like a different modality. No, I don't really have a good answer to that. I just only I think I can base my answer on these examples.
And I completely agree with you that there are some abstractions like that. I don't know. I don't know.
it is still path dependent, right? You can still trace a path from the low level description to the high level phenomena. But the reason it's unintelligible and it's surprising is because the path is a long one and it's computationally irreducible.
So you have to perform every single intermediate computation to get there. We just it's confusing for us because we can't actually analyze or understand the path between the causal stuff that made this high level phenomena. So we just kind of see it on its own, like disconnected.
And many phenomena in the world are like that. Yes, also sort of like emergence you're talking about, like where you have to call it emergent because you just can't trace, you're not capable of like the computational graph of that is just not tractable for you to That's true, but another feature of emergence is compression.
So like with the glider for example, we almost don't need to know how it came about because the new description encompasses it's a complete description of the new phenomena. It's almost like it's the start of a new piece of knowledge. And a lot of our knowledge is like that.
It's such a good compression of what went underneath that we can start from it. Like for example, we did you, calculus, for example. So calculus is within the closure of calculus or probability theory, we can do a lot of things.
We don't need to know how it came about.
So, but then is that a new layer of knowledge and we don't care how it came about? It's just kind of like saying, do we need to start from the bottom? Or in many cases, this has come about and we can just use it you know, higher up the abstraction mountain.
Yeah, think ARC is an example for the case that you can do this, at least to some level, because we see the Frontier LLM scoring quite well on this end of day wouldn't be able to if these core priors would be breaking it. But I think what is really nice about ARK is that there are some games that are so different and are still easy for humans, but they somehow like break this concept, right? You talk about disclosure of like and I gave the example of Maze But like you can maybe change the core prior concepts, and they would still make sense for a human, but they would somehow break adversarially this this like, let's say, calculus like maze like representation that helps solve the game, you can like move them around a little bit, then you suddenly break it and you still have a valid game for a human, but not for the abstracted LLM intelligence.
And that's what we see also with some of the difficult games. I think it's really difficult to not move the priors too much so that you can still make it human solvable.
fails at this. I want to talk a little bit about RKGI three. So the first two versions of the ARK challenge, they were quite abstract because Charle was trying to idealize intelligence on its own in the most abstract kind of legible way.
RKGI three, it introduces, in my opinion, and we can talk about this, the concept of agency. So agency in my definition is the ability of an agent to have goals, to plan and realize those goals. The more ambitious the goals are, and the more you realize the goals, the more agency you have.
So there's a kind of low level, no nonsense definition of agency, which is just a thing that can sense and act. So that's like if I'm doing computer programming, that's what an agent is. But the more kind of cognitive science definition of an agent is one that sort of has future point in control.
I have these big goals in the future and I can realize them and that makes me an agent. So I think that RKGI three introduces agency, not just in terms of realizing goals, but acquiring goals.
Yes. Agency and also like acquiring these goals over time through interaction and adapting them. Because not only do the levels on their own to be solved require interaction, but all the levels change.
So these goals dynamically change and you so that you why why you cannot learn them learn them directly. And then on the on the concept of agency, so I would I would argue that one of the interesting things that we saw is that what I what I thought coming into this challenge and started working on this problem is that it would be difficult for the agent to have any sort of idea what it should be doing at all when it when it solves the when it tries to solve the first level. But I think by now we got at least one solve on all the public games, at least the first level, made it be by trial and error, or maybe made it be by luck.
But still there is some somehow LLMs seem to be able to figure out what they need to do. That might be through the different biases that are already encoded for humans that have been building these games, but somehow it is possible to generate these hypotheses. They may not stand for long.
They might be kind of random. They but but there are signs of life that LLMs can already do this, I would argue. Certainly, like, there's that anthropic biology paper, and and and they kind of talk about all the capabilities of language models.
But one thing that they said language models can do is plan.
And planning is required for agency, in my opinion. And we know that language models can't plan because planning is a type of computer science algorithm that is not possible in the substrate of transformers. So what they do is a kind of simulated planning, right?
So that they can simulate planning up to a certain depth.
Do you think that matters? Maybe one important thing here is to realize that there's two types of planning that are involved in playing an arcade. So there's, need to, at some point, when you understand the rules and you understand what you need to do to win, you need to plan a path to get there.
And that's kind of more of a traditional game theory planning algorithm. So you might use a search or Monte Carlo algorithm or one of many such things, which the model is perfectly capable of because it can write Python code and run the Python code. That's one level of planning.
But the other level of planning, when you enter just a level and you see it for the first time, is you need to figure out how am I going to figure out the rules here. Oh, that looks like it might be a player. Let me try to move in the direction.
Ah, okay. Pushing up moves it up. You need to balance that exploring and exploiting, for example.
When do I start trying to win? When do I start trying to Yeah, how long do I start playing around, basically? That's all different level of planning, which you can't encapsulate in Python code.
And let's say, see our agents actually, I think it's fair to say, managing that quite effectively. So in a sense, yes, maybe transformers can't plan, but they can do a very good job of pretending essentially.
what might be formal planning in a computer science sense. And I hadn't quite incorporated that. You're absolutely right.
So even if they intrinsically, they can't explicitly plan, they can call tools, they can write code that can plan.
or whether it's enough that they can statistically generate programs that do enough planning for the circumstances they're used in. Maybe that's too Galaxy Brain for this, but they can basically plan in essence. That's what you're saying, right?
Yeah. And so what it comes down to, again, whatever is, let's say, under the, let's say Galaxy Brain understanding needed to understand what's going on, they seem to be able to do it to a degree that lets them play these games essentially, whatever it is they're doing and whatever it really means.
Okay, in which case, where's the gap then? So if they can plan, is it a representation problem?
the big gap? So maybe it's first good to discuss what the actual gap is, because indeed, if we see, for example, what Arc is publicly communicating, which is Schlage, for example, they show simply the benchmark where even frontier models are not able to play the games to any significant degree. So they score under 1%, which either means they're not beating any levels or they're being very inefficient about it, doing way too many actions.
However, if you look at publications, you can see that score can go up to, is it best now 35%, 36%? Yeah. If you give them kind of a proper harness.
So the point is they need some guidance. You need to tell them more like, okay, you're in a game, you're going to do this, you can write codes. But with that guidance, they can actually do pretty well.
That guidance, it's not game specific. Not like, oh, maybe this is a maze and you need to think about this. No, it the model still needs to do that explores exploit balance.
The model still needs to do all the move planning, but it gets some kind of guidance on the general thinking patterns that might get it there.
we can with a little bit of work, make them do anything. It's a little bit like, for example, if I already know how to solve a problem, I can use GPT 3.5 and it will give me the right answer because I'm giving it guidance.
So it's a little bit like a heuristic search problem where if you guide the search enough, if you place enough constraints in there, then statistically, like if you sample enough times, it's gonna get the right answer. Yeah. So I think that very much is Chalet's look at this.
scores are on the 1% because they don't allow harnesses for that leaderboard. Because his point is, yeah, you shouldn't need a harness. A human doesn't need a harness.
It's a tricky one. I, for myself, don't know if I really fully buy that argument in the sense that the reason a human can do this without needing a specific harness, I mean, that's fine, but most people, especially people who are capable of playing these games, have played a lot of games, be them computer games, video games, or games of real life. So they've already learned the thinking patterns that we are now encoding through a harness much more directly.
I could also imagine if that hypothesis is true, that it works that way, that if you were to take those 1% scoring things, but you do kind of a reinforcement learning loop specifically to teach them gameplay, but without hard coding any harness, they might actually do a lot better. And does that count? Does that mean they can then play it?
I don't know, but Is that the same as arguing that if you went to a deserted island where there was some tribes folk that have never used computers before, do you think they would be able to do ArcGI three? I think not immediately.
But like if you give them 100 of these things and you give them one per day and you reward the people doing well, I would assume at some point they can. And I do think it's fair to say they probably need a lot less training resources in this amount of, I don't know, the amount of compute required to learn it than right now a frontier model would use. And even when we say they can reach 36%, that is true, but it costs like a few thousand dollars, which is a lot more.
Although it's hard to, of course, want a lot more compute than the human beating these games is spending. So yeah, does that really count as the old models can play games now if it takes them so much effort? I would say that is a real gap.
And also, I'd say being able to do it for frontier models, sorry, with small open source models, would demonstrate a lot more. It's indeed what we're doing in the GAPGO competition.
might be misleading as a number if you don't look behind it. So what it really measures is action efficiency. So correct me if I'm wrong, but it's the ratio of the human baseline divided by the number of actions the AI has taken, whatever model it is, on the level or a human, whatever whoever the player is, and then squared as well.
So this plays really adversarially to anything that's a little bit action inefficient. So 36% actually in this case doesn't mean we solve for that approach solves 36% of the games, it solves way more of the games out from the training set, at least this number is on. But it just solves them inefficiently.
So I think this is really important to emphasize that, like, the current frontier models are able to solve, like, what is it, something like half of two thirds of the training games actually till the end, but just not as efficiently. What is the hardest thing in Arc AGI three? Is it the goal acquisition or is it just simply the action efficiency?
From what we see on the training set, so the testing set, the private set is set to be harder. We don't know anything about it. Nobody outside of the organization has their organization has seen it.
We we don't know how how hard it is. But as at least on the training games, we see that the that the LLMs can acquire the correct goals and pursue them somewhat effectively. The question is whether this holds also for the withheld private test, but it seems that goal setting is not the bottlenecks, rather the action efficiency and accumulation of the knowledge over a very long context, because you need millions, hundreds of thousands, if not millions of tokens to solve this.
And keeping consistent knowledge of everything that has happened over such a context is a is a major engineering challenge at this moment.
harder in Arc v three compared to the other one is this interplay between exploration and solving the game. Because in arc v one and arc v two, you would get all of the information in a static frame as you are given the puzzle. Instead in arc v three, you are given the game, but just from the first frame, you cannot understand what needs to be done.
So you start need to start interacting with the game. And through that interaction, you gather information of what the game is about and you start understanding how to solve it. And at the same time so you need to try to understand what the game is about and try to solve it at the same time.
And this interplay is very hard to kind of explain to the agent how they should do it in an effective way such that it's general and generalized to across all games. So I think this is probably one of the key part that is hard about r p three.
based exploration.
Yes. And and that's, I think, the very hard part. And I guess here is where the similarity with RP two and one are that you kind of get the right obstruction level at which to do this exploration and kind of understand that, oh, the thing moving around is probably an agent and that I can kind of control, and it's not just kind of patches of pixel moving around.
And if you're able then to get this right obstruction, then it's usually kind of trivial to to solve the game. But, yeah, making the agent acquire this this right obstruction level, yeah, it's it's the main challenge. How difficult is it to acquire the goal?
parameter models can just reasonably infer what the goal is, or or is it much more complicated than that?
at the first trial, it's it's very easy. They kind of can if they find the right hypothesis for the game, then they would go on and solve a lot of levels. But if they maybe try once, make it wrong, and then have an give another hypothesis and don't and and it's not the right hypothesis again, then it's very hard to get them out of of the loop.
And the we often find that they get stuck in very kind of not intelligent goals that it's very clear they're not right. For instance, often the agents start thinking that reducing the energy bar to the minimum is the goal or that stepping 10 times in a region is the goal, which for a human is kind of clear that that it's not the actual goal. But it's very interesting that they're not able to to see that it's it there there's no way that's the actual goal.
Yeah. Yes. And can we contrast rewards and goals?
We could use various different things. Like, the most naive implementation is just use level transitions and then the Arc AGI score for that specific level transition. So if you did office, yeah, if you did worse, took more actions than what a human would have taken, your score would be lower, but you use that per level.
Yeah. And then you can add various other types of rewards to improve your objective. And did you use reward shaping?
Yes. So for our oral training pipeline, we do reward shaping. We have 25 games.
We actually have a lot more games that we've generated ourselves. And we can, for example, train on them to make the agent better, for example, exploration, finding goals, achieving those goals, yeah, through end to end RL on just a bunch of games basically. And the thing is, this is much more difficult to do than an ArcGI ArcGI three, this was the standard approach.
You would pre train on a lot of puzzles and then you would do test time training as well on the set of puzzles. But to do this in ArcGI three, you need to train over, as Michal mentioned, like 100,200 tokens, which is extremely difficult to do. So we try and improve this by, for example, adding reward shaping, training on shorter sequences, trying to generalize to longer sequences.
And I guess what makes this so interesting for Tufa Labs as a problem is that this is exactly the kind of research that is currently the state of the art, the genetic multi turn, long context reasoning. And yeah, it simplifies the problem. We don't have to worry about environments or other complexities like safety.
We can just focus on the core problems and try and see if we can make progress there. And that's also perhaps some area we could actually contribute some new research, which might, yeah, I guess the open source at the end of the competition, or we'll see how we wanna do with that research. But yeah, basically on your reward shaping question, we had various rewards, level rewards, ArcGIS scores, whether the code gets executed or not, the length of the reasoning steps, because if it's reasons too long per step, it might take too long to actually solve it and you only have a certain amount of tokens per game.
So all of that we try and improve, yeah. You know, like the weird side effect of Archie one and two is it left me with the impression that intelligence is overrated.
And what I meant by that was, you know, yes, we could could solve problems. We could do hill climbing, you know, when we knew what the problem was. And maybe you would agree that in ARC AGI three, when we know what the goal is, it becomes a hill climbing problem.
But it feels like the challenge is abstraction. And it's almost unfair that Charle is doing this action efficiency thing. I mean, in my opinion, what he should do is, yeah, you do it the dumb way the first time, and then you compress that into reusable knowledge and it's in your library.
And then the next problem that comes up, which should ostensibly be using what you just learned in the first game, then you become more efficient. So you kind of become more efficient over time because you're compressing knowledge.
That kind of feels like the goal to me. Yes. So there's actually been discussion in the community about, let's say for Arcadia four, having that as one of the goals.
You play a game, you solve it in like a large compute or action budget, and then you have to play it again and you have to do like a speed run through it to improve it. So I think that is important, but I also think action efficiency is a practical step to reduce just brute force solutions. It counteracts that, you have to explore in a more intelligent way.
And it kind of makes sense in real world environments as well, like computer use, coding agents. You can't just explore every possibility. You have to do it in an intelligent way and improve using some heuristic.
And I think it makes sense to not always have just all the actions available to you. Okay.
three though, you do have the goal and it's a hill climbing problem, guess you still can't brute force it because you get dinged on efficiency. But in principle, you could, if you had enough computation.
You could, but even that they made, so they updated games to be more robust to that. So like level one, you could in principle brute force in a certain amount of actions. Level two, the computational power you normally need is enormous.
So you can't really, for a reasonable compute budget, even get past level two. And then you have level three, four, depending how long the game goes. So yeah, in principle, you could brute force anything, but practically with the compute budget we have, you won't be able to do that.
And could you just describe what is it about RKGI three that makes it almost impossible to brute force? The main thing is the action space. So we have eight main actions, but there's a mouse click action which has around 4,000 possible places you can click, 64 by 64.
And also the amount of actions you normally take is quite large. It might be anything from few 100 to a thousand actions on all the games. So that branching factor is just enormous if you're not doing that in an intelligent way.
Is anything in Arcadia i3 badly designed or gameable? Because didn't the preview have some It was a bit brute forceable, wasn't it? You were talking about that earlier.
Yes, yes, exactly. So guess that was the main point about the preview competition, show whether it's something you could easily exploit. And indeed there was like the Stochastic Goose algorithm and few other algorithms actually.
So I think the first four places was basically brute force algorithms that just search over large ways of actions, but do some basic form of filtering and you can get a very good score. Like I could solve two games, almost solve the third game on the private set. The game is also too easy, so they upped the difficulty level.
And for example, one other thing was the timing bar, which only changed when you actually executed a valid action. So you could easily learn what actions are valid or not. On the new set of games and Arcadia setup, I don't know if any of you have noticed anything that is easily exportable or badly designed.
Yeah, I think it's implemented much better now and that's why the scores haven't shot up initially. It's still at like 1%.
a challenging setup. I guess the million dollar question though is, do you think it's possible in principle to do really well on ArcGI three and be no closer to AGI?
Yes, I do think it is possible, but I guess that's the point of the benchmark as well, is that it doesn't have to prove that once you solve RSK AGI, you actually have AGI. But I think as the R prizes mentioned over and over again, it actually just disproves that something that can't solve this benchmark is not AGI. So yeah, I think it's definitely possible, but getting a 100% on this benchmark would definitely require some major improvement, just like a base model or something.
Like we're nowhere close to getting a 100%, which is the grand prize that is also made available at the end of this competition.
any time in the near future?
So I guess that's a debate. Some of us believe that might be possible.
yeah, I don't know if there's any other opinions on whether we might. So there's, I mean, can split the problem at least into like, will someone reach it in the Gaggle competition, where you're playing 110 games in nine hours on a single LTX 6,000 versus if someone's going to do it with a Frontier model. 100% is essentially impossible because it means you cannot kind of mess up once if you kind of have the wrong hypothesis on one level.
That's it, you're done. So I'd more like say, But no single human is likely to score 100%. So I'd more say like maybe 90%.
You could kind of say, Yeah, no, this model is capable of doing this. I kind of feel that is actually possible, even within the constraints of the Kaggle competition.
than will be achieved by November when the competition ends. Do you think there's some notion of almost Kumagura of optimal score?
you would kind of converge on this score. Does that even make sense? Probably yes.
And I do think that score would probably be a 100%.
score, if it's not 100%, it essentially means there is guesswork involved. Didn't you say before though, you could only get 100% if you already know? So if you were an agent that genuinely needed to acquire knowledge to solve the task, So you had to do the minimum number of steps to acquire the knowledge.
Do you think that would converge on a certain score?
simply because it is not reasonable to we just saw the RS20 game. There's two places you can move to. Yeah, you just can't guess that right every time.
The thing is, you're being compared to humans who also had to do the same exploration and you get a median score. So kind of with If you're averagely lucky, you're going to do it. But yeah, have 110 games.
You're going to get unlucky on one of them. Yeah. And not yet I will mention, even with our simple With Quen, we can already get sometimes 100% on certain games.
to beat those human baselines. What we were saying earlier is that one view is that intelligence is this quite crystallized process. There's a concept of IQ, for example.
And if you have a certain amount of IQ, then it's predictable how well you can generalize how efficient you are. And guess the data might just not back that up. The data might just say, actually it's kind of not random, but very specialized.
And there's huge individual differences in capability and machines and in humans. And would that make them kind of reassess their whole idea of what intelligence is?
I think we have a clear benchmark, which we know humans, which is general in some sub domain, which we can say whatever that domain is, can score that score and that is the medium score. So I think even with just in ArcGIS three, if we do bad with our, let's say the final submission score is like 5%, don't I think that would update their views because humans can achieve that.
it did converge and become more regular. So if there's a new class of algorithms that kind of consistently solve the problems in some predictable amount of time, maybe Charlei would think, Oh, that's the algorithm of intelligence. It's not just guessing anymore or something.
is going to move further. I think they will update their model. I think in some way they already have by changing the games because they thought some of the games that they made weren't brute forceable and they actually were.
And then there were some simple models, and they also changed the metrics, updated it a couple of times, both to be more fair and to represent better human performance. So certainly they have been updating their model a little bit already. It's tough to say what's going to happen, but I think the goalpost is like moving all the time.
And if we create an agent that would be able to solve all of this, then we're going to still discover new areas where it's still missing and it is not tied by the dataset. Maybe it's rumors, but I think there's like other ARC AGI competitions coming up. They're building ARG AGI four or something for next year, I think.
certainly they have more to add to the story than just ARG three. Do you think that they won't release four until people are doing well on three or they will just like release it anyway?
I don't know if it's intentional or just the speed at which they develop them, but it does seem pretty clear. Unless I don't think they knew what Arc AGI three would look like when they built Arc AGI one. They might have even thought, oh, it's going to be ten years before anyone beats ArcAgi one.
Think you could So was like five years in the end.
Yeah, sure. But the point is, at that point, you think, okay, now they can do this. Does this really mean intelligence?
Well, probably not. But I think also you need to see what happens with Arc AGI three and also the ways in which it gets solved to then also understand where the gaps still are. Because again, no one is claiming if a game can play, if you can do a six forty six for a game, you're intelligent in every sense.
It is just one more step and one more challenge that they've beaten. You can imagine maybe, for example, Arc four has kind of games, like you're playing Doom, for example, but not Doom, but of course, a new version of Doom. That is a whole, you need to kind of work with a much more vaguer representation of the world and it would be a completely new challenge again.
But I imagine, yeah, it's going to kind of taking a little time seeing what's happening here. I've also no idea what they're doing in the background right now.
I've no idea actually.
Yeah, Sholly was quite forthcoming about RKGI three. I'm wondering if it's not the idea you discussed because the community has discussed this a lot.
and like speed running it. So that's one intuition that's been going around in the field, but I guess only Greg and the team can tell us next year. The thing I really liked about ARC V1 and two was that, first of all, it was a grassroots community and loads of people without lots of funding could just focus on the kernel of intelligence.
And now it's just ridiculously complicated and it's almost de intellectualized the field, which means that loads of folks with resources are just throwing reinforcement learning algorithms at it and almost going back into kind of blindfold mode, which is, I think it's a fair description of a lot of machine learning. We just have really big models and we just train it to do next token prediction and we see what happens and we try and scale up. There there the thing that was so magical about the community was there were just so many great ideas and, like, you know, just really, really deep thinking.
It's not that simple.
it's it's still hard to make a harness that solve these games, and it's still and only starting like, and training is also hard. Like, it's not as simple as just throwing computer. There are still many details that you have to get right.
But for sure, as you said, I think it brought up a lot level needed to enter the competition. Like, we're seeing that many people are stuck in the kind of template solution, and there are, as far as we are aware, at least in the competition, not many trying kind of LLM approaches because it's so computationally expensive. So it's definitely made it harder for kind of the average person to enter the competition, but I wouldn't say that it's just a matter of throwing a lot of compute at it and brute forcing it.
Yeah. It it's not like most other benchmarks. Right?
It it does there's no language. There's no instructions. Does that make it quite distinct as well?
I think I think that's the interesting part about ARC that it kind of tries to remove as much as possible the prior that you get from language or from human knowledge and kind of strip them to the minimum and really only test for intelligence. I think, yeah, that's a cool feature.
Language is the basis of how we do a lot of thinking. And what you folks have done is you've gone to language and then gone back again. So we've got this kind of loop.
So we go to language, we do some reasoning, and then we might use that to do active fine tuning or do reinforcement learning or whatever. And we've got this virtuous cycle. So he took the language away and we're kind of putting it back again.
Yeah, I think the reason we're putting it back is because we're specifically focusing on language models, which has been extensively trained on language.
neural guided search, they might not use language at all. You don't need language for this, at least a human textual language, but it's just, in our case, we're heavily leveraging reasoning models. So it's just natural to bring them back into the domain they're experts in and what they're doing while we wanna actually just, our harness should shape that and encourage that behavior.
So yeah, I guess that's the main reason we're moving back.
to like, I think the the the reason, and I've seen I said it before already, is that the reason why language succeeds so well, because there's like higher higher level priors that are encoded that are just so difficult to strip away from the data, may that be what the humans have created these games and they have an idea of what a game is, and even though they try to remove the biases a little bit, they're still there, you might look at these games, and usually the backgrounds are some sort of like dark color or white rather, rather than green. And this is like not necessarily true for all of them, but like more or less it holds. And you can see that this was created by humans and what their idea is of games.
And that is just why language holds. I think it's incredibly difficult for a human to create something that doesn't have this in it. It might be even impossible.
So that's maybe why coming back to the language works so well. Yeah.
like I've also seen exactly that. So one thing you could do, you could easily permute the game. So you can permute the colors and also rotate the images.
And when you do that, the games become significantly harder. So if you just remove that priors, which shouldn't actually be priors, you could actually see it becomes harder for humans So to yes, perhaps that tells something about the benchmark. There's some leakage of human priors into the game.
Harder for the algorithms as well? Yes, yes. So that's one thing we found of the harness.
Not So you can provide numbers as colors.
which is interesting. That's fascinating. Yeah.
On why I think it's important to pass through language? Because, like, initially, we try to just train a neural net with RL on a lot of games and kind of not use LLM altogether and try to, yeah, get those get those prior in by just seeing a lot of games. And it's way, way easier to just use an LLM.
Like, you you get a way like, you get a good performance much faster because the prior of language is so general. It can be applied to so many different domains. And getting the same prior by just training takes it would takes, like, so many games.
Instead, by starting from language and then fine tuning to the specific games, you can get there much faster.
You know, it's quite a common technique to transform something so that it falls into a representation that has more friction with our knowledge.
like we use chars, like let's say, use a B for blue. And we can tell it like, this is blue, this is this color. This automatically helps it a lot in language to reason about it because it knows that typically when humans play games, bright colors or objects you wanna interact with or something's gonna happen, more dull colors, background or walls or something like that.
a lot. Because that's actually really interesting because I suppose like one strategy is you could have multiple representations or from a language point of view as well, you could do something really galaxy brain, like turn it into a story where the colors become dogs or agents or people. Do you sort of mean, like, is there some representation that will just make the language model come to life and it's significantly more powerful?
As far as we've seen so far, the models are actually pretty decent at perception.
They, because remember, the game in the end is not a 64 by 64 grid. It it has concepts. It has a player that's moving around.
It has a goal. It has the maze. And you're not going to play the game if you don't get those concepts.
The thing is, the LLM is pretty good at getting at those concepts. The problem is that it takes a while and we have very little time to play the games. So one aspect in the doc that we haven't discussed yet is, for example, that we give it different views on the world.
So I already mentioned Elske Grieg, but for example, we also have a simply It's very simple. It's just kind of a segmentation based on connected components. By just having that and giving it that list, it works well on some games, not well on others, but it kind of allows the model to short circuit a lot of this perception part.
So the perception, it will kind of, yeah, look at Oski Gehrer and was a bit on part of the board and make an Oski Gehrer there, oh, that looks like this, and do it on another one and spend a lot of tokens that way. What about from a vision point of view as well? Do you think language is actually necessary for reasoning?
Do you think in principle we could train a vision model to go directly to the output from the input? So, Anthony, I think it boils down to one of the big open questions in the field, like, is language critical to intelligence? I will say, let's say, at least when I'm playing the games myself, and I think that goes for all of us, I'm using language.
These games are set up that indeed Mr. Object Prior, like about a four year old could do it, but I don't think a four year old would be beating most games. And if I'm actually thinking through a strategy myself, I am thinking, let's say, yeah, out loud all the time in my head, but there's a narrative and language basically.
Oh, okay, we have to get the key, but the key is blocked. How does that work? Then yeah, how you decouple all that, I'm not sure.
Language is almost like an emergent phenomenon. So it's still almost determined by its physical instantiation and the world that we live in. But it's something which has evolved over a very, very long time.
So we could, in principle, train an end to end vision model. And if it saw an insane amount of data, like almost all of the distribution, it could potentially acquire the same abstractions. But language is almost like a shortcut because you can bootstrap those abstractions.
Yeah, it's also tricky because it could, but it might end up kind of internally developing its own language because you need stuff like reasoning change. Mean, our case, it needs to write code, which is a language itself.
That is a good point. So I did some initial oral experimentation and what I did for first experiment to show that we can generalize, is I created a arc environment that's procedurally generated with some new objects and new objectives. And in fact, get the behavior, it's just like a vision model and like a Mamba backbone.
And you can get that behavior that it learns to figure out the objects and place them in the right order just without using language, just using a latent representation. So I think it is possible in principle, but what that experiment showed is I required about 10,000 different permutations just to solve that one setup. On real arcane, there's gonna be more than that if you wanna train from scratch.
So yeah, that's why the prior helps a lot. You need less environments to actually get it to perform well. Well, you could just do it with a harness.
And if you train, the hope is that you need less environments to actually just improve on and generalize, which is the most important part. Yeah.
but we have the benefit of millions of years of evolution, right? So we're kind of, we're using all of that trial and error implicitly. And so it's almost a bit unfair that we're expecting AI algorithms to be able to do all of that.
Exactly. How do you square the circle between, like you're releasing lots of prebuilt tools that are quite specialized and you also are bitter lesson pilled. Is that is that a contradiction?
Like, what what what does it mean to be bitter lesson pilled?
As a lab, we subscribe to the bitter lesson. The, I guess, the idea behind behind the bitter lesson is that often detailed and specific solution lose compared to just more data and more compute and kind of going large scale on on a problem. I think looking at the past, it's even if the bitter lesson wins in the end, it's always useful to try first the more kind of detail specific approach.
For instance, for AlexNet, which was the paper that won kind of ImageNet and in some way started the deep learning revolution. I think a lot of the intuition that brought them to use convolution neural network for image recognition were in some way motivated to all of the tricks that were done before to do image recognition. And so I think that even if it's true that now we're doing pure engineering on the harness, we're kind of understanding the details of the problem that then will will allow us to do a more end to end kind of solution and allows us to do something that generalize hopefully more.
probably a necessary step. Yeah. Want to add that I may be a little bit less bitter lesson built than than most AI researchers, also mainly be maybe because I was doing it maybe before the bitter lesson became the the the baseline.
But I do think at least, certainly, where ARC AGI three is right now, I would kind of be willing to make a bet that the winning solution, be it ours or someone else's of this competition, will actually not be a bitter lessened solution. So if you look at Arc AGI two now, it's been pretty much bitter lessened, it's being won by creating huge amounts of training data. I don't think there's a clear path just yet for this competition, and that what is needed now is kind of really the, yeah, basic design.
Yeah. And then go for it. It's fair to say that Charle does not want it to be bitter lesson build.
it wasn't the case that it was bitter lessened. I do say I might be willing to make that bet for this year. If there's another iteration next year or one other one the year after that, I could imagine that at some point it is bitter lessened.
But I don't think the models are there to be able to do that this year already, but.
we have so much uncertainty about this. We still don't know beyond a reasonable doubt that it might not be bitter lessened pilled in two years' time. Important to bring things into distribution.
And what ARTPRIZE has done for ARKGI Free is that they specifically say that the training problems or the public problems are not representative of the testing problems. So that makes it even harder to create a distribution on which you could meaningfully train, because you still have to extrapolate in some way. So maybe that's also why an overfitting just doesn't help because you have to generalize to new problems.
then this could be done, but this is just hypothesis. I I do think maybe in a sense it is going to be bitter lessened, but the classical bitter lessening is just you make a training set so big that everything is in there, and then just bake everything into your weights, then you're there. This is kind of what happens for Arc AGI two.
I think this one does need something like that, but it needs to have, and I think that is maybe much more what Chalet would also like to see, what you need to get into the model is the basic thinking patterns. So it needs to know the general ways in which to approach a game and how do you solve a game without having memorized every game in particular. And I think if we can manage to do that, yeah, we'll feel like we've really done something past just beat a bunch of games.
the fact that you're trying to build more powerful AI, but you're worried
about it? So yeah, reconciliation is, I think you cannot do AI safety research in a vacuum. You need to be clued in about the best techniques and being able to kind of train good AI systems to be able to study them at all.
The way I would say it is if you want to make bridges safe, it's going to help to know civil engineering and how to build bridges. And so I think similarly for AI, but that runs the risk into, then you participate in the race and you make that kind of racing situation worse because this more active progress about AI capabilities runs ahead of progress towards AI safety. So I want to be very careful with this.
All right. I think now I'm very happy with the team we currently have and the kind of level of technical discussion and common interests and actually work being done. So that's all very positive.
And gone wrong, it just takes forever to get there somehow. Like it takes longer and more expensive than what you think it will. And then I guess what's wrong is broad AI progress is very fast and it feels a bit tough that can you ever catch up?
I think the next year, they might not look that different than today, but I would say five, ten years away, it may look completely different. Yeah. So I think, yeah, maybe even the software and the singularity, I think software advance will keep being made about kind of agentic engineering, stronger, like stronger AI system in the world of atoms generally.
But I think eventually we replicate this success with things like industrial robots, like self driving cars. I think they at some point, the more of a kind of falls. And then we start seeing powerful AI capabilities everywhere.
Summer is here, and Kroger is your destination for hot savings. Find unique items at low prices with our exclusive brands. Fire up the grill with cookout classics like burgers, and don't forget delicious produce like fresh melons.
Or treat yourself to summer's sweetest lineup with Kroger brand's limited edition all American ice cream collection. Whatever your summer plans, Kroger makes it easy to enjoy high quality food at affordable prices. Kroger, fresh for everyone.
Starting a business can seem like a daunting task unless you have a partner like Shopify. They have the tools you need to start and grow your business. From designing a website, to marketing, to selling, and beyond, Shopify can help with everything you need.
There's a reason millions of companies like Mattel, Heinz, and Allbirds continue to trust and use them. With Shopify on your side, turn your big business idea into Sign up for your $1 per month trial at shopify.com/ special offer.
Shared via Hopper