Why I don’t think AGI is right around the corner

Okay. This is a narration of a blog post I wrote on 06/03/2025 titled, why I don't think AGI is right around the corner. Quote, things take longer to happen than you think they will, and then they happen faster than you thought they could.

Rudiger Dornbusch. I've had a lot of discussions on my podcast where we haggle out our timelines to AGI. Some guests think it's twenty years away, others two years.

Here's where my thoughts lie as of June 2025. Continual Learning Sometimes people say that even if all AI progress totally stopped, the systems of today would still be far more economically transformative than the Internet. I disagree.

I think that the LLM's of today are magical, But the reason that the Fortune 500 aren't using them to transform their workflows isn't because the management is too stodgy. Rather, I think it's genuinely hard to get normal human like labor out of LLMs. And this has to do with some fundamental capabilities that these models lack.

I like to think that I'm AI forward here at Thorcatch podcast. And I probably spent on the order of a hundred hours trying to build these little LLM tools for my post production setup. The experience of trying to get these LLM's to be useful has extended my timelines.

I'll try to get them to rewrite auto generated transcripts for readability the way a human would, or I'll get them to identify clips from the transcript to tweet out. Sometimes I'll get them to co write an essay with me passage by passage. Now these are simple, self contained, short horizon, language in, language out tasks, the kinds of assignments that should be dead center in the LLM's repertoire.

And these models are five out of 10 at these tasks. Don't get me wrong, that is impressive. But the fundamental problem is that LLMs don't get better over time the way a human would.

This lack of continual learning is a huge, huge problem. The LLM baseline at many tasks might be higher than the average humans. But there's no way to give a model high level feedback.

You're stuck with the abilities you get out of the box. You can keep messing around with the system prompt. But in practice, this just does not produce anywhere close to the kind of learning and improvement that human employees actually experience on the job.

The reason that humans are so valuable and useful is not mainly their raw intelligence. It's their ability to build up context, interrogate their own failures, and pick up small improvements and efficiencies as they practice a task. How do you teach a kid to play a saxophone?

Well, you have her try to blow into one and listen to how it sounds and then adjust. Now imagine if teaching saxophone worked this way instead. A student takes one attempt, and the moment they make a mistake, you send them away and you write detailed instructions about what went wrong.

Now the next student reads your notes and tries to play Charlie Parker cold. When they fail, you refine your instructions for the next student. This just wouldn't work.

No matter how well honed your prompt is, no kid is just gonna learn how to play saxophone from reading your instructions. But this is the only modality that we as users have to teach LLMs anything. Yes.

There's RL fine tuning, but it's just not a deliberate adaptive process the way human learning is. My editors have gotten extremely good, and they wouldn't have gotten that way if we had to build bespoke RL environments for every different subtask involved in their work. They've just noticed a lot of small things themselves and thought hard about what resonates with the audience, what kind of content excites me, and how they can improve their day to day workflows.

Now it's possible to imagine some ways in which a smarter model could build a dedicated RL loop for itself, which just feels super organic from the outside. I get some high level feedback and the model comes up with a bunch of verifiable practice problems to RL on, maybe even a whole environment in which to rehearse the skills that thinks it's lacking. But this just sounds really hard.

And I don't know how well these techniques will generalize to different kinds of tasks and feedback. Eventually, the models will be able to learn on the job in the subtle organic way that humans can. However, it's just hard for me to see how that could happen within the next few years, given that there's no obvious way to slot in online continuous learning into the kinds of models these LLMs are.

Now LLMs actually do get kinda smart in the middle of a session. For example, sometimes I'll co write an essay with an LLM. I'll give it an outline, and I'll ask it to draft an essay passage by passage.

All that suggestions up to paragraph four will be bad. And so I'll just rewrite the whole paragraph from scratch and tell it, hey, your shit sucked. This is what I wrote instead.

And at that point, you can actually start giving good suggestions for the next paragraph. But this whole subtle understanding of my preferences and style is lost by the end of the session. Maybe the easy solution to this looks like a long rolling context window, like Claude Code has, which compacts the session memory into a summary every thirty minutes.

I just think that titrating all this rich, tacit experience into a text summary will be brittle in domains outside of software engineering, which is very text based. Again, think about the example of trying to teach somebody how to play the saxophone using a long text summary of your learnings. Even Cloud Code will often reverse a hard earned optimization that we engineer together before I hit slash compact because the explanation for why it was made didn't make it into the summary.

This is why I disagree with something that Sholto and Trenton said on my podcast, and this quote is from Trenton.

such that, to Shalto's point, we will we should expect to see them automated within the next five years. If AI progress totally stalls today, I think less than 25% of white collar employment goes away. Sure.

Many tasks will get automated. Claude for Opus can technically rewrite auto generated transcripts for me. But since it's not possible for me to have it improve over time and learn my preferences, I still hire a human for this.

Even if we get more data, without progress and continual learning, I think that we will be in a substantially similar position with all of white collar work. Yes, technically AIs might be able to perform a lot of subtasks somewhat satisfactorily, but their inability to build up context will make it impossible to have them operate as actual employees at your firm. Now, while this makes me bearish on transformative AI in the next few years, it makes me especially bullish on AI over the next few decades.

When we do solve continuous learning, we'll see a huge discontinuity in the value of these models. Even if there isn't a software only singularity with models rapidly building smarter and smarter successor systems, we might still see something that looks like a broadly deployed intelligence explosion. AIs will be getting broadly deployed through the economy, doing different jobs, and learning while doing them in the way that humans can.

But unlike humans, these models can amalgamate their learnings across all their copies. So one AI is basically learning how to do every single job in the world. An AI that is capable of online learning might functionally become a superintelligence quite rapidly without any further algorithmic progress.

However, I'm not expecting to see some OpenAI livestream where they announce that continual learning has totally been solved. Because labs are incentivized to release any innovations quickly, we'll see a somewhat broken early version of continual learning or test time training, whatever you want to call it, before we see something which truly learns like a human. I expect to get lots of heads up before we see this big bottleneck totally solved.

Computer use. When I interviewed anthropic researchers Shilto Douglas and Trenton Bricken on my podcast, they said that they expect reliable computer use agents by the end of next year. Now, we already have computer use agents right now, but they're pretty bad.

They're imagining something quite different. Their forecast is that by the end of next year, you should be able to tell an AI, go do my taxes. It goes to your email, Amazon orders, Slack messages.

And it emails back and forth to everybody you need invoices from, it compiles all your receipts, it decides which things are business expenses, asks for your approval on the edge cases, and then submits Form ten forty to the IRS. I'm skeptical. I'm not an AI researcher, so far be it to contradict them on the technical details.

But from what little I do know, here are three reasons I'd bet against this capability being unlocked within the next year. One, as horizon lengths increase, rollouts have to become longer. The AI needs to do two hours worth of agentic computer use task before we even see if it did it right.

Not to mention that computer use requires processing images and videos, which is already more compute intensive, even if you don't factor in the longer rollouts. This seems like it should slow down progress. Two, we don't have a large pre training corpus of multimodal computer use data.

I like this quote from Mechanize's post on automating software engineering. Quote, for the past decade scaling, we've been spoiled by the enormous amount of Internet data that was freely available to us. This was enough to crack natural language processing, but not for getting models to become reliable competent agents.

Imagine trying to train GPT-four on all the text to data available in 1980. The data would be nowhere near enough, even if you had the necessary compute. Again, I'm not at the lab, so maybe text only training already gives you a great prior on how different UIs work and what the relationships are between different components.

Maybe RL fine tuning is so sample efficient that you don't need that much data. But I haven't seen any public evidence which makes me think that these models have suddenly become less data hungry, especially in domains where there's substantially less practiced. Alternatively, maybe these models are such good front end coders that they can generate millions of toy UIs for themselves to practice on.

For my reaction to this, see the bullet point below. Even algorithmic innovations, which seem quite simple in retrospect, seem to have taken a long time to iron out. The RL procedure, which DeepSeq explained in their R1 paper, seems simple at a high level.

And it took two years from the launch of GPT-four to the launch of O1. Now, of course, I know that it's hilariously arrogant to say that R1 or O1 were easy. I'm sure a ton of engineering, debugging, and pruning of alternative ideas was required to revive at the solution, but that's precisely my point.

Seeing how long it took to implement the idea, hey, let's train our model to solve verifiable math and coding problems, makes me think that we're underestimating the difficulty of solving a much gnarlier problem of computer use where you're operating on a totally different modality with much less data. Reasoning. Okay.

Enough cold water. I'm not gonna be like one of these spoiled children on Hacker News who could be handed a golden egg laying goose and still spend all their time complaining about how loud its quacks are. Have you read the reasoning traces of o three or Gemini 2.

5? It's actually reasoning. It's breaking down the problem.

It's thinking about what the user wants. It's reacting to its own internal monologue and correcting itself when it notices that it's pursuing an unproductive direction. How are we just like, oh, yeah.

Of course, machines are gonna go think a bunch, come up with a bunch of ideas, and come back with a smart answer. That's what machines do. Part of the reason some people are too pessimistic is that they haven't played around with the smartest models operating in the domains that they're most competent in.

Giving Claude code a vague spec and then sitting around for ten minutes until it's zero shots of working application is a wild experience. How did it do that? You could talk about circuits and training distributions and RL and whatever, but the most proximal, concise and accurate explanation is simply that it's powered by baby artificial intelligence.

At this point, part of you has to be thinking, It's actually working. We're making machines that are intelligent. Okay, so what are my predictions?

My probability distribution is super wide. And I want to emphasize that I do believe in probability distributions, which means that work to prepare for a misaligned 2028 ASI still makes a ton of sense. I think that's a totally plausible outcome.

But here are the timelines at which I'd make a $50.50 bet. An AI that can do taxes end to end for my small business as well as a competent general manager code in a week, including chasing down all the receipts on different websites and finding all the missing pieces and emailing back and forth with anyone we need to hassle for invoices, filling out the form, and sending it to the IRS.

2028. I think we're in the GPT-two era for computer use, but we have no free training corporates and the models are optimizing for a much sparser reward over a much longer time horizon using action primitives that they're unfamiliar with. That being said, the base model is decently smart and might have a good prior over computer use tasks.

Plus, there's a lot more compute and AI researchers in the world, so it might even out. Preparing taxes for a small business fuels like for computer use, what GPT-four was for a language. And it took four years to get from GPT-two to GPT-four.

Just to clarify, I'm not saying that we won't have really cool computer use demos in 2026 and 2027. GPT-three was super cool, but not that practically useful. I'm saying that these models won't be capable of end to end handling a week long and quite involved project, which involves computer use.

Okay. And the other prediction is this. An AI that learns on the job is easily, organically, seamlessly, and quickly as a human for any white collar work.

For example, if I hire an AI video editor, after six months, it has as much actionable, deep understanding of my preferences, our channel, and what works for the audience as a human would. This, would say, 2032. Now, while I don't see an obvious way to slot in continuous online learning into current models, seven years is a really long time.

GPT-one had just come out this time seven years ago. It doesn't seem implausible to me that over the next seven years, we'll find some way for these models to learn on the job. Okay, at this point you might be reacting.

Look, you made this huge fuss about how continual learning is such a big handicap. But then your timeline is that we're seven years away from what at a minimum is a broadly deployed intelligence explosion. And yeah, you're right.

I'm forecasting a pretty wild world within a relatively short amount of time. AGI timelines are very log normal. It's either this decade or bust.

Not really bust, more like lower marginal probability per year, but that's less catchy. AI progress over the last decade has been driven by scaling training compute for frontier systems over four acts a year. This cannot continue beyond this decade, whether you look at chips, power, even the raw fraction of GDP that's used on training.

After 2030, AI progress has mostly come from algorithmic progress. But even there, the low hanging fruits will be plucked, at least under the deep learning paradigm. So the yearly probability of AGI craters after 2030.

This means that if we end up on the longer side of my fiftyfifty bets, we might well be looking at a relatively normal world up to the 2030s or even the 2040s. But in all the other worlds, even if we stay sober about the current limitations of AI, we have to expect some truly crazy outcomes. Many of you might not be aware, but I also have a blog, and I wanted to bring content from there to all of you who are mainly podcast subscribers.

If you wanna read future blog posts, you should sign up for my newsletter at thewarcache.com. Otherwise, thanks for tuning in, and I'll see you on the next episode.