“Vibe Coding is a Slot Machine” - Jeremy Howard

It it it literally disgusts me. Like, I literally think it's it's inhumane.

And the funny thing is they're both right. LLMs cosplay understanding things. Like, they pretend to understand things.

No one's actually creating 50 times more high quality software than they were before. So we've actually just done a study of this, and there's a tiny uptick, tiny uptick in what people are actually shipping. The thing about AI based coding is that it's like a slot machine in that you you have an illusion of control, you know, you can get to craft your prompt, and your list of MCPs, and your skills, and whatever, and then but in the end you pull the lever.

Right? Here's a piece of code that no one understands. Yep.

And am I going to bet my company's product on it? And I the answer is I don't know, because like I I don't I don't like I don't know what to do now, because no one's like, been in this situation. They're they're really bad at software engineering.

And then I think that's possibly always gonna be true. The idea that a human can do a lot more with a computer when the human can, like, manipulate the objects in inside that computer in real time, study them, move them around, and combine them together. Whoever you listen to, you know, whether it be Feynman or whatever, like, you always hear from the great scientists how they build deeper intuition by building mental models, which they get over time by interacting with the things that they're learning about.

A machine could kind of build an effective hierarchy of abstractions about what the world is and how it works entirely through looking at the statistical correlations of a huge corpus of text using a deep learning model. That was my premise.

I mean, welcome to my home. Thanks for coming. Yeah.

Well, where are we now? We are in beautiful Moreton Bay in Southeast Queensland.

The weather didn't disappoint. Oh, it certainly didn't. It doesn't often, but if you were here yesterday, it would have been very different.

to train them and and specialize them. But apparently, this was not received wisdom. Oh, this was the first time it happened.

Yeah. Kind of the first or second. So Corkley and Andrew Dye had done something a few years ago, but they'd missed the key point, which is the thing you pre train on has to be a general purpose corpus.

So no one quite realized this key thing, and maybe I had a bit of fortune here that my background was in philosophy and, cognitive science, and so I'd spent some decades thinking about this. The technical architecture of of of, ULM fit. Just just sketch that out.

I'm a huge fan of regularization. I'm a huge fan of taking a model that's incredibly flexible, and then making it more constrained not by decreasing the size of the architecture, but by adding regularization. So even that at the time was extremely controversial, but that was by no means a unique insight of ours.

So what Stephen Merritt had done is he'd taken the extreme flexibility of an LSTM, a kind of ver classic stateful recurrent neural net towards which things are kind of gradually heading back towards nowadays, and it added five different types of regularization. He added every type of regularization you can imagine, and then that was my starting point, was to say okay, I now have a massively flexible deep learning model that can be as powerful as I want it to be, and it can also be as constrained as I need it to be, and then I needed a really big corpus of text. Funnily enough, this is also Steven, he had been at Common Crawl, and I think he helped or made the Wikipedia data set.

Then I realized actually the Wikipedia dataset made lots of assumptions. It had all these like unc for unknown words, because it all assumed classic NLP approaches. So I redid the whole thing, created a new Wikipedia dataset, and that was my general corpus.

And then I used a WDLSTM, trained it so it was actually overnight, so for eight hours on a gaming GPU, you know, because I was at the University of San Francisco, we didn't have heaps of resources. Probably like a twenty eighty TI or something, I suspect. And then the next morning when I woke up, I then it's the same three stage architecture that we do today, you know, pre training, mid training, post training.

Mhmm. So then I figured, okay, I've now that I've trained something to predict the next word of Wikipedia, it must know a lot about the world. I then figured if I then fine tune it on a corpus specific, so what we could now call supervised fine tuning dataset, which in this case was the dataset of movie reviews.

It would become especially good at predicting the next word of those, so I'd learn a lot about movies. Did that for like an hour, and then like a few minutes of fine tuning the downstream classifier, which was a classic academic, data set. It's kind of considered the hardest one, which was to take like 5,000 word movie reviews and to say like, is this a positive or negative sentiment, which today is considered easy.

around how you do the fine tuning? Yeah. So the how we do the fine tuning was something we had developed at Fast dot ai.

So this is kind of year one of Fast dot ai. So this is still in our very early days. And one of the extremely controversial things we did was we felt that we should focus on fine tuning existing models, because we thought fine tuning was important.

Some other folks were doing work contemporaneously with that, so Jason Usinski did some really great research, I think it's during his PhD, on how to fine tune models, and how good they can be, and some other folks in the in the computer vision world, we were, you know, amongst the first. There's a bunch of us kind of really investing in fine tuning. And so yeah, we we felt that using a single learning rate to fine tune the whole thing all at once made no sense because the different layers have different behaviors.

This is one of the things Jason Yusinski's research also showed. We developed this idea of like, well, it's also way faster if you just train the last layer. Right?

Because it only has to back prop the last layer. And then once that's pretty good, back prop the neck the last two, and then the last three, and then we use something called discriminative learning rates. So different layers we would give different learning rates to.

And then another critical insight that no one realized for years, even though we had told everybody, was that you actually have to fine tune every batch norm. So all the normalization layers you do actually have to fine tune because that's moving the whole thing up and down, they're all changing at scale. So yeah, when you do that, you can often just fine tune the last layer or two.

And we found that actually with ULM fit, although we did end up unfreezing all the layers, only the last two were really needed to get the close to state of the art result. So it like took like seconds.

I mean, there there was no received wisdom because nobody talked about it. No one cared. You know?

It's just this so like nearly no one cared. Transfer learning was just not something anybody thought about, and Rachel and I felt like it matters more than anything, you know, because only one person has to train a really big model once and then the rest of us can all fine tune it. So we thought we just should learn how to do that really well.

So we spend a lot of time just trying lots of things. But in the end, the intuition was pretty straightforward, and what intuitively seemed like it ought to work basically always did work, which is another big difference between how people still today tend to do ML research is I think it's all about ablations, and you can't make any assumptions or guesses. And it's not at all true.

activations and don't actually look at their gradients. So something we do in our software, in our fast AI software, is we have built into it this ability to to see in a glance what your entire network looks like. And once you've done it a few times, it just takes a couple of hours to learn, you can immediately see, oh, I see.

This is overtrained or undertrained or at this layer that something went wrong. It's not a mystery, you know. So basically what happens is, for example, you end up with with dead neurons that go to a point where they've got zero gradient regardless of what you do with them.

That often happens if they, you know, head off towards infinity. You can always fix that. So yeah, it's it's not as bad as people think by any means.

Something that trains well for continuous learning when done properly can also be done well to train well for a particular task if you're careful.

extremely helpful when it comes to thinking about AI. I find they behave more similarly than differently, and my intuition about each tends to work quite well. You know, with a human, when you learn something new, it's not about unlearning something else.

you know, semi supervised and self supervised learning was such an unappreciated area. And, yeah, Jan Lecun was absolutely one of the guys who was also working on it. Yeah.

I actually did a post because I was so annoyed at how few people cared about semi supervised learning. Did a whole post about it years ago. Jan LeCun looked at it for me as well and, you know, suggested a few other pieces of work that I'd I'd missed and but I was kind of surprised at how, you know, how incredibly useful it is to basically say like basically come up with a pretext task, right?

So envision so we did this envision before ULM fit. So it was like in medical imaging, you know, take a histology slide and predict, you know, mask out a few squares and predict what used to be there. So some of my students at USF I had doing stuff with that.

It was basically entirely taking stuff that we and others had already done in vision. Yep. So like this idea of masking out squares.

We didn't invent it. Masking out words was the obvious thing, you know, and this idea of gradually unfreezing layers we had done before in computer vision. The whole idea of starting with a pre trained model that was general purpose had been in computer vision.

There was a really classic paper actually in computer vision in might have been around 2015, was entirely an empirical paper saying look what happens when we take a pre trained ImageNet model predicting what sculptor created this or predicting what architecture style this is, and like in every task it got the state of the art result. And it really surprised me people didn't look at that and think like, I bet that ought to work in every other area as well, whether it be genome sequences, or language, or whatever. But people have a bit of a lack of imagination, I find.

They tend to assume things only work in one particular field.

Well, I I have always thought of it as a hierarchy of of abstractions. You know. It it it needs if it's gonna predict, you know, if the if the document is here was the, you know, opening that, you know, that Bobby Fischer used and has chest notation to predict the next thing, it needs to know something about chest notation or at least openings.

If it's like, you know, and this was vetoed by the 1956 US president, comma, you need to know like, you don't even you don't just need to know who the president was, but the idea that there are presidents. And therefore, the the idea that there are leaders, and therefore, the idea that there are groups of people who have hierarchies, and therefore, that there are people, and therefore, that there are objects. And, like, you can't predict the next word of a sentence well without knowing all of these things.

So that knowing, my hypothesis for why I created ULM fit was to say it would end to compress that as well as possible to get that knowledge, it would have to create these abstractions, these hierarchies of abstractions somewhere deep inside its model. Otherwise, how could it possibly do a good job of predicting the next word?

You know, I know I've actually, I remember chatting with Peter Norvig on camera, and both of us said, well, actually they kind of are creative. Like, we just got to a bit careful about our choices of words, I guess. So, you know, Peter Wozniak, who's a guy I really really respect, who kind of rediscovered space repetitive learning, built the super memo system, and is the modern day guru of memory.

The entire reason he's based his life around remembering things is because he believes that creativity comes from having a lot of stuff remembered, which is to say putting together stuff you've remembered in interesting ways is a great way to be creative. LLMs are actually quite good at that, but there's a kind of creativity they're not at all good at, which is, you know, moving outside the distribution. So, which I think is where you're heading with your question.

But I'm just kind of I'm framing it this way to say you have to be so nuanced about this stuff because if you say like they're not creative, it gives you the can give you the wrong idea because they can do very creative seeming things. But if it's like, well, can they really extrapolate outside the training distribution? The answer is no, they can't.

But the training distribution is so big, and the number of ways to interpolate between them is so vast, we don't really know yet what the limitations of that is. But I see it every day, you know, because I my my work is R and D. I'm constantly on the edge of and outside the training data.

I'm doing things that haven't been done before. And there's this weird thing, I don't know if you've ever seen it before, I see it but I see it multiple times every day, where the LM goes from being incredibly clever to like worse than stupid, like like not understanding the most basic fundamental premises about how the world works. Yeah.

Yeah. And it's like, oh, whoops, I fell outside the trading data distribution. It's gone dumb.

And then like there's no point having that discussion any further because Yes. You know, you've lost it at that point. Yes.

I mean, I think they can't go outside their distribution because it's just something that that type of mathematical model can't do. You know, I mean, can do it, but it won't do it well. You know, when you look at the kind of two d case of of fitting a curve to data, once you go outside the area that the data covers, the curves disappear off into space in wild directions, you know.

And that's all we're doing, but we're doing it in multiple dimensions. You know, I think Boden might be pretty shocked at how far compositional creativity can go when you can compose the entirety of the human knowledge corpus. And I think this is where people often get confused because it's like so for example, I was talking to Chris Latner yesterday about how Claude Anthropic, you know, had had got Claude to write a C compiler, And they were like oh, is a clean room C compiler.

You can tell it's clean room because it was created in Rust, you know, and so Chris created the kind of, you know, I guess it's probably the top most widely used c c plus plus compiler nowadays playing on top of LLVM, is the most widely used kind of foundation for compilers. They're like, oh, well, Chris didn't use Rust. This is, know, and we didn't give it access to any compiler source code, so it's a clean room implementation.

But that misunderstands how LLMs work, right? Which is all of Chris's work was in the training data. Many many times LLVM is used widely and lots and lots of things are built on it, including lots of C and C plus plus compilers.

Converting it converting it to Rust is an interpolation between parts of the training data, you know. It's a style transfer problem. So it's definitely compositional creativity at most, if you can call it creative at all.

And you actually see it when you look at the the repo that it created. It's copied parts of the LLVM code, which today Chris says like, oh, I made a mistake. I shouldn't have done it that way.

Nobody else does it that way, You know? Oh, wow. Look.

They're the only other one that did it that way. That doesn't happen accidentally. That happens because you're not actually being creative.

translates them to Rust. Like, that's that's all it did, which is impressive. Yeah.

And if I'm much less familiar with math than I am computer science, but from talking to mathematicians, they tell me that that's also what's happening with like Adolf's problems and stuff. It's some of them are newly solved. Yeah.

But they are not sparks of insight.

Elon Musk said something a bit similar a few days ago, saying like, oh, LLMs will just spit out the machine code directly. We won't need libraries, programming languages. Yeah.

Yeah. Look, the thing is none of these guys have have been software engineers recently. I'm not sure Dario's ever been a software engineer at all.

Software engineering is a unusual discipline, and a lot of people mistake it for being the same as typing code into an IDE. Coding is another one of these style transfer problems. You you take a specification of the problem to solve, and you can use your compositional creativity to find the parts of the training data which interpolated between them solve that problem, and interpolate that with syntax of the target language, and you get code.

There's a very famous essay by Fred Brooks written many decades ago, No Silver Bullet, and which it almost sounded like he was talking about today. It it but he was specifically saying something he was responding to something very similar, which is in those days it was all like, what about all these new fourth generation languages and stuff like that, you know, we're not going to need any coders anymore, any software engineers anymore, because software is now so easy to write, anybody can write it. And he said, well, he guessed that you could get at maximum a 30% improvement.

He specifically said a 30% improvement in the next decade, but I don't think he needed to limit it that much because the vast majority of work in software engineering isn't typing in the code.

So in some sense, parts of what Dario said were right. Just like for quite a few people now most of their code is being typed by a language model. That's true for me.

Say like maybe 90%. But it hasn't made me that much more productive, because that was never the slow bit. It's also helped me with kind of the research a lot and figuring out, you know, which files are going to be touched.

But anytime I've made any attempt at getting an LLM to like design a solution to something that hasn't been designed lots of times before, it's it's horrible because what it actually every time gives me is the design of something that looks on its surface a bit similar. And often that's going to be an absolute disaster because things that look on the surface a bit similar and, like, I'm literally trying to create something new to get away from the similar thing. It's very misleading.

Well, that's true, and that's terrifying. There's often these arguments online between people who are like, LLMs don't understand anything. They're just pretending to understand.

Mhmm. And then other people are like, don't be ridiculous. Look what this LLM just did for me.

Right? And the funny thing is they're both right. LLMs cosplay understanding things.

Like, they pretend to understand things. And this was the interesting thing about the early kind of, work with like, cognitive science work with like Daniel Dennett. That's basically what the Chinese room experiment is.

Right? It is you've got a guy in a room who can't speak Chinese at all, but he sure looks like he does because you can feed in questions and he gives you back answers, but all he's actually doing is looking up things in a huge array of books or machines or whatever. The difference between pretending to be intelligent and actually being intelligent is entirely unimportant as long as you're in the region in which the pretense is actually effective, you know.

So so it's actually fine for a great many tasks that LLMs only pretend to be intelligent, because for all intents and purposes it it just doesn't matter until you get to the point where it can't pretend anymore.

Yeah. Yeah. So that so Searle versus Dennett, or is it versus Searle and Dennett, was what everybody was talking about when I back when I was doing my undergrad in philosophy, you know, so think Consciousness Explained came out about then, probably Chinese Room a little bit before.

That's interesting because the discussions were the same discussions we're having now, but they've gone from being abstract discussions to being real discussions. It's helpful if people go back to the abstract discussions because it helps you get out of your, you know, it's very distracting at the moment to look at something that's cosplaying intelligence so well, and go back to the fundamental question. So anyway, I just wanted to mention that's kind of it's this interesting situation we're now in, where it's very easy to really get the wrong idea about what AI can do, particularly when you don't understand the difference between coding and software engineering.

Yeah. Which then takes me to your point or your question about the implications of that for organizations. Yep.

You know, a lot of organizations are basically betting their futures on a speculative premise, which is that AI is gonna be able to do everything better than humans, or at least everything in coding better than humans. I I worry about this a lot both for the organizations and for the humans, you know. For the humans, when you're not actively using your design and engineering and coding muscles, you don't grow.

You might even wither, but you at least don't grow. And, you know, speaking of the CEO of an r and d startup, you know, if if my staff aren't growing, then we're gonna fail. You know?

We that we can't let that happen. And getting better at the particular prompting skills, whatever details of the current generation of AI, CLI frameworks isn't growing. You know, that's that's like that's as helpful as learning about the details of some AWS API when you don't actually understand how the internet works, you know.

It's not it's not reusable knowledge, it's ephemeral knowledge. So like if you wanted to, you can actually use it as a learning superpower, but also it can do the opposite.

I I created a the the first deep learning for medicine company called Denlytic back in, what was that, like 2014, and our initial focus was on radiology, and a lot of people were worried that this would cause radiologists to become less effective at radiology. Yeah. And I strongly felt the opposite, which is and I did quite a bit of research into this of, like, what happens when there's, like, fly by wire in airplanes or anti lock brakes in cars or whatever.

If you can successfully automate parts of a task that really are automatable, you can allow the expert to focus on the things that they need to focus on. And we saw this happen. So in radiology we found if we could automate identifying the possible nodules in a lung CT scan we were actually good at it, which we were, and then we've the radiologist then can focus on looking at the nodules and trying to decide if they're malignant or what to do about it.

So again it's one of these subtle things. So if there's things which you can fully automate effectively in a way that you can remove that cognitive burden from a human so that they can focus on things that they need to focus on, that can be good, you know. I don't know where we sit in software development, because, you know, I've been coding for forty ish years, so I've written a lot of code and I can glance at a screen of code and then, you know, unless it's something quite weird or sophisticated, can immediately tell you what it does and whether it works and whatever.

I can kind of see intuitively things that could be improved, you know, possible things to be careful of. I'm not sure I could have got to that point if I hadn't have written a lot of code. So the people I'm finding who can really benefit from AI right now are either really junior people who can't code at all, who can now write some apps that they have in their head, and as long as they work reasonably quickly with the current AI capabilities, then they're happy.

And then really experienced people like like me or like Chris Latner, because we can basically have it do some of our typing for us, you know, and some of our research for us. People in the middle, which is most people, most of the time, it really worries me because how do you get from point A to point B without typing code? It might be possible, but we don't have a we have no experience of that.

We don't is it possible? How would you do it? Like, is it kind of like going back to school where at primary school we don't let kids use calculators so that they develop their number muscle?

Do we need to do that for like first five years as a developer? You have to write all the code yourself. I I don't know.

But if I wasn't between like two and twenty years of experienced developer, I would be asking that question of myself a lot because otherwise you might be in the process of making yourself obsolete.

they didn't learn anything. Right. Yeah.

No, exactly. Desirable difficulty is the concept that kind of comes up in education. But even going back to the work of Ebbinghaus, who was the original respect repetitive space learning guy in the nineteenth century, and then Piotr Wozniak more recently, it's we find the same like, we we we know that memories don't get formed unless it is hard work to form them.

So, you know, that's where you kind of get this somewhat surprising result that says revising too often is a bad idea because it comes to mind too quickly. And so with repetitive spaced learning with stuff like Anki and SuperMemo, the algorithm tries to schedule the flashcards at a just before the moment you're about to forget. So then it's hard work.

So I I studied Chinese for ten years in order to try to learn about learning myself, And I really noticed this that I used Anki, and because it was always scheduling my cards just before I was about to forget them, it was always incredibly hard work. You know, to do reviews because almost all the cards were once I was on the verge of forgetting. That was absolutely exhausting.

But my god, worked well. Here I am. I don't really haven't done any study for fifteen plus years, and I still remember my Chinese.

Yeah. Absolutely. And so, know, all I know is in in my company, I just I tell our staff all the time, almost the only thing I care about is how much your your personal human capabilities are growing.

You know, I don't actually care how many PRs you're doing, how many features you're doing. Like, there's that nice, you know, John Osterhout, the TCL guy, recently released some of his Stanford Friday takeaway lectures, and he has this nice one called a little bit of slope makes up for a lot of intercept. Just basically the idea that that, you know, in your life if you can focus on doing things that cause you to grow faster Yeah.

It's way better than focusing on focusing on the things that you're already good at, you know, that has that high intercept. So the only thing I really care about, and I think is the only thing that matters for my company, is that my team, I'm focusing on their slope. Yeah.

If you focus on just driving out results at the limit of whatever AI can do right now, you're only caring about the intercept, you know. So I think it's basically a path to obsolescence through both a company and the people who are in it. And so I'm really surprised how many executives of big companies are pushing this now because it feels like if they're wrong, which they probably are, and they have nowhere to tell if they are because this is an area they're not at all familiar with and never learned it in their MBAs.

They're basically setting up their companies to be destroyed. Yeah. And really surprised that, you know, shareholders would let them do that.

It all set up such an incredibly speculative action. Yeah. Here we are.

50 times more high quality software than they were before. So we've actually just done a study of this, and there's a tiny uptick, tiny uptick in what people are actually shipping. That's the facts.

Obviously, I'm an enthusiast of AI and what it can do.

It's this really awkward situation where it's very almost everybody I know who got very enthusiastic about AI powered coding in recent months have totally changed their mind about it when they finally went back and looked at, like, how much stuff that I built during those days of great enthusiasm am I using today? Are my customers using today? Am I making money from today?

Almost all the money is being made by influencers, you know, or by the companies that produce the tokens. The thing about AI based coding is that it's like a slot machine, and that you you have an illusion of control, you know, you can get to craft your prompt, and your list of MCPs, and your skills, and whatever, and then but in the end you pull the lever, right? You put in the prompt, and something comes back, and it's like cherry, cherry, it's like oh, next time I'll change my prompt a bit, I'll add a bit more context, pull the lever again, pull the lever again.

It's the stochastic thing. You get the occasional win. It's like, oh, I won.

I got a feature. So it's got it's got all these hallmarks of like loss disguised as a win, somewhat stochastic, feeling of control, all the stuff that, gaming companies try to engineer into their gaming rooms. Now none of that means that AI is not useful, but gosh, it's hard to tell.

Yeah. Definitely. I've had some successes.

Right? And so, in fact, we've spent the last couple of years building a whole product based around where we know the successes are going to be, which is when you're working on reasonably small pieces that you can fully understand, and that you can design, and you can build up your own layers of abstraction to create things that are bigger than the parts that you're building out of. Had a very interesting situation recently where I just it's kind of an experiment basically, which is we rely very heavily on something called IPykernel, which is the thing that powers Jupyter notebooks, And there had been a major version release of IPykernel from '6 to '7, and it stopped working.

And it stopped working in both of the products that we were trying to use it with. One was was called today nb classic, which is the original Jupyter notebook, and then our own product called Solveit that would just randomly crash. And IPy kernel's over 5,000 lines of code.

It's very complex code, multiple threads, events, blocks, interfaces with IPython, you know, with ZMQ, you know, all kinds of different pieces, DebugPy, and I couldn't get my head around it, and I couldn't see why it was crashing. The tests are all passing. I wonder if AI can solve this.

You know, it's like I'm always interested in the question of like how big a chunk can AI handle on its own right now. The answer turned out to be yes, I think it can just. It was like, so I spent a couple of weeks, I didn't develop a lot of understanding about how IPykernel really worked in the process, but I did spend quite a bit of time kind of pulling out separate comp like, so the answer was in two hours codecs five point I think it was 5.

2 at that time or maybe three had just come out, couldn't do it. Then if I got the $200 a month GPT 5.3 Pro to fix the problems, it could.

And so by rolling back between those two pieces of software and those two models, I could get things working over a couple of weeks period. And like you say, it wasn't at all fun. It was very tiring, and it felt stressful because I wasn't really in control.

But the interesting thing is I now am in a situation where I have the only implementation of an of a Python Jupyter Kernel that actually works correctly, as far as I can tell, with these new version seven protocol improvements. And now I'm like, well this is fascinating, because we don't have a kind of a software engineering theory of what to do now. It's like, here's a piece of code that no one understands.

Yeah. Am I going to bet my company's product on it? And I the answer is I don't know, because like I I don't I don't like I don't know what to do now, because no one's like been in this situation, and like will it does it have memory leaks?

Will it still work in a year's time if there's some minor change to the protocol? Is there some weird edge case that's going to destroy everything? No one knows because no one understands his code.

It's a really curious situation.

and to be clear, I quite like that, framing. But, you know, what that actually does is it says, well, software engineering sure is important then, because software engineering is all about finding what those pieces are and how they should behave, and then how you can put them together to create a bigger piece, and then how you can put them together to create a bigger piece. And if we do that well, then in ten years time, we could have software that is far more capable than anything we could even imagine today.

But you're only going to get that with really great software engineering. Yeah. You want to be careful.

I think in the end like IPykernel I'm finding for example, it's just too big a piece, right? Because in the end the team that made the original IPykernel were not able to create a set of tests that correctly exercised it, and therefore real world downstream projects, including the original nb classic, you know, which is what IPykernel was extracted from, didn't work anymore. So this is this is kind of where our focus is on now on the development side at Answer dot ai, is finding the right sized pieces and making sure they're the right pieces.

Knowing how to recognize what those pieces are, and how to design them, and how to put them together is actually something that normally requires some decades of experience before you're really good at it. Certainly it's true for me. I reckon I got pretty good at it after maybe twenty years of experience.

Yeah, it's a big question. It's like how do you build these software engineering chops which are now even more important than they've ever been before, they're the difference between somebody who's good at writing computer software and somebody who's not. That feels like a challenging question.

Yeah, I'd say probably not. You see companies like Instagram and WhatsApp dominate their sectors whilst having 10 staff, and beating companies like Google and Microsoft in the process. I would argue this way of building software in very large companies is actually failing, and I think we're seeing a lot of these very large companies becoming, you know, increasingly desperate.

And, you know, for example, the quality of Microsoft Windows and Mac OS has very obviously deteriorated greatly in the last five to ten years. You know, back when Dave Cutler was looking at every line of the NT kernel and making sure it was beautiful, it was a elegant and marvelous piece of software, you know, and this I don't think there's anybody in the world who's gonna say that Windows 11 is an elegant and marvelous piece of software. So I actually think we do need to find these smaller components that we do fully understand, and that we need to build them up.

And here's the problem, AI is no good at that. So and and so I say that empirically. They're really bad at software engineering.

And then I think that's possibly always going to be true, because, you know, we're we're asking them to often move outside of their training data. You know, if we're trying to build something that literally hasn't been built before and do it in a better way than has been done before, we're saying like don't just copy what was in the training data. So and again, this is a confusing point for a lot of people because they see AI being very good at coding, and then you think like oh, that's software engineering.

You know, it's like oh, it must be good at software engineering. But it's they're different tasks. There's not a huge amount of overlap between them, and there's no current empirical data to suggest that LLMs are gaining any competency at software engineering.

Every time you look at a piece of software engineering they've done, like the browser, for example, which Cursor created, or the C compiler which Anthropic created. Like I've read the source code of those things quite a bit. Chris Latner is much more familiar with the compiler example than me, but they're they're very very obvious copies of things that already exist.

So that's the challenge, you know, is if you want to build something that's not just a copy, then you can't outsource that to an LLM.

and for our own internal development. The stuff I've been working on for twenty years has turned out to be the thing that makes this all work. Stephen Wolfram should get credit for this.

He was the guy that created the notebook interface, although also lots of ideas kind of go back to Smalltalk, and Lisp, and APL. But basically the idea that a human can do a lot more with a computer when the human can like manipulate the objects in inside that computer in real time, study them, and move them around, and combine them together. Yeah.

That's what small talk was all about, you know, with objects, and APL was the same with arrays. Mathematica basically is a super powered Lisp, which then also added on this very elegant notebook interface that allowed you to construct kind of a living document out of all this. So I built this thing called nbdev a few years ago, which is a way of creating production software inside these notebook interfaces, inside these rich dynamic environments.

And I found that made me dramatically more productive as a programmer. And, like, today, even though I've never been a full time programmer as my job, when you look at my kind of GitHub repo output, I think GitHub produced some statistics about it, and I was like just about the most productive programmer in in Australia. You know, like it it's working.

And a lot of the stuff I build has lots and lots of people use it because it's such a rich, powerful way to build things. And so it turns out, we've now discovered that if you put AI in the same environment with the human, again in a in a rich interactive environment, AI is much better as well, which perhaps isn't shocking to hear, but the the normal, like if you use code, which I know you do, it's a very good piece of software, but the environment we give Claude code is very similar to the environment that people had forty years ago, you know. It's a it's a line based terminal interface.

You know, it can use MCP or whatever. Most of the times it just nowadays uses bash tools, which again very powerful. I love bash tools.

I use them all the, you know, CLI tools all the time. But it's still just it's using text files, you know, as its as its interface to the world. It's it's it's really meager.

So so we put the human and the AI inside a Python interpreter, and now suddenly you've got the full power power of a very elegant expressive programming language that the human can use to talk to the AI, the AI can talk to the computer, the human can talk to the computer, the computer can talk to the AI. Like, you have this really rich thing, and then we let the human and the AI in real time build tools that each other can use. And that's what it's about to me.

Right? It's about, like, creating an environment where humans can grow and engage and share. It's like for me, when I use Solveit, it's the opposite of that experience you described with Claude Code.

After a couple of hours, I feel energized and happy and fulfilled.

But also Claude Code was not designed to do this. It's not very good at it, and it doesn't make it the natural way of working with it. I don't want to say it's an AI literacy problem because that's like saying like oh, it's a you problem.

To me, if a tool is not making it the natural way for a human to become more knowledgeable, more happy, more connected, with a deeper understanding and a deeper connection to what they're working on, that's a tool problem. That that should be how tools are designed to work.

called I don't like notebooks. It was hilarious. It was really well done.

And, I was totally wrong. And all the things he said notebooks can't do, they can. And all the things he said you can't do with notebooks, I do with notebooks all the time.

So it was a very good, amusing incorrect talk. So then I did a kind of a parody of it called I like notebooks, in which I basically copied with credit most of his slides and showed how every one of them was totally incorrect. But like I actually think your comment about it does come down to the heart of it, is this this difference between like how software engineering is normally done versus how scientific research and similar things is normally done.

And I think and I agree there is a dichotomy there. And I think that dichotomy is a real shame because I think software development is being done wrong. It's being done in this way which is, yeah, all about reproducibility and these like dead these dead pieces, you know.

It's it's all dead code, dead files. I I will never be able to express this one millionth as clearly as as Brett Victor has in his work, so I'd encourage people who haven't watched Brett Victor to to to watch him. But, know, he he shows again and again how a direct connection, you know, direct visceral connection with the thing you're doing is is all that matters, you know.

And that's his mission is to make sure people have that connection. And that's basically my mission as well. So for me, traditional software engineering is as far from that as it is possible to get.

I think it's I think it's gross. Like I I find it disgusting, and I find it sad that people are being forced to work like that. It's like I think it's inhumane, and I just don't think it works very well.

I mean, empirically it doesn't work very well, and it's much less good for AI as well, as it's much less good for humans. It hasn't always been that way, like, know, with with Alan Kay and Smalltalk and Iverson and APL, you know, Lisp Wolfram with Mathematica. To me these were the golden days when when people were focused on the question of how do we get the human into the computer to work as closely with it as possible.

You know, that's where the the mouse came from, for example. I've got to like click and drag and visualize entities in your computer as things you can move around. So I feel like we've lost that.

I think it's really sad. Yeah. With Claude code and stuff, the the default way of working with them is to go super deep into it.

It's like, okay, there's a whole folder full of files. You never even look at them. Your entire interaction with it is through a prompt.

Yeah. I it it it literally disgusts me. Like, I literally think it's it's inhumane, and it's my mission remains the same as it has been for, like, twenty years, which is to stop people working like this.

you don't want to remove people from their exploratory environment, you know. Research and science is developed by people building insight, you know. Whoever you listen to, you know, whether it be Feynman or whatever, like it's you always hear from the great scientists how they build deeper intuition by by building mental models, which they get over time, by interacting with the things that they're learning about.

And like in Feynman's case, because it was theoretical physics, he couldn't actually pick up a spinning quark, but he did literally study spinning plates, you know. You've got to find ways to to deeply interact with with what you're working with. Like so so many times I've seen data science teams because you're right, data science teams aren't very familiar with Git, and aren't very familiar with things that they do need to understand.

And so often I've seen a software engineer will become their manager, and their fix to this will be to tell them, oh, to stop using Jupyter notebooks. And now they have to use all these reproducible blah blah virtual, you know, virtual end blah blah. They destroy these teams over and over again.

I've seen this keep happening because the solution is not create more discipline and bureaucracy. It's solve the actual problem. So for example, we we built a thing called an nb merge driver, which a lot of people don't realize this, but actually notebooks are extremely git friendly.

It's just that git doesn't ship with a merge driver for them. So git only ships with a merge driver for line based text files, but it's fully pluggable. And so you can easily plug in one for JSON files instead.

And so we wrote one. So now when you diff, you know, when you get a git diff with our merge driver, you see cell level diffs. If you get a merge conflict, get surge level cell level merge conflicts.

The notebook is always openable in Jupyter. Nbdime did the same thing, so two independent implementations of this. So yeah, there were problems to solve, you know, but the solution to it was not throw away Brett Victor's ideas and make people further away from from their exploratory tools.

But to fix the exploratory tools, and I think all software developers should be using exploratory based programming to deepen their understanding of what they're working with. So that they end up with a really strong mental model of the system that they're building, and they're working with, and then they can come up with better solutions, more incrementally, better tested. I basically never have to use a debugger, because I basically never have bugs.

And it's not because I'm a particularly good programmer. It's because I build things up small little steps, and each step works. And I can see it working, and I can interact with it.

Yeah. So do you do this one absolutely. Dude, but, like, there's absolutely no, like so nbdev ships with out of the box CI integration, and that's, like, the tests are literally there.

Like, because the source is a notebook, the entire exploration of like how does this API work, you know, what does it look like when you call it, the implementation of the functions, the examples of them, the documentation of them, the tests of them are in one place. So it's much easier to be a good software engineer in this environment.

Tell me about that. Do you think we should be worried about AI existential risk? I mean, that was a certain time, wasn't it?

And I feel like things have changed a bit. Thank God. I feel like we we, not just being Aravind, but broadly speaking, the community of which we're apart kind of probably won that.

Now we have other problems to worry about. But, you know, basically at that point, the prevailing narrative was AI is about to become autonomous. It could become autonomous at any moment and could destroy the world.

So very much comes from, you know, Alicia Yukowski's work, which I think clearly has been shown to be wrong at many levels at this point. That they would refute that, obviously. Of course they would.

Yeah. It's one of those things that they can always refute just like any doomsday cult, unless you give it a date, and the date passes.

which was and is, that it's misunderstanding where the actual danger is. Which is that when you have a dramatically more powerful technology entering the world that can make some people dramatically more powerful, people who are in love with power will seek to monopolize technology, and the more powerful it is, the more strong that urge from those power hungry people will be. So to ignore people so here's the problem.

If you're like, I don't care about any of that. All I care about is autonomous AI taking off, you know, singularity paper clip, nano goo, whatever. The obvious solution to that is, oh, let's centralize power.

And this is which is what we kept seeing, particularly at that time. Let's give either very rich technology companies, or the government, or both, all of this power, and make sure nobody else has it. In in my threat model, that's the worst possible thing you can do, because you've centralized the ability to control in one place, and therefore these people who are desperate for power just have to take over that thing.

even if it turns out to be incredibly powerful, right? Like I don't I don't even want to argue about whether it's going to be powerful, because that's speculative. Even if it's going to be incredibly powerful, you still shouldn't centralize all of that power in the hands of one company or the government.

Yeah. Because if you do, all of that power is going to be monopolized by power hungry people and used to destroy civilization, basically. You'll end up with a case where all of that wealth and power will be centralized with the kinds of people who who want it centralized.

So, like, society for hundreds of years have faced this again and again and again, you know. So when it's like, you know, writing used to be something that only the most exclusive people had access to knowing about writing. And the same arguments were made.

If you let everybody write, they're gonna use it to write things that we don't want them to write, and it's gonna be really bad, you know. Ditto with printing, ditto with the vote, like, like and again and again society has to fight against this natural predilection of the people that have the status quo power to be like, no, this is a threat. So when we're saying like, okay, what if AI turned out to be incredibly powerful?

Would it be better for society to be that to be kept in the hands of a few, or spread out across society? My argument was the latter. Now, there's also an argument which is like, hey, don't worry about it.

It's not going be that powerful anyway. I I just didn't want to go there, because it's not an argument that's easy to win, because you can't really say what's going to happen. We're all just guessing.

But I can very clearly say, like, well, if it happens, would it be a really good idea to only let Elon Musk have it? Or would it be good idea to only let Donald Trump have it? Dan Hendricks spoke about this offense defense asymmetry.

I mean, I think some of them are the ones we've discussed, which is people enfeebling themselves by basically losing their ability to be to become more competent over time. That's that's that's the big risk I worry about the most. The privacy risk, it's there, but I'm not sure it's much more there than it was for Google and Microsoft before.

Like, you know, you used to work at Microsoft. You know how much data they have about the average Outlook, Office, etcetera, user. Ditto for Google, you know, the average Google Workspace or Gmail user.

Those privacy issues are real, although I think there are bigger privacy issues around these companies which the government can outsource data collection to. So back in the day, used to be companies like ChoicePoint and Acxiom. Nowadays, it's probably more companies like Palantir.

The US government is actually prohibited from building large databases about US citizens, for example. But it's not prohibited companies are not prohibited from doing so, and the government's not prohibited from contracting things to those companies. So I mean, that's a huge worry, but I don't think it's one that AI is uniquely creating.

It certainly it's like you're in The UK, as you know, in The UK surveillance has been universal for quite a while now. It certainly makes it easier to use that surveillance. But a sufficiently well resourced organization could just throw a thousand bodies at the problem.

So, yeah, I'm not sure these are new privacy problems. There's maybe more common ones than they used to be. Yeah.

Hope you had a nice trip. Thank you so much.