Context Engineering for Agents - Lance Martin, LangChain

How'd you get into it? Well, you know, it's funny. Buzzwords emerge oftentimes when people have a shared experience.

And I think lots of people started building agents kind of early this year, mid this year, quote, unquote, the year of agents. And I think what happened is when you put up put when you kind of put together an agent, it's just tool calling a loop. It's relatively simple to lay out, but it's actually quite tricky to get to work well.

In a particular, managing context with agents is a hard problem. Karpathy put out that tweet, canonizing the term context engineering. And he kind of mentioned this nice definition, which is context engineering is the challenge of feeding an LM just the right context for the next step, which is highly applicable to agents.

And I think that really resonated with a lot of people. I, in particular, had that experience over the past year working on agents. And I wrote about that a little bit in my piece talking about building open deep research over the past year.

So I think it was kind of an interesting point that the term capture a common experience that many people were having, and it took hold because of that.

the term? Like, what what is it? Well, I think that, you know, prompt engineering is kind of a subset of context engineering.

I think when we kind of move from chat models and chat interactions to agents, there's a big shift that occurred. So with chat models, working with ChatGPT, the human message is really the primary input. And, of course, a lot of time and effort is spent in crafting the right message that's passed to the model.

With agents, the game is a bit trickier though, because the agent's getting context not just from the human, but now context is flowing in from tool calls during the agent trajectory. And so I think this was really the key challenge that I observed and many people observed is like, oof. When you put together an agent, you're not only managing, of course, the system instructions, system prompt, and, of course, user instructions.

You also have to manage all this context that's flowing in at each step over the course of a large number of tool calls. And I think there's been a number of good pieces on this. Manus put out a great piece talking about content engineering with madness.

And they made the point that the typical Manus task is like 50 tool calls. Anthropics Multi Agent Research is another nice example of this. They mentioned that the typical production agent, and this is probably referring to Claude Code, could be other agents that they've produced, is like hundreds of tool calls.

When I had my first experience with this, and I think many people have this experience, you put together an agent. You're sold the story that's just tool calling in a loop. That's pretty simple.

You put it together. I was building deep research. These research tool calls are pretty token heavy.

And suddenly you're finding that my deep researcher, for example, with a naive tool calling loop was using 500,000 tokens. It was like a dollar to $2 per run. I think this is an experience that many people had.

And I think it's kind of that the challenge is realizing that, oof, building agents is actually a little bit tricky because if you just naively plumb in the context from each of those tool calls, naively, you just hit the context window of the LLM. That's kind of the obvious problem. But also, Jeff from Chroma spoke about this on the recent pod.

There's all these weird and idiosyncratic failure modes as context gets longer. So Jeff has a nice report on context rot. And so you have both these problems happening.

If you build a naive agent, context is flowing in from all these tool calls. It could be dozens to hundreds, and there's degradation of performance with respect to context length, and also the trivial problem of hitting the context window itself. So this was kind of, I think, the motivation for this new idea of actually, it's very important to engineer the context that you're feeding to an agent.

I will say it's funny. And, actually, I was hoping you'd bring this up. I also have a blog post, but it's all moving so quickly Yeah.

That I did a meetup after the blog post and updated the story a little bit with this meetup. So, actually, this is a better thing to show. But I do have a blog post too.

But things change between my blog post and the meetup, which are, like, two weeks apart. So that's how quickly these things are moving. Exactly.

That's the blog post. Should we do this sequentially then? I think it's actually okay to just hit the meetup because it's just easier to follow one thing.

And it it's it's like a super settle blog post story.

naive agents, and the first time I built an agent, agent makes a bunch of tool calls. Those tool calls are passed back to the LLM at each turn, and you naively just plumb all that context back. And of course, what you see is the context window grows significantly because this tool feedback is accumulating in your message history.

A perspective that Mana shared in particular I thought was really good. It's important and useful to offload context. Don't just naively send back the full context of each of your tool calls, you can actually offload it, and they talk about offloading it to disk.

So they talked about this idea of using the file system as externalized memory, rather than just writing back the full contents of your tool calls, which could be token heavy, write those to disk, and you can write back a summary. It could be a URL, something so that the agent knows it's retrieved a thing. It can fetch that on demand, but you're not just naively pushing all that raw context back to the model.

So that's this offloading concept. Note that it could be a file system. It could also be, for example, agent state.

So Landgraf, for example, has this notion of state. So it could be kind of the agent runtime state object. It could be the file system.

But the point is you're not just plumbing all the context from your tool calls back into the agent's message history. You're saving it in an externalized system. You're fetching it as needed.

This saves token costs significantly.

summary or blurb about what's in the file? This is actually a very interesting and important point. So I'll give an example from what I did with OpenDeepResearch.

So OpenDeepResearch is a deep research agent that I've been working on for about a year, and it's now, according to deep research spends, the best performing deep research agent, at least on that particular benchmark. So it's it's pretty good. Listen.

It's not as good as open as deep research, which use end to end RL. It's all fully open sourced, and it's pretty strong. So I just do carefully prompted summarization.

I try to prompt the summarization model to give an exhaustive set of kind of bullet points of the key things that are in the post, just so the agent can know whether to retrieve the full context later. So I think it's kind of prompting, if you're doing summarization, carefully for recall. Compressing it, but, like, making sure that all the key bullet points necessary for the LLM to know what's in that piece of full context is actually very important when you're doing this kind of summarization step.

Now Cognition had a really nice blog post talking about this as well. And they mentioned you can really spend a lot of time on summarization, so I don't wanna trivialize it. But at least my experience has been, it's worked quite effectively.

Prompt a model carefully to capture exactly. So in this post, they talk a lot about even using a fine tuned model for performing summarization. In this case, they're talking about agent to agent boundaries and summarizing, for example, message history.

But the same challenges apply to summarizing, for example, the full contents of token heavy tool calls so the model knows what's in context. I basically spent a lot of time prompt engineering to make sure my summaries capture with high recall what's in the document, but compress the content significantly.

That's exactly right. And actually, one of the other big themes I hit and and talk about quite a bit is context isolation with multi agent. And I do think this does link back to the cognition take.

Which is interesting. So their argument against multi agent- Is literally called don't build multi agent. Correct.

And what they're arguing is a few different things. One of the main things is that it is difficult to communicate sufficient context to sub agents. They talk a lot about spending time on that summarization or compression step.

They even use a fine tuned model to ensure that all the relevant information so they actually show it a little bit down below as kind of a linear agent. But even at those agent agent boundaries, they talk a lot about being careful about how you compress information and pass it between agents.

I think that's the thing that's still unclear to me. That's one of the most important points about this context isolation kind of bucket. So Cognition argues, which actually I think is a very reasonable argument, they argue, don't do sub agents because each sub agent implicitly makes decisions, and those decisions can conflict.

So you have sub agent one doing a bunch of tasks, sub agent two doing a bunch of tasks, those kind of decisions may be conflicting, and then when you try to compile the full result, in your example of coding, there could be tricky conflicts. I found this to be the case as well. And I think a perspective I like on this is use multi agent in cases where there's very clear and easy parallelization of tasks.

Cognition, and Walden Jens spoke on this quite a bit, he talks about this idea of kind of read versus write tasks. For example, if each sub agent is writing some component of your final solution, that's much harder. They have to communicate like you're saying, and agent to agent communication is still quite early.

But with deep research, it's really only reading. They're just doing context collection, and you can do a write from all that shared context after all the sub agents work. I found this worked really well for deep research.

Actually, Anthropic report on this too. Their deep researcher just uses parallelized sub agents for research collation, and they do the writing in one shot at the end. This works great.

It's a very nuanced point that what you apply context isolation to in terms of the problem, yes, you can see this is their work, matters significantly. Coding may be much harder. In particular, if you're having each sub agent create one component of your system, there's many potentially implicitly conflicting decisions each of the sub agents are making.

When you try to compile a full system, there may be lots of conflicts. With research, you're just doing context gathering in each of those sub agent steps, and you're writing in a single step. So I think this was kind of a key tension between the cognition take, don't do multi agents, and the anthropic take, hey, multi agents work very well.

It depends on the problem you're trying to do with multi agents. So this was a very subtle and interesting point. What you apply multi agents to matters tremendously, and how you use them.

I like the take that apply multi agents to problems that are easily parallelizable, that are read only, for example, context gathering for deep research, and do the final quote unquote write, in this case, report writing, at the end. I think this is trickier for coding agents. I did find it interesting that Claude code now allows for sub agents.

So they obviously have some belief that this can be done well. Or at least it can be done. But I still think I actually kind of agree with Walden's take.

It can be very tricky in the case of coding if sub agents are doing tasks that need to be highly coordinated.

So RAG is obviously a it has been in the air for now many years, obviously, well before LMs and this client whole wave. One thing I found pretty interesting is, for example, different code agents take very different approaches to retrieval. Varun from Windsurf shared an interesting perspective on how they approach retrieval in the context of Windsurf.

So they use classic code chunking along carefully designed semantic boundaries, embedding those chunks. So classic kind of sem semantic similarity vector search and retrieval. But they also combine that with, for example, GREP.

They then also mention knowledge graphs. They then talk about combining those results, doing your ranking. So this is kind of your classic complicated multistep rag pipeline.

Now what's interesting is Boris from Anthropic and Cloud Code has taken a very different approach. He's spoken about this quite a bit. Cloud Code doesn't do any indexing.

It's just doing, quote unquote, agentic retrieval, just using simple tool calls, for example, using grep, to kind of poke around your files. No indexing whatsoever. And obviously works extremely well.

So there's very different approaches to kind of rag and retrieval that different code agents are taking. And this seems to be kind of an interesting and emerging theme. Like, when do you actually need more hardcore indexing?

Yeah. I actually just saw a client posted, I think, yesterday, talking about that they only scrap. They don't do indexing.

And so I think within the retrieval area of context engineering, there are some interesting trade offs you can make with respect to, are you doing kind of classic vector store based semantic search and retrieval with a relatively complicated pipeline like Varun's talked about with Windsurf, or just good old kind of agentic search with basic file tools? I will note, I actually did a benchmark on this myself. I think there's a shared blog post somewhere.

I'll bring it up right now. Yep. I actually looked at this a bit myself.

This was a while ago. But I compared three different ways to do retrieval on all Landgraf documentation for a set of 20 coding questions related to Landgraf. So I basically wanted to allow different code agents to write Landgraf for me by retrieving from our docs.

I tested ClotCode and Cursor. I used three different approaches for grabbing documentation. So one was I took all of our docs, around 3,000,000 tokens.

I indexed them in the vector store and just did classical vector store search and retrieval. I also use an LMs dot TXT with just a simple file loader tool. So that's kind of more like the agentic search.

Just basically look at this LM dot TXT file, which has all of the URLs of our documents with some basic description, and let the LM or the code agent in this case just make tool calls to fetch specific docs of interest. And I also just tried context stuffing. So take all the docs, 3,000,000 tokens, and just feed them all to the code agent.

So these are just some results I found comparing Claude code to cursor. And interesting, what I actually found this is only my particular test case. But I actually found that LM dot text with good descriptions, which is just very simple.

It's just basically a markdown file with all the URLs of your documentation and, like, a description of what's in that doc, just that passed to the code agent with a simple tool just to grab files is extremely effective. And what happens is the code agent can just say, okay, here's the question. I need to grab this doc and read it.

I'll read it. I need to grab this doc. Read it.

Read it. This worked really well for me, and I actually use this all the time. So I actually personally don't do vector store indexing.

I actually do LM dot TXT with a simple search tool with Cloud Code. It's kind of my go to Cloud Code. In this case, this was done a few months ago.

These things are always changing. This particular point in time, Cloud Code actually outperformed cursor for my test case. This actually Cloud Code pilled me, this was I did this back in April.

So I've been kind of on Cloud Code since. But that was really it. So this kind of goes to the point that Boris has been making about Cloud Code, about incline as well.

You give an LM access to simple files file tools. In this case, I actually use an LM dot TXT to help it out so it can actually know what's in each file.

Take a repo, compile it down to some kind of easily kind of readable, yeah, LM dot TXT. What I actually found was even using an LM to write the descriptions helped a lot. So I have actually a little package on my GitHub where it can rip through documentation and just pass it to a cheap LLM to write a high quality summary of each doc.

This works extremely well. And so that llm dot text then has LLM generated yeah. This one.

This is a little repo. It got almost no attention. And but I find it to be very useful.

So basically, can it's it's trivial. You just point it to some documentation. It can kind of rip through it, grab all the pages, send each one to an LLM.

And LLM writes a nice description, compiles it into an LLM dot TXT file. I found when I did this and I then fed that to Claude Code, Claude Code's extremely good at saying, okay. Based on the description, here's the page I should load.

Here's the page I should load from the question asked. I use it when I'm trying to generate Elm dot TXT for new documentation. But I've done this for Landgraf.

I've done it for a few other libraries that I use frequently. You just give that to Cloud Code, then Cloud Code can rip through and grab docs really effectively. Super simple.

The only catch is I found that the descriptions in your elemental text matter a lot because the element actually has to use the descriptions to know what to read. You know? Anyway, that's just a nice little utility that I use all the time.

When we had a client that said the Context seven MCP by Upstash, which is like an MCP for, like, project documentation and stuff like that, was one of the most used. Have you seen have you tried it? Have you seen anything else like that that automates some of the stuff away?

Well, you know, it's funny. We have an MCP server for Landgraf documentation that basically gives, for example, Clot code, the n l m dot txt file in a simple search file search tool. Now Cloud has built in fetch tools, but at the time we built it, it didn't.

But it's a very simple NCP server that exposes LM dot TXT files to, for example, Cloud Code. It's called MCP doc. So it's a little very simple utility.

I use that all the time. Extremely useful. So you basically can just point it to all the elm dot text files you wanna work with.

very early. Yeah. This is actually a very interesting point.

I've spoken with folks from Anthropic about this quite a bit. It is I found that storing prompts in MCP servers is actually pretty important, but in particular, tell the LM or code agent how to use the server. And so I actually end up do having kind of separate servers for different projects with specific prompts.

And also sometimes I'll have you can also sort of resources. So sometimes I'll have specific resources for that particular project in the server itself. So I actually don't mind separating servers project wise with project specific kind of context and prompts necessary for that particular task.

That's exactly right. And actually, the the prompting thing is pretty important because even to use our little simple MCP doc server for Landgraf docs, You actually I found it it's better, of course, if you prompt it. But then I had to put in the read me initially, oh, okay.

Here's how you should prompt it. But of course, that prompt can just live in the server itself. So you can kind of compartmentalize the prompt necessary for the LM to use the server effectively within the server itself.

And this was a problem I saw initially. A lot of people were using our MCP doc server, and then finding, oh, this doesn't work well. And it like, oh, it's a skill issue.

You need to prompt it better. But then that's that's our problem. This the prompt should actually live in the server and should be available to the code agent.

Right. So it knows how to use the server. Right.

So that that's maybe retrieval, and that's a whole retrieval is a big theme. It obviously predates this new term of context engineering, but there's a lot going on in the retrieval bucket. It certainly is an important subset of context engineering.

I'm wondering if there's any other trends in retrieval before we leave the topic.

Alright. Happy to move on. We could talk about maybe reducing context briefly.

Everyone's had an experience with this because if you use Claude code, you hit that 95 you know, you've hit 95% of the context window, and you're about to and cloud code's about to perform compaction. So that's like a very intuitive and obvious case in which you want to do some kind of context reduction when you're near the context window. I think an interesting take here, though, is there's a lot of other opportunities for using summarization.

We talked about it a little bit previously with offloading. But actually, at tool called boundaries is a pretty reasonable place to do some kind of compaction or pruning. I use that in OpenDeepResearch.

Hugging Face actually has a very interesting OpenDeepResearch implementation. It actually uses, like, not a coding agent, but the code agent agent implementation. So instead of tool calls as JSON, tool calls are actually code blocks.

They go to a coding environment that actually runs the code. And one argument they make there is that they perform some kind of summarization or compaction and only send back limited context to the LLM. Leave the raw tool call itself, which is often token heavy as we're talking about deep research, in the environment.

It's another example. Anthropica, the multi agent researcher, also does summarization of findings. I think you see pruning show up all over the place.

It's pretty intuitive. I think an interesting counter to pruning was made by Manus. They make the point and the warning that pruning comes with risk, particularly if it's irreversible.

And cognition kinda hits this too. They talk about we have to be very careful with summarization. You can even fine tune models to do it effectively.

That's actually why Manus kind of has the the perspective that you should definitely use context offloading. So perform tool calls, offload the raw observations to, for example, disk, so you have them, then sure. Do some kind of pruning summarization, like Alessio was asking before, to pass back to the LM useful information, but you still have that raw context available to you.

So you don't have kind of lossy compression or lossy summarization.

Drew Brunick has a nice blog post that hits this point. He talks about this theme of context poisoning, and apparently, Gemini reports on this in their technical report. He talked about, for example, a model can perform a hallucination, and that hallucinations then is stuck in the history of the agent.

And it can kind of poison the context, so to speak, and and kind of steer the agent off track. And I think he cited a very specific example from Gemini two five playing Pokemon they mentioned in the tech report. So that's one perspective on this issue of we should be very careful about mistakes in context that can poison the context.

That's perspective one. Perspective two is like you're saying is if an agent makes a mistake, for example, calling a tool, you should leave that in so it knows how to correct. So I think there is an interesting tension there.

I will note, it does seem that Claude code will leave failures in. I noticed when I work with it, for example, it'll it'll kind of have an arrow. The arrow will get printed, and it'll it'll kind of use that to correct.

So and in my experiences, work with agents in particular, for tool call errors, I actually like to keep them in personally. That's just been my experience. I don't try to prune them.

Also, what it's worth, it can be kind of tricky to prune from the contact from the message history. You have to decide when to do it. So you if you're introducing a bunch more code, have to manage.

So I'm not sure I love the idea of kind of selectively trying to prune your message history when you're building an agent.

Any other commentary on on stuff that, you know, you particularly like or disagree with? So he and I did did a meetup on this. And I I kinda like this quote from Stuart Brand.

It was kind of comical. If you wanna know where the future is being made, look for where language is being invented and lawyers are congregating. And he was talking about this this idea of why buzzwords emerge.

And he actually was the one who turned me on to this idea that a term like context engineering catches fire because it captures an experience that many people are having. They don't come out of nowhere. And if you scroll down a little bit, he kinda talks about this.

He's a whole post about kind of I think it's how to build a buzzword. But he talks a lot about this idea of of kind of successful buzzwords are capturing a common experience that many of us feel. And I think that's kind of the the genesis of context engineering is also largely because many of us build agents.

Oh, there's lots of ways that can be quite tricky. And, oh, context engineering is kind of what I've been doing. And you hear a number of people saying, and then you can it kind of resonates.

You say, oh, okay. Yes. That describes my experience.

people try to create language that doesn't really resonate, that doesn't capture common experience. It tends to flop. So which is to say that buzzwords kind of coevolve with the ecosystem.

They tend to kind of become big and and resonate because they actually capture experience. Many people try to coin terms that don't actually resonate that go nowhere. Alessia, do you have experience with that?

ThotLead. That hits a lot of the major themes. I can maybe just talk very brief about one more.

We could talk about Bitter Lesson and some other things. Yeah. If you go back to that table, I just wanted to give Manus a shout because I thought they had one other very interesting point.

We've talked about offloading, reducing context, retrieval, context isolation. Those are, I think, the big ones. You can see very commonly used.

I do wanna highlight Manas. I thought they had a very interesting take here about caching, and it's a good argument. When people have the experience of building an agent, the fact that it runs in a loop and that all those prior tool calls are passed back through every time is quite like a shock the first time you have an agent.

You have one token every tool call, and you incur that token cost every pass through your agent. And so Manus talks about the idea of just caching your prior message history. It's a good idea.

I haven't done it personally, but it seems quite reasonable. So caching reduces both latency and cost significantly. But don't most of the APIs auto cache for you?

I mean, if you're using, like, OpenAI, you would just automatically have a cache hit. I'm actually not sure that's the case. For example, when you're building age, you're passing your message history back through every time.

As far as I know, it's stateless.

Yeah. Yeah. Yeah.

Yeah. Well, you know, it's it's interesting, though. So APIs are now supporting caching more and more.

That's fantastic. I'd use Anthropic's explicit cacher caching header in the past. I do think an important and subtle point here is that caching doesn't solve the long context problem.

So it, of course, solves the problem of, like, latency and cost. But if you still have a 100,000 tokens in context, whether it's cached or not, the LM is utilizing that that context. This came up.

I actually asked Anton this in their Context Rod meetup or in their context route webinar, and they kind of had mentioned that the characterization of context route that they made, they they think they would expect to apply whether or using caching. Caching shouldn't actually help you with all the context route and long context problems. It absolutely helps you with latency and cost.

Well, I do wonder what else can be cached.

Yeah. You know, that that's a great point. It actually echoes some of Walt and Dan's points from cognition.

Also that the summarization compaction step is just is nontrivial. You have to be very careful with it. Devin uses a fine tuned model for doing summarization within the context of coding.

So they obviously spend a lot of time and effort on that particular step. And Menace kind of calls out that they are very careful about information loss. Whenever they do pruning, compaction, summarization, they always use a file system to offload things so they can retrieve it.

Yeah. So that's a good thing to hit actually. I maybe think about this on two dimensions, Writing memories, reading memories, and then the degree of automation on both of those.

So take the simplest case, which actually I quite like, Claude code. How do they do it? Well, for reading memories, they just suck in your CloudMDs every time.

So every time you spin up a Cloud Code, it pulls in all your CloudMDs. For writing memories, the user specifies, hey, I want to save this to memory, and then Cloud Code writes it to CloudMD. So on this axis of, like, degree of automation across read write, it's kinda like the zero zero.

It's very simple, and it's kinda very, like, Boris pill, but, like, super simple, and I I actually quite like it. Now the other extreme is maybe ChatGPT. So behind the scenes, ChatGPT decides when to write memories and decides when to suck them in.

And actually, I thought Simon at AI Engineer had a great talk on this, and he it wasn't about memory, but he hit memory in the talk. And he mentioned, I don't know if you remember this, but it was a failure mode in image generation because he wanted an image of a particular scene, and it sucked in his location and put it in the image. Like, it sucked in half, like, half moon bay or something and sucked in an image.

And it was a case of memory retrieval gone wrong. He didn't actually want that. So even in a product like ChatGPT that spent a lot of time on memory, it's nontrivial.

And I think my take is the writing of memories is tricky. Like, when actually should the system write memories is is nontrivial. Reading of memories actually kinda converges with the concept engineering thing of retrieval.

Like, memory retrieval at large scale is just retrieval. Right? I I kinda view them as It's retrieval in a certain context, which is your past conversations, which That's right.

You know, it it it is different than retrieval from a knowledge base, different than retrieval from the public web. By the way, this is sentence write up on his website on on here where he was just trying to generate images, and then suddenly it shows up. There you go.

Actually, it's a subtle point. I I don't know exactly know what OpenEye does behind the hood with respect to memory retrieval. My guess is they're indexing your past conversations using semantic vector search and probably other things.

So Yeah. It may still be using some kind of knowledge base or vector store for retrieval. So in that sense, I kind of view it just simply as, you know, in the case of sophisticated memory retrieval, it is just like a a complex rag system in the same way we talked about with like Varun and building Windsurf.

It's kind of a multistep rag pipeline. So I I kind of view memories, at least the reading part, as just, you know, it's just retrieval. And, actually, I quite like Klaus' approach.

It's very simple. Just retrieval is trivial. Just suck it in every time.

and I built I had this little email assistant that I used to run my email. I actually think this is made of a sidebar memory. Memory pairs really well with human in the loop.

So for example, in my little email assistant, it's just an agent that runs my email. I have the opportunity to pause it before it sends off an email and correct it if I want, like change the tone of this email, or I can literally just modify the tool call to have a little UI for that. And every time when you have these ambient agents, you edit, for example, or you give it feedback, you edit the tool cost itself, that feedback can be sucked into memory.

And that's exactly what I do. So I actually think memory pairs very nicely with human loop. And, like, when you're using human loop to make corrections to a system, that should be captured in memory.

And so that's a very nice way to use memory in kind of a narrow way that's just capturing user preferences over time. It actually uses LLM to actually reflect on the changes I made, reflect on the prior instructions in memory, and just update the instructions based upon my edits. And that's a very simple and effective way to use memory when you're building ambient agents that I quite like.

And yeah. I mean, you know, you guys have done plenty of talks on async agents. That's right.

But I think it's a very good point that memory is often kind of confusing when to use it. I think a very clear place to use it is when you're building agents that have human loop. Because human loop is a great place to update your agent memory with your preferences.

So it kind of gets smart over time and learn stream. It is exactly what I do with my little email assistant. So Harrison, I'm I'm sure I think he said this publicly, uses an email assistant for all his emails.

He gets a lot as a CEO. I get much fewer because I'm just a lowly guy. But I still use it.

So there's a great talk from Hyunghwang Chung, previously OpenAI, now I have MSL, on the bitter lesson and his approach to AI research. The take is compute 10 x's every five years for the same cost. Of course, we all know that.

The history of machine learning has shown yeah. Exactly this slide exactly. History of machine learning has shown that actually capturing this scaling is the most important thing.

In particular, algorithms that are more general with fewer inductive biases and more data and compute tend to beat algorithms with more, for example, hand tuned features, inductive biases built in, which is to say, just letting a machine learn how to think itself with more compute and data rather than trying to teach machine how we think tends to be better. So that's kind of the bitter lesson piece simply stated. So his argument is this subtle point that at any point in time, when you're, for example, doing research, you typically need to add some amount of structure to get the performance you want at a given level of compute.

But over time, that structure can bottleneck your further progress. And that's kind of what he's showing here, is that in the low compute regime, kind of on the left of that x axis, adding more structure, for example, more modeling assumptions, more inductive biases, is better than less. But as compute grows, less structure and this is exactly the bitter lesson point, less structure, more general, tends to win out.

So his argument was we should add structure at a given point in time in order to get something to work with the level compute that we have today, but remember to move it later. And a lot of his argument was like people often forget to remove that structure later. And I think my link here is that I think this applies to AI engineering too.

And if you kind of scroll down, I have the same chart showing my Exactly. Little This is my little example of building deep research over the course of a year. So I started with a highly structured research workflow.

Didn't use tool calling. I embedded a bunch of assumptions about how research should be conducted. In particular, don't use tool calling because everyone knows tool calling is not reliable.

This was back in 2024. Decompose the problem into a set of sections and parallelize each one. Those sections written in parallel into the final report.

What I found is you're building LM applications on top of models that are improving exponentially. So while the workflow was more reliable than building an agent back in twenty eight four, that flipped pretty quickly as LMs got better and better. It's exactly like was mentioned in the Stanford talk.

You have to be constantly reassessing your assumptions when you're building AI applications given the capabilities of the models. I talk a lot about here the specific structure I added, the fact that I used the workflow because we know tool calling doesn't work. This was back in 2024.

The fact that I decomposed the problem because it's how I thought I should perform research, and this basically bottlenecked me. I couldn't use MCP as MCP got, for example, much more popular. I couldn't take advantage of the fact that tool calling was getting significantly better over time.

So then I moved to an agent, started to remove structure, allow for tool calling, let the agent decide the research path. A subtle mistake that I made, which links back to that point about failing to remove structure, I actually wrote the report sections within each sub agent. So this talks this kind of links back to what we talked about with sub agents in isolation.

Sub agents don't communicate effectively with one another. So if you write report sections in each sub agent, the final report is actually pretty disjoint. This is exactly Alessio's challenge and problem about using multi agent.

So I actually hit that exact problem. So I ripped out the independent writing and did a one shot writing at the end. And this is the current kind of version of OpenDeepResearch, which is quite good.

And this is kind of the thing that's, at least on deep research, the best performing open deep research assistant. At least that that's open source. So it was kind of my own arc.

Although we do have some bad results with g p d five that that are quite strong. So, you know, the models are always getting better. And so indeed, our open source assistant actually takes advantage and rides that wave.

But I actually kind of experienced I felt like I got actually a bitter lesson to myself because I started with a system that was very reliable for the current state of models back in mid twenty twenty four, early twenty twenty four, but I was completely bottlenecked as models got better. I had to rip out the entire system and rebuild it twice, rechecking my assumptions in order to kind of capture the gains of the model. So I think I just wanna flag.

I think this is an interesting point. It's hard to build on top of rapidly expanding models or rapidly improving model capability. And Actually, I really enjoyed from AI engineer Boris' talk on Cloud Code, and their very bitter lesson build.

He talks a lot about the fact that they make Cloud Code very simple and very general because of this fact. They want to give users unfettered access to the model without much scaffolding around it. But I think it's an interesting consideration in AI engineering that we're building on top of models that are improving exponentially.

And one of the points he makes is a correlator of the bitter lesson, is that more general things around the model tend to win. And so when building applications, we should be thinking about this. We should be adding structure necessary to get things to work today.

And by keeping a close eye on models improving rapidly and removing structure in order to unbottleneck ourselves. I think that was my takeaway. So I really liked the talk from Kyung Wan Chung.

I think that's worth everyone listening to. And I think a lot of lessons apply to AI engineering.

In the earlier model regime, the structure approach is actually better. And so an interesting take on this, Jared Kaplan, the founder of Anthropic, has a great talk at startup school from a couple weeks ago. And he mentions this point about oftentimes building products that explicitly don't quite work yet can be a good approach because a model under them is premixed essentially, and it'll kind of unlock the product.

We saw that with Cursor. Part of the Cursor lore is that it it did not work particularly well. Quad three five hits, and then boom.

It kind of unlocks the product. And so you kind of hit that knee of the curve when the model capability catches up to the product needs. But in that earlier regime, the structured approach appears better.

So it's kind of this interesting subtle point that for a while, the more structured approach appears better, and then the models finally hits the capability needed to unlock your product, and suddenly your product just takes off.

is very important if you believe in the bitter lesson, which which you do. No. No.

This is super important, actually. And I actually talked about this at end of the post. Yeah.

There's an interesting subtlety when you talk about Asian frameworks. And a lot of people are anti framework. I completely understand and sympathetic to those points.

But I think when people talk about frameworks, there's two different things. So there can be a low level orchestration framework. There's a great talk, for example, at Anthropic from Shopify.

They built this orchestration framework called Roast internally. And it's basically Landgraf. It's some kind of way to build kind of internal orchestration workflows with LMs.

And Roast, Landgraf provides you low level building blocks, nodes, edges, state, which you can compose into agents. You can compose into workflows. I don't hate that.

I like working with low level building blocks. They're pretty easy to tear down, rebuild. In fact, I used, for example, Landgraf to build OpenDeep Research.

I had a workflow. I rip it out. I rebuild with agent.

The building blocks are low level. Just nodes, edges, state. But the thing I'm sympathetic to is there's also, in addition to just kind of low level orchestration frameworks, there's also agent abstractions from framework import agent.

That is actually where you can get into more trouble, because you might not know what's behind that abstraction. I think when a lot of people kind of are anti framework, I think what they're really seeing is they're largely anti abstraction, which I'm actually very sympathetic to. And I per I don't particularly like agent abstractions for this exact reason.

And I think Walden Yance made a good point. Like, we're very early in the arc of agents. We're like in the HTML era.

And agent abstractions are problematic because you don't know what's necessarily under the hood of the abstraction. You don't understand it. And if I was building, for example, OpenDeepResearch with an abstraction, I wouldn't necessarily know how to rip it apart and rebuild it when models got better.

So I'm actually wary of abstractions. I'm very sympathetic to that part of the critique of frameworks. But I don't hate low level orchestration frameworks that just provide nodes, edges.

You can just recombine them in any way you want. And then the question is, why use orchestration at all? And, actually, I use Landgraf because you get some nice use you get checkpoint and you get state management.

It's low level stuff. And that's the way I happen to use Landgraf, and that's why I like Landgraf. And that's actually why a lot of I found, like, a lot of customers like Landgraf.

It's not necessarily for the agent abstraction, which I agree can be much trickier. Some people like agent abstractions. That's completely fine as long as you understand what's under the hood.

But I think that's a very interesting debate about frameworks.

Just kind of my bread and butter. It's good. And what you know, what's funny is this talk follows exactly the arc we often see when we're talking to companies about Landgraf.

It is people wanna build agents and workflows internally. Everyone rolls their own. It becomes hard to kind of manage and coordinate and review code in lot in this context of large organizations.

It can be very helpful to have a standard library or framework that people are using with low level components that are easily composable. That's what they build with Roast. That's effectively what Landgraf is, and that's why a lot of people like Landgraf.

I actually thought the talk on MCP that I believe it was it was Jean Welsh.

John Welsh's talk. It's actually very good. It's very good.

He makes a case for a lot of the reason why people actually for example, enterprises, large companies like Landgraf, which is the fact that when tool calling got good within Anthropic in, you know, sometime mid last year, he actually makes this point explicitly. So he mentions, okay. So you're in Anthropic.

Tool calling gets good in mid twenty twenty four. Everyone's building their own integrations. It becomes complete chaos.

And that's actually where kind of MCP came from. Let's build a kind of a standard protocol for accessing tools. Everyone adopts it.

Much easier to kind of have auth and have review, and you minimize cognitive load. And this is actually the argument for standardized tooling, whether it be frameworks or otherwise within larger orgs is practicality. And he act his his whole talk is making that very pragmatic point, which is actually why people do tend to like frameworks, for example, in larger in larger organizations.

thing that they do. That's right. Lance, you've been so generous of your time.

Thank you. Any shameless plugs, calls to action, stuff like that. Yeah.

If if you'll made it this far, thanks for listening. We have a a bunch of different courses I've taught, one on ambient agents, one on building OpenDeep research. So I actually was very inspired by Karpathy had a tweet a long time ago talking about building on ramps.

So he talked about he had his micrograb repo. A few people looked at it, but not that many. He made a YouTube video, and that created an on ramp, and the repo skyrocketed in popularity.

So I like this one two punch of building a thing like OpenDeep research, then creating a class so people can actually understand how to build it themself. And I kinda like that. Build a thing, create an on ramp for it.

So I have a class on building open deep research. Feel free to it's for free. But it walks through a bunch of notebooks as to how I build it.

And and you can see the agent is quite good. We even have better results coming out soon with g p d five. So if you want kind of an open source deep research agent, have a look at it.

It's been pretty fun to build. And that's exactly what I talked about in that Bidder Lesson blog post as well. Awesome, Lance.

Thank you for joining. Yeah. A lot of fun.

Great to be on.