Notion’s Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future — Simon Last & Sarah Sachs of Notion

we've put a lot of effort into our MCP, and we have a fantastic team that we're building.

How's it feel? We ship things slowly. So it had been in alpha for a little bit.

And at the point at which is it's in alpha, there's a group of people that are making sure it's ready for prod. And then there's a group of people working on the next thing. So sometimes some of these launches are a bit delayed satisfaction.

So it's quite nice to remind yourself all the work he did because we do have a habit of being two or three milestones ahead just because you have to be. You can't get complacent. But it's been great that people understood how this is helpful.

And I think that's just easier in general building AI tools today than it was two, three years ago. People get it. And so that user education, there's just it was our most successful launch in terms of free trials and converting people and things like that.

It was really successful. So yeah. But there's a lot to build.

Making it free for three months helps.

with the Frontier Labs and with Fireworks, like a function calling model on Notion functions. This is right when I joined. I joined because we needed a manager.

Simon was needed to be able to go on vacation. That's around when I joined, you so can speak much more to it. Yeah.

kind of a similar capability. And that one just took longer because we just wanted to get the reliability up a lot higher because it's actually running in the background. And the product interface of, like, permissions and understanding this custom agent is shared in a Slack channel with x group of people and has access to documents that are surfaced to y group of people.

And the intersect of x versus y might not be whole. And so how do you build the product around making sure administrators understand that permissioning took multiple swings.

and maintaining a code base of a service where there's a bunch of agents working together inside? And, like, how does that work? If you think back to your initial question, like, why did this take so long?

I think something Notion's I didn't say that. But, yes. Okay.

Go ahead. What changed over the three a years of trying? Because most people always say it didn't work yet, then reasoning models came, then it worked.

I was like, okay. Let's go a little bit. I mean, that's part of it.

But I think the other part of it that I actually think is really what will set Notion apart for every new capability is we have two skills that are crucial when it comes to frontier capabilities. One is not letting yourself swim upstream. So like quickly realizing if you're just pressing against model capabilities versus not exposing the model to the right information, not having the right infrastructure set up, that in of itself is a skill of intuition.

And the second is to see, okay, you're not swimming upstream. Which direction is the river flowing? And what is like how do we think ahead about the product and start building it even if it's not great yet so that when it is there, we're ready for it.

And like those can sometimes feel like counterintuitive things. Like we can be trying to fine tune a tool calling model when they don't exist yet. And the trick is to not do that for too long, but realize that there was something there.

And we've had a lot of things which like we're just like not swimming in the right direction with the streams. I think we had multiple versions of transcription before we got meeting notes. Right?

Oh my god. I talk about that. Yeah.

Yeah. I I think that like, we we really closely partner with the Frontier Labs on capabilities. And we also have to have strong conviction on as those capabilities move.

Notion is about being the best place for you to collaborate and do your work. And how does that narrative change if the way that we work changes? Yeah.

Yeah. You told me you're a fan of the AgentLab thesis. And this is I show that thesis to so many candidates.

Like, I have it as my Chrome autofill at this point. Like, it's one of my most visited Is this the here's why you should work in Notion and not Here's what's different about it. Yeah.

And here's why it's not just a wrapper. I actually think more and more people understand it's not just a wrapper. And by the way, like, in the beginning, parts of what we build are wrappers on functionality that works well.

That's not really the most I would say that's not the product that that drives revenue, and that's not necessarily always what users need.

that I've been coming back to is Datadog and AWS. Yeah. Datadog could not exist without cloud storage.

That it's fundamental that works. And AWS has a CloudWatch product. But Datadog is an expert on understanding how people want observability on the products they launch.

And we're experts in understanding how people wanna collaborate. And that's really where our expertise lies Totally. Regardless of the tools that we use.

I'm curious how you think about implicit versus explicit expertise.

I actually think the failure of our team is when we focus too much on what are cools that are what are tools that are cool tools. I actually think that's when we make have the least velocity. Because you still need some sort of focus on a user journey.

So, like, for instance, we'll all sit down every Friday and look at the p 99 of the most token exhaustive custom agent transcript and just look at why it didn't do well and cut a bunch of tasks. Like, we still focus on this has like, this should work. Email triaging should work.

Right? And similarly, like, when we're talking about before building chatting before we started filming about, okay, how can I do PDF export? That's functionality that then merits maybe we should build a tool that has access to a computer sandbox and a file system and the ability to write code.

But it's because we're thinking about the fact that our users to do their daily work need to export PDFs not because we're like, I think a computer tool could be cool. Let's just see what happens. Like, we we have to focus on some user journeys.

Otherwise, we just don't have enough strategy to to prioritize.

I think the Taoist working with Sarah Sachs is it have to it depends if you ask. I think it depends if you're on my team or a partner or a vendor. Yeah.

The other people want to run their teams the way that you're you're you're, like, running these things. But then also similarly, Simon, when you did the custom agents demo, you had we've been using custom agents and here's the super long list of everything that we do. No humans ever read it.

That's what you said. I was like Yeah. So I think for me, something that I learned very quickly and became very comfortable with was that my job was not to be the ideas person or the technical expert.

My job was to make it so that everybody understood the objective, had a resource to help prioritize what they should work on, and had an avenue to prioritize what they thought was important. And I think it's true with all leadership. But I think especially on the AI team, almost all of our best ideas come from prototypes, from people that have a cool idea because they saw a user problem.

And it's a huge disservice if all of those ideas have to pass like the sniff test of what me and a product partner or Simon and Ivan decided were the direction. Right? Because a lot of what we're doing is leaning into capabilities.

So I think that's the first thing is I don't really view like the role of engineering leadership as hierarchical, nor has it ever been. But especially now, like very willing to change direction based on like proof is in the pudding. And I think we have rebuilt our harness three or four times.

And when you do that, then the second rule of engineering leadership is like, need to build a team that's comfortable deleting their own code. And is very low ego and is driven by what's best for the company and doesn't write design docs because they think it's their promotion packet. And that's a culture that Notion had long before I joined.

But, like, our willingness to just swarm on different problems and redo things that we've built before because something has changed, there's a lot of friction that can happen at companies when you do that. And it doesn't happen at Notion. And because it doesn't happen when new people join, like, don't wanna be the ones that are saying we shouldn't do this.

I wrote that code. So then it's you create a culture that everyone adopts. And that culture comes directly, I think, from Simon and Ivan, though, because they're very open minded.

Hence, obviously, just doing that in a loop every six months. Yeah. Do you believe in internal hackathons for this stuff?

I think there's two different versions. So one is we just have a solid bench of senior engineers that come and go on what we call the Simon Vortex and productionizing what we built. Right?

Because when you're in the Simon Vortex, the velocity is super high. The direction changes daily. And it's meant to be like the equivalent of a Skuckworks lab.

We don't need to do hackathons for that. We need to have senior engineers that we trust to come in and out of those projects. For instance, like, management boundaries are really loose.

Like, you report to him, but you work for her right now. Like, that's something that when we hire managers, it's important they don't care about because we tend to form work structures. Yeah.

Don't be too territorial. We form work structures after we ship things, not before, just historically. The second thing is we do have company wide hackathons.

Actually, we just had our demos day for the hackathon we had last week this morning. That's more for people that aren't directly working on the project, feeling like they have the time to pause and learn how to make themselves more productive or how they would use Notion custom agents to build something. Or part of the hackathon was actually encouraging everyone across the company to build their own agentic tool loop calling from scratch.

Following like in every blog post on how to do it, I think. Because we want Is that the compound engineering one? Yeah.

We want everyone to use Cloud Code in the company or whatever the coding agent they please and understand that fundamental. So we set aside a day and a half where all leadership encourage everyone on their teams across the company to do it. So we have hackathons like that.

I would say, like, facetiously. Like, everything we build is a little bit like a hackathon until it graduates and puts on big boy pants and as a product ops rollout later and has assigned data scientists and stuff like Security review, enterprise stuff. Actually, security review is one of the things that we bring in first because it just slows us down way more and causes a lot of tension, and they build better product if they're involved early.

That is probably the first person to get involved in something. PR approved answer. No.

No. But it's not just PR approved. It's like It's real.

It's actually real. It's I'm just saying dark tissue. Yeah.

Because my background's also I worked at Robinhood for a number of years. So compliance and things like that are a little bit more you learn the hard way when it doesn't come naturally. Yeah.

where that was necessarily aligned in product priorities. It'd be a lot of work. And we had someone on the database collections team, Jimmy, who was like, I really wanna do image generation for cover photos and inside Notion.

And we're like, if you wanna build it, like it's do it, please. Like, we encourage you. We gave him all the resources of working directly with Gemini and being able to track the token usage and it got working through our endpoints.

We gave him eval support, everything. And then it became a full project. Yeah.

That's why you can't have, like, ego as a leader. That's how we work. What's the size of the team today, both engineering and overall?

I manage the team that's what we'll call, like, core AI capabilities and infrastructure. That's about 50 people. But then we have I partner teams that do packaging.

So how it shows up in the corner chat versus custom agents versus meeting notes, that's another thirty, forty people. And then every team that has a product service at Notion that a user can interface with owns the tool that the agent interfaces with. The editor team.

The team that did CRDT for offline mode is the same team that handles how two agents edit competing blocks. It's the same problem. The team that built the underlying SQL engine is the same team that owns how the agent asked it to run a SQL query and it does it performantly.

And so from that regard, anyone working on product engineering is tasked with making them work for customers that are humans and agents. Because over time, a majority of our traffic will be coming from agents using our interface, not humans. And our objective is to make it so that the whole product org is building for agents.

Yeah. How does it change internally?

a prototype looks like just making it a feature flag that actually works. That's a bar. So the thing to understand that's really unique about Notion, one of the reasons I joined, we're super lucky, is no one uses Notion in their job as much as people that work at Notion.

Of course. So I think there's very few companies. Maybe if you worked on Chrome, I guess.

But like everything that we ship internally first and get a lot of really quick feedback. And also sometimes our dev instance is totally borked and you have to change a bunch of flags to get things done. And that's everyone.

So people that do IT ticketing, people that do supply chain procurement recruiting, everyone is using the same instance of Notion with a lot of flags on for these prototypes people build. I mean, so we have this Brian Levin, one of the designers on our team, I think evangelized this concept of demos over memos. Oh.

Too good. Which has been very good for building demos. And I think it's put a big pressure point on us to have really strong product conviction.

Because if anything can be demoed, you really need a strong filter of making sure that if you're doing x amount of work, you're making the you're focusing on one tower. You're not just building a really flat hill. That's actually where I think there has to be more conviction from our PMs and our designers, the company really to have conviction of what journey we're going on.

But overall, feel like it works pretty well.

But I think that okay. So that is true, Simon. But, like, being the person that manages the evals team, like, there is a level of intensity that it adds to the platform team.

But if we're gonna do image generation in Notion, all of sudden, the way that we do attachments and the way that we our LLM completion, like Cortex talks and expects tokens back, and now it's getting images back. There's a lot of platform work that we do need to solidify a little bit. So sometimes it'll be in dev for a couple weeks before it makes it to prod just because we still have to, like, make it robust, make it HIPAA compliant, ZDR compliant, figure out the right contracting with the vendor, whatever it is.

And we need to eval it because we want the team to still maintain what they build. That's the one thing. If we have a bunch of prototypes, it can't just be like a small group of people that then maintain whatever in prototypes.

So we have invested a lot of people in an eval and model behavior understanding teams that we call it agent dev velocity. So your dev velocity building agents can be faster if we invest in that platform. And so we have a whole org dedicated to Asian platform velocity so that you can build your own eval and then maintain it once you ship it.

So if a new model release comes out and we Every team maintains their own eval? We maintain the eval framework. Every team owns their own evals.

And a lot of them we've integrated to opt into CI or we run them nightly. And we have a team, a custom agent that triggers to a team to look at the major failures. That's really critical because if we have all these different services now, a lot of it's on the same agent harness.

So it's easier to maintain. It's just packaging of different agent harnesses. But new functionality of the agent let's say that, like, we wanna update, like, they deprecated Sonnet four or whatever it is.

And we need to auto update already?

37 just got deprecated. 37. I think it's a 52.

Or yeah. No. It's not 5.

2. It's not 5.2.

Yeah. Yeah. Yeah.

Five four is 40% more expensive than five two.

we definitely noticed flickiness. We've definitely noticed particularly for some providers that things are slower during working hours and There's a latency argument, not a quality argument. No.

I think the quality difference that's interesting is even though companies that say they're selling the same it's really into like quantize quantization. But like, companies that say they're selling the same model through different vendors, whether it be through First Party or BetterOc, Azure, etcetera, we do see different qualities sometimes. And that's not necessarily what's advertised.

that's Yeah. Embarrassing. We hire subprocessors to figure that out for us.

So we just wanna understand where it's regressing or where it's optimized. And sometimes we're okay with regressions that optimize latency if they're the appropriate regressions. Our job is to make sure we have the evals to understand the changes that are important to us.

And even like when we're partnering with labs on prereleases of models, they'll send us multiple snapshots. And this is less about quantization, but more just regressions. Like, they have shipped models that were not the snapshots that we wanted.

And they have changed the snapshots that they shipped based on the feedback that we give because our feedback tends to be more enterprise work focused and not coding agent focused. And definitely, those can be boomers. We know that this wasn't the version you wanted, but we'll help you make it work.

We always make it work, but that definitely happens. Yeah. Do you have failing evals that you're just hoping Oh.

That will have success eventually when a good model comes out? Yeah. So I think I could talk about this for sixty minutes, so I will limit myself.

I think it's a real issue when people say evals. And it's just like, that's quality. That's like unit it's like saying testing.

It's not just unit tests. So we have the equivalent of unit tests, regression tests. Those live in CI.

Those have to pass a certain percent within some stochastic error rate. Then we have, as you're building a product, evals of these aren't passing right now and this is launch quality. So we have a report card.

And we need to, on these categories, be at 80 or 90% of all of these user journeys to launch. And then what we have, what we call frontier or headroom evals, where we actively want to be at 30% pass rate. And that's actually been an effort that we took in partnership with Anthropic and OpenAI in the past maybe two or three months because we actually hit a point where our evals were saturated And we weren't able to really give insightful feedback other than it wasn't worse.

And not only is that not helpful for our partners, it's not helpful for us to understand where the stream is going, going back to that analogy. And so we spent a lot of time thinking about what Notion's last exam looks like. Right?

Not just humanity's last exam. MLE. Notion's last exam.

And there's a lot of dreams about what that would look like. I know we've talked a lot about benchmarking SWICS. But, yeah, Notion's last exam is a big thing inside the company, and we have people full time staff to it exclusively.

We have a data scientist, a model behavior engineer, and an full time evals engineer just dedicated to the evals that we pass 30% of the time. What you're hiring for? MBEs?

I am hiring. What is an MBE? A model behavior engineer.

Model behavior engineers started with a title data specialist before I joined when they were working with Simon on Google Sheets. And Simon just needed someone to look through Google Sheets and say, yes, no. This looks bad.

This looks good. And so we hired people with diverse linguistics background. We had a linguistics PhD dropout and a Stanford comp lit new grad.

And they're amazing. And they formed a new function, basically. And over time, we've built a whole team with a manager who's now reinventing what that role is with coding agents.

So they used to be manually inspecting code. Now they're primarily building agents that can write evals for themselves or LLM judges. There's a really funny day.

I can send you the picture where Simon, about a year and a half ago, was teaching them how to use GitHub. And they're on the whiteboard. And it was like, okay, I think it would be so much faster if our data specialists learned how to use GitHub and learned how to commit these things into code.

And that was then. And now I think coding has been a lot more accessible. But moving forward, it's this mix of like data science, test, PM, and prompt engineer.

Because there's craft in understanding, like, even what models can and can't do things. How do we define, like, that headroom? How do we define, like, what a good journey is?

Is this model better or not? Why is this failing? There's some qualitative work, but then there's also like a lot of instinct and taste to it.

Software engineering. And so we have like very firm conviction. And we have had for a number of years now that is its own career path.

be totally general. Yeah. I think it it'll be a mistake to fix it on any particular coding agent.

At the end of the day, it's just like CLI tools. It's like the same way that you would have a coding agent write the unit test. You should have a coding agent write the eval.

Yeah. But there's a lot of supervision in that still. We just don't believe that supervision has to come from software engineers because a lot of it is UXRE and whatever.

And these are the people that also triage failures and tell us where we should be investing next. Yeah. I'm gonna go ahead and ask a spicy question.

Is there a day there are no software engineers at Notion?

it depends on what you define as a software engineer. Yes. That's changing for sure.

I think every software engineer at Notion this summer went through like this sheer one of our engineering leads at the company called it like, every software engineer is going through the identity crisis that every manager goes through, where all of a sudden they realize their ability to write code is less important than their ability to delegate and context switch. And I think that is a transition out of being a software engineer.

that extra layer of human involvement in the process to make it and, like, one of our biggest use cases is bug triaging. So if someone posts something in Slack, and you just have a custom agent that lives there that has its own routing constitution of what team this belongs to, creates a task in your task database, and then posts in that Slack channel? That's one of the first things that we built internally, I think.

And it's completely changed the way that Notion functions as a company. Nothing falls through. Most things don't fall through the crack.

We don't know what we don't know. But it's not replacing people. It's replacing processes.

When we integrate with Gmail, for instance, we have a series of tools available that are available via MCP or API to Gmail. When we integrate with Notion Mail, we have the Notion Mail engineering team to build us the exact right tools that optimize latency, optimize performance and quality. They own that quality.

There's product leads there that are directly thinking about the user problems that happen in mail. So it tends to be when we build integrations and connections, we build natively first and then think about extending them generally just because it's also easier to build natively first. So that tends to be how we phase things out.

Talking about integrations, you prompted me, so I gotta ask MCP, CLI, what's going on? What's the Yeah. Opinion?

not from it working well for Notion, but how Notion, like, commits to both platforms. Notion is dedicated to being the best system of record for where people do their enterprise work. So we will always support our MCP insofar as other people are using MCPs.

Right? So regardless of our perspective, we've put a lot of effort into our MCP, and we have a fantastic team that we're building to do more there. And the second thing I'll say, I think we all think a lot, but lately, I've been thinking a lot about making sure there's a value alignment and pricing with capability.

needing language to execute deterministic tasks feels wasteful. And requiring on a language model to interface with third party providers seems wasteful for tasks that don't require it. And particularly because our custom agents are using usage based pricing, we think of pricing as like the barrier of entry for use of our product.

And we're quite committed to making sure that it's not wasteful. Not just because it's a bad deal for our customers, but it's also bad business. We want as many buyers.

Like, there's a there's an elasticity of demand. And so if we can have our agents properly execute code that calls on CLI deterministically, it's a one time cost. Right?

Versus constantly having a language model integrate with an MCP over and over and paying those like repeated token fees. And if it's happening outside the cash window, then you're paying for it over and over. And it's just unnecessary and less deterministic when it doesn't have to be.

Yeah.

Yeah. In Notion AI, though. Not explaining Yeah.

Notion AI. Ship an MCP Yeah. Yeah.

Yeah. In any AI. Yeah.

Internally. Okay. Is there ever a discussion on we're not gonna ship it because we're not able to tighten it down?

Or are you happy to just No. There are a lot of things where we choose not to use MCP because we wanna add more high touch to quality. I think search in Agentic Find is like the largest instance of that, where we have Slack and Linear and Jira search in Notion that is not using necessarily the Search MCP functionality that is provided by those companies.

And that's because it's quite critical, we think, to how our agent trajectories work is for us to have a little bit more control on the functionality of the search journey. And so it usually comes from quality. And there's a long tail of things.

And that's why we built an MCP client or an MCP server, excuse me, so that people can connect whatever they want. There is that long tail. Right?

But we for search particularly, I would say that's like the primary entry point. But there are other connections as well that it's a little bit of secret sauce about when we are okay with MCP functionality and user driven auth and when we actually wanna wanna carry along more ourselves. I think that there's not really a conflict here.

There's just like different layers of the stack and different abstractions.

call. Yeah.

need to hit continue. If you're curious, we can keep going. I'm just saying, like, that's really That's another one.

Yeah. Mean, no. Because there was tool calling.

And then there was research mode, which wasn't a fully agentic tool calling. Then we moved away from few shot prompting entirely to tool definitions.

basically moving from a DAG to like a true system with feedback, that's when we could distribute tool ownership to the teams much better. Because when it was all a few shots, it was everyone truly editing one string. And things would would compete in like the order.

There were all these all these papers about, oh, not all context is created equal. The higher up it is in your examples, like, the more the model listens and we're trying really hard to fight against the order and the selection of the fuchsia. And that really had to be a center of excellence.

And it didn't scale with the number of people for the need the company had. It was really just five or six people that were allowed to even touch that or had to approve it rather in our code base. And then now we can actually, with the right eval setup, distribute so that everyone owns their tool and their tool definition.

And sometimes we have crazy things where, like, we write two tools that have the same title and the agent crashes and stuff like that. So there are issues. Actually, believe it or not, Anthropic couldn't take it.

Sonic couldn't handle two tools with the same name in OpenAI, GPT 5.2. Was like, I can figure this out.

So that was an interesting one that we learned by accident through a SEV.

But so that was like a big shift for the company and velocity. Not immediate because the AI team that was the center of excellence team that owned that one file of few shop prompts had to become a platform team overnight, and that wasn't natural. Yeah.

the way the system works. And a big part of that is, what are the tools? How do they work?

Like, how should I prompt it to use the tools in the right way? I'd actually say we don't try to make it as easy as possible to use. Because the more we do that, the more we abstract away that interpretability that Simon's talking about that basically nerfs the model or nerfs the agent from being super capable.

So a huge, I would say, turning point. I can think about the week and a half that we all came together on this As we were building custom agents was that alignment that we're not trying to build for everyone here. We're not trying to build the model that or build the user experience that anyone can figure out how to use.

Because the more we do that, the more we just diminish its capabilities. And that was a big everyone in a couple Slack messages aligned on that actually made us all work faster again because we all were, like, more centralized on who we were building for.

And so when you ask it to write system prompt, it's just your agent itself is doing that. So this is just the model preference. You're not really injecting And then into the model too much.

I was saying we make such a good custom agent and Yeah. And things like that.

any of the settings manually. You can just talk to it. And the inside baseball is like how this works was probably the launch blocking part of this.

Especially because we had a lot of early adopters that were used to the old way. And that's like the benefit of adopting in public, but then changing how people think about setting up custom agents when they already had this flow in and of itself was difficult.

design, which I probably care about more than Simon. But the people that built this are three engineers from three different teams because we're like, we need to launch this and we to fix this. And then we've just built a company where then we just put people on it and no one complains.

The manager doesn't complain, and we were able to unblock and just ship it. Yeah.

value based pricing based on the task and things like that? So they are the credits and payment structures associated with the token usage. Reason that we had to make it not just throughput of tokens is that it's not always priced that way.

Like, our fine tuned and open source models are served on GPUs. Web search is priced differently. If we were to host sandboxes, those are priced differently.

We had to think of an abstraction above tokens. And it's also not just tokens. It's the token model and serving tier trade off.

Right? Because we can have priority tier processing. We can have asynchronous processing.

The cash rate could be different depending on who uses it when. Right? And so we wanted to, from the get go, commit to making sure that customers were getting the fair deal.

Not necessarily that we were making a ton of money off of it, but that customers were paying for what was reasonable. That's the fundamental of where we started. And also, we're selling enterprise SaaS.

So if we sell credit packs, then you get discounts. If you're an enterprise and you buy a certain amount of credit packs and things like that. So it also just helped the sales motion work a little bit easier.

So that's the answer on the abstraction of credits to dollars.

if there was any discussion. That that's not where the market is right now, number one. The second reason that we're not doing that is it ended up being complicated to figure out what was complicated or not.

So we at first, we're like, let's just charge on agent runs. And, you know, you went through all the different versions that ultimately just brought you back to a lot of complexity that mapped directly to token throughput. And so it it's also just simpler.

It's quite difficult to build those pricing systems. And I actually think that one of the biggest reasons we want had usage based pricing for this capability is we've had our core agent for a while with a model picker. And there were certain models or certain functionality that we had margins to maintain.

And if we wanted to ship this functionality, we couldn't afford it. It would bankrupt the company if we let for instance, like autofill or the database autofill feature will soon be agentic. That will be associated with usage based pricing because if every single autofill action was an agent running on Opus, on every single database cell, it would be billions of dollars.

Right? And so we had to find a way for the customers that wanted to do more and wanted to give us their money and pay more to find the outlet for them to do it that we didn't have to apply to the lower end of the curve. And also not all knowledge work is equal.

There's different points. A lot of the agent workflows here really saturate model capabilities. Like, you don't need a complicated model for it.

And so charging based on token usage. We couldn't just decide for you that you wanted your email client to be dumb or not. We want you to decide.

If you want to have Opus auto triage all of your emails, we will actually give you nudges in the product to rethink if that's the right choice. Because also not every user Understand. You'd be surprised in user interviews.

People will like, oh, I didn't know that. So now we actually have a little hover that tells you, like, if it's expensive or not. Yeah.

It's also slower. So the thing that's interesting is, like, people don't care about speeding custom agents. And so the incentive of Haiku being faster, people don't care when it's asynchronous.

And so we want to only provide the service of extra benefit that people want. And the best way to do that is to incentivize them because it's their own money. Must be confusing for people that are not familiar.

It's like, why is there no 5.3?

was heavily I think what's actually been hard for us is to convince people that auto is not just our cheapest, dumbest model, but actually the model that's best for the task that you wanna do.

And a lot of our job is actually figuring out auto because it's This is the agent lab. Every agent lab has an auto. Yeah.

And That's the job. Exactly. Because if you think about like I said, I come from Robinhood.

Like, you could spend a lot of time keeping up with the markets or you could have a auto investing. Right? And you can have an index fund Or you can have Robo advisors of fellow maker.

And so, like, at a certain point, we also can be robo advisors. And, like, we have a lot of people figuring out what model is best for the right task. And right now, we're not using auto as a margin maker.

We're just using it to reduce stress. It's not Opus. That's for sure.

Because a majority of the tasks people are doing aren't Opus level intelligence.

We're actually really interested in giving you the right tool for the job. A lot of the time, the right tool for the job is actually just writing code and not even using agent at all. So that's something that we're investing in a lot is imagine your agent can actually automate itself out of a job.

We would love if that were true. I feel very strongly about this because I don't necessarily feel like that's the skews that Frontier Labs give you. I feel like they are just getting more and more capable and more and more expensive, which is fantastic for the use cases of when people wanna do really complicated things on Notion.

What's difficult is like that market that I think right now is no man's land of where reasoning models were six months ago that the nano, haikus, etcetera haven't caught up to. Because now we're just paying more for those for like extra capability that we didn't necessarily need and so are our customers. And labs aren't necessarily incentivized right now with how few players they are to be meeting the market everywhere.

They just need to be the cheapest. They don't need to be at value that the customer wants. If no one's cheaper than them, then they're the cheapest and that's good enough.

And so we're doing a lot to make sure that we have the right optionality to switch between models and also invest in open source. Because the open source models actually are getting to be the place where reasoning models were three, four months ago. And that's what's filling that gap right now.

So you'll see we offer mini max. And we're collaborating a lot with different open source labs to think about Notion's last exam and how they can do better on these types of tasks so that we can offer them for that intelligence to price to latency trade off. Because in that triangle of intelligence price intelligence price and latency, excuse me, users get to choose where they are.

But right now, there's not the whole triangle isn't filled with models. Right? And the more that different models fill that triangle capability Everyone's clustered or everyone's clustered Haiku is not that much cheaper.

No one's really in the middle. Like, people really tend to cluster around too. This is really capable and it's really fast, but it's really expensive or whatever.

And so we just wanna make sure that triangle's filled.

I don't know if what do you mean by train your model? We don't know if we have no money to train a fund You go raise it. Yeah.

You can raise it. You can raise it. That's your job, Simon.

No. I don't think that needs to be our core competency.

that we're training for all these reasons, these tend to be, like, quite large institutions thinking about how to let people bring their own models. But those models have to function with, like Right. Understanding how to call our tools.

And that's where, again, having more public system prompt is, like, beneficial to Notion. Right? We want all models to plug into Notion as as well as they can.

That being said, like, of course, there are certain aspects of Notion where we do fine tune and do reinforce and fine tuning on our own capabilities. But that's not necessarily trained on user data. You don't need that that much data in the first place.

and I'm confident that they won't be done by the time I wake up. Really? Eight hours.

There is a I won't say which coding frontier lab, but there is a point where he had outlived, like, the thread length and context length that that coding agent provided. And I DM you DM'd them being like, hey, I need more. And our account rep DM'd me directly, and they're like, is Simon trying to prove string theory?

What is he doing? Yeah. I I had a single coding agent thread going for, I think it was, like, seventeen days.

other thing that reminded me about fine tuning that I think you and I have aligned on is that our tools change really frequently. And right now, we spend a lot of time rethinking and building tools for capability. And fine tuning a model to understand your tool.

Like, we don't have legal expertise or coding expertise. So if we were to fine tune a model, it would either be expertise about the enterprise and we have ZDR, new data retention offerings for those enterprises. So we'd have to really rethink how we structure if an enterprise wanted to opt into that.

Or it would be fine tuning and better capability on navigating our tools. That doesn't match with the velocity with which we create new tools. And so it actually really slow us down to have a model that was fine tuned on our tools because we'd have to retrain it and cut a new model every time we did that.

And that's not how we're set up right now, particularly with the way that we're changing our I guess we could fine tune a model to, like, search for tools. It's just the amount of time it takes to do that, ship it, have the right system. You're basically making a bet against a frontier capability, not serving that in the time it takes you to build it.

And that timeline hasn't happened for us yet. It hasn't been Yeah. It's just the wrong trade off, I think.

making really good tools, making a good harness, like verifying it works? The one place that we do invest more in model training now necessarily though is actually in retrieval. Because we're at a point right now in our business and enterprise or AI enabled plans where the search load and the search traffic, a majority of it's coming from agents, not humans.

And so for every query that's hitting our Elasticsearch or vector indices, they're not coming from humans. And the queries are structured differently and what's returned has a different requirement. Positional ranking matters less, but top k retrieval mode matters more.

Isn't top k a form of position? Of course, is. But when you're trading on, like, click through rate, it's really Yeah.

It matters much less. Is very different Yeah. Than it needs to be in the top 100.

Like the slope is just Yeah. Higher. It's a different optimization function for a retrieval model.

Similarly, what snippet you include matters more or less. So we are rethinking a lot of that functionality to work with how the agents like to write queries and how they wanna receive information. So we are doing like another kind of reinvestment into rethinking not only search for how do agents do searches versus how humans do searches, but we're also investing in like indexing different things now.

Because how are how do you index the setup generator for Notion Agent? It breaks our block model entirely where all blocks are nested in each other. Same with meeting notes.

I don't know why. Yeah. But the other thing is that I was just talking about this with a peer, like, how much is ranking important versus being able to do parallel exhaustive queries?

They're important? They're both important, but they're both two tools to the same user outcome or the same agent outcome. Mhmm.

Right? And that that's something that we're also rethinking a lot. Even on we just did an experiment on Notion ranking at this point.

For Notion retrieval, vector embeddings are less and less. Did you see that? Yeah.

so that you get Right. Mesmerized space. And so like the people that are working on ranking and retrieval are the same people working on what query generation is.

It's all one journey. We call it agentic find. And we're actually realizing, for instance, that it's less about selection.

Like, we don't spend a lot of time trying to optimize what vector embedding we use anymore. That was a period of time, but that's just not the right level of optimization.

Meeting notes was one of those things where at first we were nervous that we'd have to teach people a different way to work. And we were nervous that was a lot of user friction. I think one of the reasons why I mean, they're one of our biggest growth levers.

I think they're one of the most, like, in terms of virality of adoption and retention, quite strong. And so we've invested more and more as we did that. I think what's really powerful about it is, again, Notion is the system of record of where and how you work.

The way that I use meeting notes is every one on one meeting I have is meeting notes. When I do my performance review for myself, my self review, I say, primarily look at all my conversations with my manager and write up what I did this year. Right?

Because if I didn't talk about it in my one on one with my manager, it probably wasn't relevant for my performance review. So it also just adds a ton of signal on prioritization that's really helpful for a good system of record. That's really helpful for, like, our agent.

It's also caused a lot of scaling for search and for the agent. And it's just an explosion of content when you have transcripts like that. How we do compaction, a lot of that was triggered by meeting notes passed into context, things like that.

It's been a good impetus for us to think about longer form content when you think of it as like a priority primitive. But it's been one of the most powerful signals for our agent because it's Unsurprising. Right?

You're capturing a whole new thing. So it's like our own data like, we want users or they're creating their own data flywheel. Right?

Like, serves me to prefer Notion to put all my stuff because it has my other stuff. Totally. The way that, like, our teams run right now is there's a custom agent that does a pre read before stand up.

It looks through all of Slack and GitHub and just says it it creates a summary and it creates a meeting note. And it says, everyone do this preread. Then we just press play.

We have the meeting. We talk through the preread. We talk about what needs to happen next.

And then we have a custom agent integrated with our calendar and triggers that then files tasks for tomorrow or today based on what we spoke about and sends off Slack messages that we decided in the meeting needed to be follow ups. Like, our meetings are hands off keyboard.

It will be. When you look at it thinking when it's doing the summarization, it's saying, figuring out who Simon is. Most probable Simon is.

Yeah. And we also have people to people similarity cache and stuff like that in the attendees. Like, here's a way to generate a profile for each person and and use that.

Of course, can get it wrong, but the goal is for it not to get it wrong. Meeting Huts is just like the agent primitive packaged on top of a transcription primitive. Yeah.

Yeah. And then a vertical team. It's probably one of the only teams at Notion that's completely a vertical team around quality and product, like UX design, because it's still a Tiger team with a fantastic manager, Zach, that joined recently from Embra.

But Zach Chattar. Yeah. Yeah.

I chatted with him when he was talking about when he was working in Embra. Yeah. So he's managing that team now and thinking about it as data capture.

That's what MIDI notes is data capture. Get all the Reframing where MIDI notes are valuable as a data capture problem. And then working inside like, the summarization used to not be agentic.

Yeah. Now it is. Because it does all the things like figure out who the right Simon is.

And one day you can have a custom agent directly integrated in it that knows like what task database the meeting is referring to. And as you're having the meeting, perhaps update the tasks and things like that. There's a lot of that experience of where we do our work in meetings that we want to invest in making more seamless.

Yeah. Opening eyes doing hardware. Would you ever ship one of these?

Yeah. Put me on. But what This is meeting notes in person.

Yeah. Yeah. But I'd be excited about I'm excited about the product category in general for sure.

Yeah. I think it's it's a mechanism, and it it one of those needs to work really well with Notion. We would partner with whoever is building one of those.

I think They would be bought by Amazon. I don't know. I can't refer you.

And there's there's some wild companies doing, like, really cool things that come to our partnerships team that I like to sit in on the demos of wearables. I always like to sit in on the demos because I think they're Oh, okay. Pretty cool.

And all of them want to make sure not just Notion, but, like, you can imagine the ones that talk to you being able to do search and build context. So if you're entering, a conference, being able to do look at your CRM and do things like that, then you can utilize the Notion agent to do that. So we are in like the very beginnings of those partnerships.

I think what's unique about that particular technology is it goes against what I talked about with custom agents right now, which is the more simple it is, the harder it is to have, like, advanced controls over its capabilities. And so that would be a great investment for data capture, but not necessarily like our agent who's workflows. It's something with a different slice of the problem.

That's gonna be deeply personal. Like, your company is not gonna force you to wear a wrist wristband. I think It's good to hear that from me, from you.

Yeah. It will yeah. The CEO is gonna force everyone to wear a wristband.

The slice of the problem that that we care about is can the company have all the context of what everyone said at every single meeting? And then use that to Yeah. To derive value for themselves.

That kind of reminds me. I remember once you very strongly reminded me, our job is to not make the best harness for agentic work. Our job is to be the best place where people collaborate.

It's like our job isn't to build the best wearable to capture meeting notes. Our job is to build the best place where meeting notes live. Yeah.

So basically, you're saying everyone else can just pipe to you and that that's fine. Right? Yeah.

That's a reasonable thing. All I will say is that peep there's people walking around with Notion tattoos on them. They'll they'll wear Notion anything.

So just, I don't know, do a limited run. No. We have such understated swag that the idea like, our swag has so few Notion labels on it.

The idea that people have Notion tattoos is pretty antithesis to our design principles, so that's pretty funny.