Retrieval After RAG: Hybrid Search, Agents, and Database Design — Simon Hørup Eskildsen of Turbopuffer

I don't think I've said this publicly before, but I just called and was like, local Locky, like, if this doesn't have PMF by the end of the year, like, we'll just, like, return all the money to you. But it's just like, don't really Justin and I don't wanna work on this unless it's really working. So we wanna give it the best shot this year, and like, we're really gonna go for it.

We're gonna hire a bunch of people, and we're just gonna be honest with everyone. Like, when I don't know how to play a game, I just play with open cards. Lockheed was the only person that didn't that didn't freak out.

He was like, I've never heard anyone say that before.

Yeah. I was writing a post not that long ago about sort of the influences. So I grew up in Denmark.

Right? I left I left when I was 18 to go to Canada to to work at Shopify. And so I like, I've I would still say that I feel more Danish than than Canadian.

This is also the weird accent. I can't say t h because it this is like I don't you know, my wife is also Canadian. And I think I think like one of the things in in Denmark is just like there's just such a ruthless pragmatism, and there's also a big focus on just aesthetics.

Like, there are like very people really care about like where what things look like. And Canada has a lot of attributes. US has has a lot of attributes.

But I think there's been lots of great things to carry. I don't know what's in the water in Albus, though. And I don't know that I could be considered part of the Albus mafia mafia quite yet compared to the Oh, it's decode.

Phenomenal individuals we just mentioned. Is also a Danish Canadian.

which is maybe a bad word now in some circles. You could be a search engine. It's like, let let's just start there, and then we'll maybe run through the history of how you got to this point.

For sure. Yeah. So Turbo Puffer is, at this point in time, a search engine.

Right? We do full text search and we do vector search, and that's really what we're specialized in. If you're trying to do much more than that, then this might not be the right place yet, but Turbo Puffer is all about search.

The other way that I think about it is that we can take all of the world's knowledge, all of the exabytes and exabytes of data that there is, and we can use those tokens to train a model, but we can't compress all of that into a few terabytes of weights. Right? We can compress into a few terabytes of weights how to reason with the world, how to make sense of the knowledge, but we have to somehow connect it to something external that actually holds that like in full fidelity and truth.

And that's the thing that we intend to become. Right? That's like a very holier than thou kind of phrasing.

Right? But being the search engine for unstructured unstructured data is the focus of Turbo Puffer at this point in time. And let's break down.

is this search on my data? Is this like closer to Rag than to like a XR or like a public search thing? Like, how do how do you segment like the different types of search?

The way that I generally think about this is like, there's a lot of database companies. And I think if you want to build a really big database company, sort of you need a couple of ingredients to be in the air, which only happens roughly every fifteen years. You need a new workload.

You basically need the ambition that every single company on earth is gonna have data in your database multiple times. You look at a company like Oracle. Right?

You will like, I don't think you can find a company on Earth with a digital presence that it not doesn't somehow have some data in an Oracle database. Right? And I think at this point that's also true for Snowflake and Databricks, right?

Fifteen years later, even more than that. There's not a company on Earth that doesn't indirectly or directly is consuming Snowflake or Databricks or any of the big analytics databases. And I think we're in that kind of moment now, right?

I don't think you're gonna find a company over the next few years that doesn't directly or indirectly have all their data available for search and connected to AI. So you need that new workload. Like you need something to be happening where there's a new workload that causes that to happen.

And that new workload is connecting very large amounts of data to AI. The second thing you need, the second condition to build a big database company is that you need some new underlying change in the storage architecture that is not possible from the databases that have come before you. If you look at Snowflake and Databricks, right, commoditized like a massive fleet of HDDs, like that was not possible and it just wasn't in the air in the nineties, right?

So we just didn't build these systems. S3 and so on was not around. And I think the architecture that is now possible, that wasn't possible 15 ago, is to go all in on NVMe SSDs.

It requires a particular type of architecture for the database that is difficult to retrofit onto the databases that are already there, including the ones you just mentioned. The second thing is to call all in on object storage, more so than we could have done fifteen years ago. We don't have a consensus layer.

We don't really have anything. In fact, you could turn off all the servers that Turbo Puffer has and we would not lose any data because we are completely all in on object storage. And this means that our architecture is just so simple.

So that's the second condition, right? First being a new workload, that means that every company on earth, either indirectly or directly is using your database. Second being there's some new storage architecture that means that the the companies that have come before you can't do what you're doing.

I think the third thing you need to do to build a big database company is that over time you have to implement more or less every query plan on the data. What that means is that you can't just get stuck in like, this is the one thing that a database does. It has to be ever evolving.

Because when someone has data in the database, they over time expect to be able to ask it more or less every question. So you have to do that to get the storage architecture to the limit of what what it's capable of. Those are the three conditions.

I just wanted to get a little bit of like the motivation. Right? Like, so you left Shopify, you're like principal engineer, Infragay.

I was on the infrastructure team from the fairly fairly early days around 2013. At the time, it felt like it was growing so quickly and everything, all the metrics were, you know, doubling year on year. Compared to the what companies are contending with today, it's very cute growth.

I feel like my luck, some companies are seeing that month over month. Of course, Shopify has been compounding for a very long time now, But I spent a decade doing that, and the majority of that was just make sure the site is up today and make sure it's up a year from now. And a lot of that was really just the, you know, the Kardashians would drive very, very large amounts of of data to to Shopify as they were rotating through all the merge and building out their businesses, and we just needed to make sure we could handle that.

Right? And sometimes these were events with a million requests per second. And so, you know, we we had our own data centers back in the day, and we were moving to the cloud, and there was so much sharding work and all of that that we were doing.

So I spent a decade just scaling databases, because that's fundamentally what's the most difficult thing to scale about these sites. The database that was the most difficult for me to scale during that time, and that was the most aggravating to be on call for, was Elasticsearch. It was very, very difficult to deal with.

And I saw a lot of projects that were just being held back in their ambition by using it. And I mean Self hosted. Self hosted.

Because it's Yeah. And it's also just like 2015. Right?

So it's like a very particular vintage. Right? It's probably better at a lot of these things now.

It was difficult to contend with. And I'm just like, I just think about it. It's an inverted index.

It should be good at these kinds of queries and do all of this. And it was we we often couldn't get it to do exactly what we needed to do or basically get Lucene to do, like expose Lucene raw to to to what we needed to do. So that was like just something that we did on the side and just panic scaled when we needed to, but not a particular focus of mine.

So I left. And when I left, I wasn't sure exactly what I wanted to do. I mean, it spent like a decade inside of the same company.

I'd like grown up there. I started working there when I was 18. You only do rails.

Yeah. I mean, yeah, rails and And he's a rails guy. I love rails.

So good.

know there's a way to do it. I love it. I don't know that I would use it now, like given Cloud Code and Cursor and everything, but still, like if I'm just sitting down and writing a T signal code, that's how I think.

But anyway, I left and I wasn't I talked to a couple of companies and I was like, I don't I need to see a little bit more of the world here to know what I'm gonna focus on next. And so what I decided is I was gonna, I called it angel engineering where I just hopped around in my friend's companies in three months increments and just helped them out with something. And just vested a bit of equity and solved some interesting infrastructure problem.

So I worked with a bunch of companies at the time. Readwise was one of them. Replicate was one of them.

Causal, I don't know if you've tried this. It's like a spreadsheet engine, yeah, where you can do distribution. They sold recently.

We've been we used that in FP and A at Turbo Puffer. So bunch of companies like this, and it was super fun. And so when the ChatGPT moment happened, I was with with Readwise for a stint.

We were preparing for the reader launch, right, which is where you you queue articles and read them later. And I was just getting their postgres up to snuff, like, which basically boils down to tuning auto vacuum. So I was doing that, and then this happened, and we were like, oh, maybe we should build a little recommendation engine and some features to try to hook in the LLMs.

They were not that good yet, but it was clear there was something there. And so I built a small recommendation engine, just, okay, let's take the articles that you've recently read. Right?

Embed all the articles and then do recommendations. It was good enough that when I ran it on one of the cofounders of Rebwise, like, I found out that I got articles about about having a child. I'm like, oh my God, I didn't know that they were having a child.

Wasn't sure what to do with that information, but the recommendation engine was good enough that it was suggesting articles about that. And so there was recommendations and it actually worked really well. But this was a company that was spending maybe $5 a month in total on all their infrastructure.

And when I did the napkin math on running the embeddings of all the articles, putting them into a vector index, putting it in prod, it's gonna be like $30 a month. That just wasn't tenable, right? Like Readwise is a proudly bootstrapped company and paying $30 for infrastructure for one feature versus five, it just wasn't tenable.

So sort of in the bucket of this is useful, it's pretty good, but let us let's return to it when the cost comes down. Did you say it grows by feature? So for five to 30 is by the number of like, what's the what's the scaling factor?

It scales by the number of articles that you embed. It does. But what I meant by that is like $5 for like all of the other, like the Heroku Dinos, Postgres, like all the other end this Then the storage is 30?

Yeah. And then like $30 for one feature. Right?

Which is like what other articles are related to this one? So it was just too much to power everything. Their budget would've been maybe a few thousand dollars, which still would've been a lot.

And so we put it in the bucket of, okay, we're gonna do that later. We'll wait for the cost to come down. And that haunted me.

I couldn't stop thinking about it. I was like, okay, there's clearly some latent demand here. If the cost had been a tenth, we would've shipped it.

And this was really the only data point that I had. I didn't go out and talk to anyone else. So I started reading.

I couldn't help myself. I didn't know what a vector index is. I barely do about how to generate the vectors.

There is a lot of hype about This is early twenty twenty three. There was a lot of hype about vector databases. They're raising a lot of money.

Was like, I really didn't know anything about it. It's you know, trying these little models, fine tuning them. Like, I was just trying to get sort of a lay of the land.

So I just sat down. I have this GitHub repository called napkin math. And on napkin math, there's just rows of like, oh, this is how much bandwidth.

Like, this is how many you know, you can do 25 gigabytes per second on average to DRAM. You can do five gigabytes per second of rights to an SSD, blah blah blah. All of these numbers.

Right? And S3, how many you could do How much bandwidth can you drive per connection? I was just sitting down.

Was like, why hasn't anyone build a database where you just put everything on object storage and then you puff it into NVMe when you use the data, and you puff it into DRAM if you're querying it a lot? It's just like, this seems fairly obvious, and the only real downside to that is that if you go all in on optic storage, every write will take couple hundred milliseconds of latency. But from there, it's really all upside.

Right? You do the first query, it takes half a second, and it sort of occurred to me as like, well, the architecture is really good for that, is really good for OPIC storage, that's really good for NVMe SSD. Well, you just couldn't have done that ten years ago, back to what we were talking about before.

You really have to build a database where you have as few round trips as possible, right? This is how CPUs work today, it's how NVMe SSDs work, it's how S3 works that you wanna have a very large amount of outstanding requests, right? Basically go to S3, do that thousand requests to ask for data in one round trip, wait for that, make a new decision, do it again, and try to do that maybe a maximum of three times.

But no databases were designed that way. With NVMe SSDs, you can drive within a very low multiple of DRAM bandwidth if you use it that way. And same with S3, right?

You can fully max out the network card, which generally is not maxed out. You get very, very good bandwidth. But no one had built a database like that.

So I was like, okay, well, can't you just take all the vectors and plot them in the proverbial coordinate system, get the clusters, put a file on S3 called clusters. Json, and then put another file for every cluster, cluster1. Json, cluster2.

Json, like it's two round trips, Right? So you get the clusters, you find the closest clusters, and then you download the cluster files, like the closest end, and you could do this in two round trips. You ran Nearest Neighbors locally.

Yes. Yes. And you would build this file, right?

Just like ultra simplistic, but it's not a far shot from what the first version of TurboPuffer was. Why hasn't anyone done that? In that moment, from a workload perspective, you're thinking this is gonna be like a read heavy thing because you're doing recommended.

Like, is the fact that like writes are so expensive now, oh, with AI, you're actually not writing that much? At that point, I hadn't really thought too much about Well, no, actually it was always clear to me that there was gonna be a lot of writes because at Shopify, the search clusters were doing, you know, I don't know, tens or hundreds of QPS, right? Because you just have to have a human sit and type in.

But we did, you know, I don't know how many updates there were per second. I'm sure it was in the millions, right, into the cluster. So I always knew there was like a 10 to 100 ratio on the read write.

In the read wise use case, it's even even in the read wise use case, there'd probably be a lot fewer reads than writes. Right? There's just a lot of churn on the amount of stuff that was going through versus the amount of queries.

I wasn't thinking too much about that. I was mostly just thinking about what's the fundamentally cheapest way to build a database in the cloud today using the primitives that you have available. And this is it, right?

Just, now you have one machine and let's say you have a terabyte of data in S3, you pay the $200 a month for that, and then maybe 5 to 10 percent of that data needs to be in NVMe SSDs and less than that in DRAM. Well, you're paying very, very little to inflate the data.

just build a completely new database. I don't know if we were the first. Like it was very much It was I mean, I hadn't I just looked at the napkin math and I was like, this seems really obvious.

So I'm sure like a 100 people came up with it at the same time. Like the light bulb and every invention ever. Right?

It was just in the air. I think Neon Neon was was first to it and they're trying they retrofitted it onto Postgres. Right?

And then they built this whole architecture where you have you have it in memory and then you sort of like, you know, M map back to S3. And I think that was very novel at the time to do it for OLTP. But I hadn't seen a database that was truly all in, right?

Not retrofitting it, the database built purely for this, no consensus layer, even using compare and swap on OPIC stories to do consensus. I hadn't seen anyone go that all in. And I I mean, there there I'm sure there was someone that did that before us.

I don't know. I was just looking at the napkin math. And when you say consensus layer, are you strongly relying on s three strong consistency?

You are. Okay. So that is your consistency layer?

It it is the consistency layer. And I think also, like, this is something that most people don't realize, but s three only became consistent in December 2020. I remember this coming out during COVID, and, like, people were like, oh, like it was like was it just like a free upgrade.

Yeah. They they were just they just announced it. We saw consistency guys.

They're like, k. Cool. I'm sure that they just they probably had it in prod for a while.

They're just like, it's done. Right? And people are like, okay.

Cool. But that's a big moment. Right?

Like, NVMe SSDs were also not in the cloud until around 2017. Right? So you just sort of had like twenty seventeen NVMe SSDs, and people were like, okay, cool.

There's like one SKU that does this. Whatever. Right?

It takes a few years. And then the second thing is like S3 becomes consistent in 2020. So now it means you don't have to have this like big FoundationDB or like Zookeeper or whatever sitting there contending with the keys, which is how, that's what Snowflake and others have to So do much with for gone.

Exactly, just gone, right? And so just push to the, whatever, how many hundreds of people they have working on S3 solved. And then compare and swap was not in S3 at this point in time.

By the way, I don't know what that is. So maybe you wanna explain this. Yes.

Yes. So what compare and swap is, is basically you can imagine that if you have a database, it might be really nice to have a file called metadata. Json.

And metadata dot JSON could say things like, hey, these keys are here and this file means that, and there's lots of metadata that you have to operate in the database. Right? But that's the simplest way to do it.

So now you have my you might have a lot of servers that wanna change the metadata. They might have written that file and want the metadata to contain that file. You have a 100 nodes that are trying to contend with this metadata.

Json. Well, what compare and swap allows you to do is basically just you download the file, you make the modifications, and then you write it only if it hasn't changed while you did the modification. And if not, you retry, right?

You just have this retry loops. Now you can imagine if you have a 100 nodes doing that, it's gonna be really slow, but it will converge over time. That primitive was not available in S3.

It wasn't available in S3 until late twenty twenty four, but it was available in GCP. The real story of this is certainly not that I sat down and big brained it. I was like, okay, we're gonna start on GCS.

S3 is gonna get it later. Like, it was really not that. We started We got really lucky.

We started on GCP, and we started on GCP because Shopify ran on GCP. And so that was the platform I was most available with. And I knew the Canadian team there because I'd worked with them at Shopify.

So And it was natural for us to start there. And so when we started building the database, we're like, oh yeah, we have to build a We really thought we had to build a consensus layer, like have a zookeeper or something to do this. But then we discovered the compare and swap.

Was like, oh, we could kick the can. We'll just do metadata. Json and just, it's fine.

It's probably fine. And we just kept kicking the can until we had very, very strong conviction in the idea. And then we kind of just hinged the company on the fact that S3 probably was gonna get this.

It started getting really painful in mid twenty twenty four because we were closing deals with Notion actually that was running in AWS. And we're like, Trust us. You really want us to run this in GCP?

And they're like, No, I don't know about that. We're running everything in AWS. And the latency across the clouds were so big.

And we had so much conviction that we bought dark fiber between the AWS regions in Oregon, like in the inter exchange. And GCP is like, we've never seen a startup, like what's going on here? And we're just like, no, we don't wanna do this.

We were tuning like TCP, Windows, like everything to get the latency down, because we had so high conviction in not doing like a metadata layer on S3. So those were the three conditions. Right?

Compare and swap to do metadata, which wasn't in s three until late twenty twenty four, s three being consistent, which didn't happen until 12/20/2020, and then NVMe SSDs, which didn't land in the cloud until 2017.

I mean, it's very common when you're a big company. Right? You're like connecting your own, like, data center or whatever, but it was uniquely just a pain with Notion because if you're buying in Ashburn, Virginia, US East, the GCP and AWS data centers are within a millisecond on each other on the public exchanges.

But in Oregon, uniquely, the GCP data center sits a couple 100 kilometers East Of Portland, and the AWS region sits in Portland, but the network exchange they go through is through Seattle. So it's like a full fourteen milliseconds or something like that. And so anyway, yeah, it's it's so we were like, okay.

We can't we have to go through an exchange in Portland. Yeah. And you'd rather do this than, like, run your zookeeper and Yes.

Way rather. It doesn't have state. I don't want state in two systems.

And I think all of that is just informed by Justine, my co founder, and I had just been on call for so long. And the worst outages are the ones where you have state in multiple places that's not syncing up. So it really came from a a like, just a a very pure source of pain of just imagining what we would be okay being woken up at 3AM about, and having something in Zookeeper was not one of them.

We're we're talking to like a Notion or something. Do they care? Or do they just They just cared about latency.

Latency cost. That's it. They just cared about latency.

Right? And we just absorb the cost. We're just like, we have high conviction in this.

At some point, we can move them to AWS. Right? And so we just We'll buy the fiber.

It doesn't matter. Right? And it's like 5,000 And we Usually when you buy fiber, you buy like multiple lines and we're like, we can only afford one.

This this workload still runs on GCP for what it's worth. Right? Because it's so it was just it was so reliable.

So it was never about moving off GCP. It was just about honestly, it was just about giving Notion the latency that they deserved. Right?

And we didn't want them to have to care about any of this. We also They were like, oh, egress is gonna be bad. It was like, okay, screw it.

Like, we're just gonna Like, v VPC peer with you in AWS. We'll eat the cost. Yeah.

Whatever needs to be done. And what were the actual workloads? Because I think when you think about AI, it's like fourteen milliseconds.

It's like really It doesn't really matter in the scheme of like a model generation. Yeah. We were told the latency, right, that we had to beat.

Right? We're just looking at the traces, right? And then sort of like hand draw, like, you know, like looking at the trace and then thinking, what are the other extensions of the trace?

And there's a lot more to it because it's also, if you have fourteen versus seven milliseconds, you can fit another round trip. So we had to tune TCP to try to send as much data in every round trip, pre warm all the connections, and there was there's a lot of things that compound from having these kinds of round trips. But in the grand scheme, it was just like, well, we have to beat the latency of whatever we're up against.

Which is like they I mean, Notion is a database company. They could have done this themselves.

at at Notion. And they were just trying to make sure that the per user cost matched the economics that they needed. It's like the way I think about it is like I have to earn a return on whatever the clouds charge me, and then my customers have to earn a return on that.

And it's like very simple, right? And so there has to be gross margin all the way up, and that's how you build the product. And so then our customers have to make the right set of trade off that Turbo Puffer makes.

And if they're happy with that, that's great. Do you feel like you're competing with build internally versus buy or buy versus buy? Yeah.

So, sorry, this was all to build up to your question. So one of the Notion engineers told me that they'd sat and probably on a napkin, like drawn out, like, why hasn't anyone built this? And then they saw Turbo Puffer and was like, well, literally that.

So And I think AI has also changed the buy versus build equation in terms of, it's not really about, can we build it? It's about do we have time to build it? And I think they felt like, okay, if this is a team that can do that and they they feel enough of like an extension of our team, well, then we can go a lot faster, which would be very, very good for them.

And I mean, they put us through the through the test. Right?

the day after we launched, which was just, you know, I'd worked the whole summer on on the first version. Justine wasn't part of it yet because I just I didn't tell anyone that summer that I was working on this. I was just locked in on building it because it's very easy otherwise to confuse talking about something to actually doing it.

And so it's just like, I'm not gonna do that. I'm just gonna do the thing. I launched it.

And at this point, Turbo Puffer is like a Rust binary running on a single eight core machine in a TMux instance. And me deploying it was like looking at the request log and then command c ing it or control c ing it to just like, okay, there's no request. Let's upgrade the binary.

It was literally the scrappiest thing you could imagine. It was on purpose because just like at Shopify, we did that all the time. We ran things in TMux all the time to begin with before something had like at least the inkling of PMF.

And I like, okay, is anyone gonna hear about this? And one of the Cursor co founders, Arvid, reached out and he just you know, the the Cursor team are like all IOI, IMO, like contenders. Right?

So they just speak in bullet points and and facts. So it's like this amazing email exchange just of this is how many QPS we have. This is what we're paying.

This is where we're going, blah blah blah. So we're just conversing in bullet points. I tried to get a call with them a few times, but they were so they were like really riding the PMF bowl here just like late twenty twenty three.

And one time Swally emails me at like five no. What was it? Like 4AM Pacific time saying like, hey, are you open for a call now?

And I'm on the East Coast and I it was like 7AM. Was like, yeah, great. Sure.

Whatever. And we just started talking and something then I didn't know anything about sales. It would something that just compelled me.

I have to go see this team. Like there's something here. So I I went to San Francisco and I went to their office.

And the way that I remember it is that Postgres was down when I showed up at the office. Did Swale tell you this? No.

No. Okay. So Postgres was down.

And so it's like they were distracted with that. And I was trying my best to see if I could help in any way. Like I knew a little bit about databases back to tuning auto vacuum.

It's like, I think you have to tune auto vacuum, Saleh. And so we talked about that. And then that evening just talked about like, would it look like?

What would it look like to work with us? And I just said, look, like we're all in. Like we will just do whatever you tell us.

Right? They migrated everything over the next like week or two. And we reduced our costs by 95%, which I think like kind of fixed their per user economics.

And it solved a lot of other things. And we were just Justine this is also when I asked Justine to come on as my co founder. She was the best engineer that I ever worked with at Shopify.

She lived two blocks away, and we were just, okay, we're just gonna get this done. And we did. And so we helped them migrate, and we just worked like hell over the next, like, month or two to make sure that we were never an issue.

they will embed the entire code base. Right? So they will chunk it up in whatever they would they do.

They have their own embedding model, which they've been public about. And they find that on their evals, there's one of their evals where it's like a 25% improvement on a very particular workload. They have a bunch of blog posts about it.

I think it works best on larger code basis, but they've trained their own embedding model to do this. And so you'll see it. If you use the cursor agent, it will do searches.

And they've also been public around how they've I think they post trained their model to be very good at semantic search as well. And that's that's how they use it. And so it's very good at, like, can you find me on the code that's similar to this or code that does this and just in in disc queries?

They also use grep to supplement it. Yeah. Of course.

It's been a big topic of discussion. Like, is Rag dead because grep? You know?

See demand. And so, I I like case studies. I don't like just doing like thought pieces on this is where it's going and trying to be all macroeconomic about AI that has turned out to be a giant waste of time because no one can really predict any of this.

So I just collect case studies. And I mean, Cursor has done a great job talking about what they're doing, and I hope some of the other coding labs that use Turbo Puffer will do the same. But it does seem to make a difference for particular queries.

I mean, we can also do text. We can also do regex. But I should also say that cursor's like security posture into Turbo Puffer is exceptional.

Right? They have their own embedding model, which makes it very difficult to reverse engineer. They obfuscate the file paths.

They like, you it's very difficult to learn anything about a code base by looking at it. And the other thing they do too is that for their customers, they encrypt it with their encryption keys in TurboPuffer's bucket.

that Swale at Cursor talks about it, which is I'm gonna butcher it here. And I'm a database scalability person. I don't know anything about training models other than what the internet tells me.

The way he describes it is that this is just like cache compute, right? It's like you have a point in time where you're looking at some particular context and focused on some chunk and you say, this is the layer of the neural net at this point in time. That seems fundamentally really useful to do cache compute like that.

And how the value of that will change over time, I'm not sure, but there seems to be a lot of value in that.

Yeah. What are maybe some of the new types of workloads or like changes you've had to make to your architecture for it? I think you're right.

When I think of Rag, I think of, Hey, there's an 8,000 token context window and you better make it count. And search was a way to do that. Now, yeah, everything is moving towards just let the agent do its thing.

And so back to the thing before, right? The LLM is very good at reasoning with the data. And so we're just the tool call.

And that's increasingly what we see our customers doing. What we're seeing more demand from our customers now is to do a lot of concurrency. Like Notion does a ridiculous amount of queries in every round trip, just because they can't.

And I'm also now when I use the cursor agent, I also see them doing more concurrency than I've ever seen before. So a bit similar to how we designed the database drive as much concurrency in every round trip as possible. That's also what the agents are doing.

So that's new. It means just an enormous amount of queries all at once to the dataset while it's warm in as few turns as possible. Can I clarify one thing on that?

Yes. Is it Are they batching multiple users or one user is driving multiple cookies? One user driving multiple One agent driving there.

Parallel searching a bunch of things. Exactly. Yeah.

And I think that's probably also where the hybrid comes in, where that's another way to diversify. It's a completely different way to do the search. That's a big change.

So before it was really just one call and then the LLM took however many seconds to return, but now we just see an enormous amount of queries. So we just see more queries. So we've like tried to reduce query.

We've reduced query pricing. This is probably the first time actually I'm saying that, but the query pricing is being reduced five X. And we'll probably try to reduce it even more to accommodate some of these workloads of just doing very large amounts of queries.

That's one thing that's changed. I think the write ratio is still very high. There's still an enormous amount of writes per read, but we're starting probably to see that change if people really lean into this pattern.

for search engines before Turbo Puffer was very serverful, right? It was like, here's the VM, here's the per hour cost, right? Great.

And I just sat down with like a piece of paper and said like, if Turbo Puffer is like really good, this is probably what it would cost with a little bit of margin. And that was the first pricing of Turbo Puffer. And I just sat down and I was like, Okay, this is probably the storage amp or whatever on a piece of paper.

Was very bad price and I got it wrong. Oh. Well, I didn't get it wrong, but Turbo Puffer wasn't at the first principle pricing, right?

So when Cursor came on Turbo Puffer, it was like, I didn't know any VCs. I didn't know anything about raising money or anything like that. I just saw that my GCP bill was was high was a lot higher than the cursor bill.

So Justine and I was just like, well, we have to optimize it. And I mean, to the chagrin now of of it the VCs, it now means that we're profitable because we've had so much pricing pressure in the beginning because it was running on my credit card. And Justine and I had spent like tens of thousands of dollars on compute bills and spinning off the company and very bad Canadian lawyers things to get all of this done because we just we didn't know.

Right? If you're steeped in San Francisco, you just know. Okay.

You go out and raise a pre seed round. I never heard the word pre seed at this point in time. When you had Kershaw, had Notion, had no funding?

With Kershaw, had no funding. Yeah. By the time we had Notion, Locky was here.

Yeah. So it was really just, we vibe priced it 100% from first principles, but it wasn't, it was not performing at first principles. So we just did everything we could to optimize it in the beginning for that so that at least we could have like a 5% margin or something so I wasn't freaking out because Cursor's bill was also going like this as they were growing.

And so my liability and my credit limit was like actively calling my bank. Was like, I need a bigger credit. Anyway, that was the beginning.

But the pricing was, yeah, storage, rights, and quarry. And the pricing we have today is basically just that pricing with duct tape and spit to try to approach you know, like a margin on the physical underlying hardware. And we're doing this year, you're gonna see more and more pricing changes from us.

Yeah. And like is how much does stuff like VPC peering matter because you're working in AWS land where egress is charged and all that, you know? We probably don't like, we have like an enterprise plan that just has like a base fee because we haven't had time to figure out SKU pricing for all of this.

But I mean, yeah, you can run Turbo Puffer either in SaaS, right? That's what Cursor does. You can run it in a single tenant cluster, so it's just you.

That's what Notion does.

Turbo Puffer hired like, I don't know what what number this was, but we had a full time CFO. It was like the twelfth hire or something at Turbo Puffer. I think I hear a lot of company I don't know how they do it.

Like, they have a 100 employees and not a CFO. It's like Having a CFO is like You run out of business, man. Like, you know?

So good. Yeah. Like, Money Mike, like, he just, you know, just handles the money and a lot of the business stuff.

And so he came in and just helped with a lot of the operational side of the business. So, like, COO, CFO, like, somewhere in between. Just a quick mention of Lucky, just because I'm curious.

I've met Lucky and, like, he's obviously a very good investor now on physical intelligence.

This is an excellent question. No one's asked me this. Why Lockheed?

Because there was a couple of people that we were talking to at the time. And when we were raising, we were almost a little we were like a bit distressed because one of our peers had just launched something that was very similar to Turbo Puffer. And someone just gave me the advice at the time of just choose the person where you just feel like you can just pick up the phone and not prepare anything and just be completely honest.

And I don't think I've said this publicly before, but I just called and was like, local Lockheed, like, if this doesn't have PMF by the end of the year, like, we'll just like return all the money to you. But it's just like, I don't really Justine and I don't wanna work on this unless it's really working. So we wanna give it the best shot this year and like, we're really gonna go for it.

We're gonna hire a bunch of people and we're just gonna be honest with everyone. Like, when I don't know how to play a game, I just play with open cards. And Lockheed was the only person that didn't freak out.

He was like, I've never heard anyone say that before. As I said, I didn't even know what a seed or pre seed round was, like before probably even at this time. So I was just, like, very honest with him.

And I asked him, like, Lockheed, have you ever have have you ever invested in database company? Was just like, no. And at the time, was like, am I dumb?

Like but I think there was something that just, like, really drew me to Lockheed. He is so authentic, so honest, and, like and there's something just, like I just felt like I could just play like, just say everything openly. And that was that was I think that that was, like, a perfect match at the time And and and honestly still is.

He was just like, okay. That's great. This is like the most honest, ridiculous thing I've ever heard anyone say to me.

But like that like that Why is it ridiculous to say competitor launch, this may not work out? It was more just like, if this doesn't work out, I'm gonna close-up shop by the end of the year. Right?

Like it was I don't know. Maybe it's common. I I don't know.

He told me it was uncommon. Don't know. That's why we chose him.

And he'd been phenomenal. The other people we were talking at the time were database experts. Like, knew a lot about databases and Lockheed didn't.

This turned out to be a phenomenal asset. Justine and I know a lot about databases. The people that we hire know a lot about databases.

What we needed was just someone who didn't know a lot about databases, didn't pretend to know a lot about databases, and just wanted to help us with candidates and customers. And he did. And I have a list of the investors that I have a relationship with.

And Lockheed has just performed excellent in the number of sub bullets of what we can attribute back to him. Just absolutely incredible. And when people talk about like no ego and just the best thing for the founder, I like I don't think that anyone like, even my lawyer is like, yeah, Lockheed is like the most friendly person you will find.

Yeah. So joining versus starting. I never thought that I would be a founder.

I would start with it. Like, Turbo Puffer started as a blog post, and then it became a project, and then sort of almost accidentally became a company. And now it feels like it's it's like becoming a bigger company.

That was never the intention. The intentions were very pure. It's just like, why hasn't anyone done this?

And it's like, I wanna be the like I wanna be the first person to do it. I think some founders have this like, I could never work for anyone else. I I really don't feel that way.

Like, it's just like, I wanna see this happen, and I wanna see it happen with some people that I really enjoy working with, and I wanna have fun doing it. And this has all felt very natural on that on that sense. So it was never at like join versus versus versus found.

It was just this found me at the right moment.

was that on your mind at all? Or Before taking the the small check from Loki, I did have, like, a hard, like, look at myself in the mirror of like, okay, do I really wanna do this? And because if I take the money, I really have to do it.

Right? And so the way I almost think about it is like you kinda need to have like, kinda need to be like fucked up enough to wanna go all the way. And that was the conversation where I was like, okay, this is gonna be part of my life's journey to build this company and do it in the best way that I possibly can.

Because if I ask people to join me, ask people to get on the cap table, then I have an ultimate responsibility to give it everything. And I don't I think some people it doesn't occur to me that everyone takes it that seriously. And maybe I take it too seriously.

I don't know. But that was a very intentional moment. And so then it was very clear, okay, I'm gonna do this, and I'm gonna give it everything.

to talk about candidates and talk about how we wanted to build the company. And like everyone else is like, we want a talent dense company. And I think that's almost become trite at this point.

What I credit the Cursor founders a lot with is that they just arrive there from first principles of like, we just need a talent dense team. And I think I've seen some teams that weren't talent dense and like seen the counterfactual run, which if you've run and been in a large company, will just see that. It's logically will happen at a large company.

And so that was super important to me and Justine, and it's very difficult to maintain, and so we just needed We needed wording for it. And so I have a document called Traits of the P 99 Engineer. And it's a bullet point list.

And I look at that list after every single interview that I do and in every single recap that we do. And every recap we end with, I end with some version of I'm gonna reject this candidate completely irregardless of what the discourse was, because I wanna see people fight for this person. Because the default should not be, we're gonna hire this person.

The default should be, we're definitely not hiring this person.

let's do it. Right? And it doesn't have to be absolutely everyone.

Right? And like the interviews are always the sign that you're checking for different attributes. And if someone is like knocking it out of the park in every single attribute, that's fairly rare.

But that's really important. And so the traits of the P99 engineer, there's lots of There's also the traits of the P99 engineer and the quadruple nine engineer. This is like it's a long list.

Okay. I'll give you some samples, right, of what we look for. I think that the P99 engineer has some history of having bent their trajectory or something to their will.

Some moment where they just made the computer do what it needed to do. There's something like that. And it will occur to them at some point in their career and hopefully multiple times.

Right? Give me an example of one of your engineers that like I'll give an engine So we launched this thing called ANN V3. We're working on V4 and V5 right now, but ANN V3 can search a 100,000,000,000 vectors with a P50 of around forty milliseconds and a p 99 of two hundred milliseconds.

Maybe other people have done this. I'm sure Google and others have done this, but we haven't seen anyone, at least not in a public consumable SaaS that can do this. And that was an engineer, the chief architect of TurboPuffer, Nathan, who more or less just bent this the software was not capable of this, and he just made it capable for a very particular workload in six to eight week period with the help of a lot of the team.

There's numerous of examples of that, like, at at Turbo Puffer, but that's like really bending the software and x 86 to your will. It was incredible to watch. You wanna see some moments like that.

Isn't that triple nine?

like this is too high for a 99. Nathan Nathan is Nathan is like yeah. There's a lot of nines after that p.

So I think that's one trait. I think another trait is that the P99 spends a lot of time looking at maps. Generally, it's their preferred UX.

They just love looking at maps. You ever seen someone who just sits on their phone and just scrolls around on a map? Or did you not look at maps a lot?

You guys don't look at maps? I guess I'm not feeling that. I don't know, buddy.

You just disqual what about trains? Do you like trains?

love looking at maps. Like, it's like my preferred UX and just like, I, you know, I like lots of You get it of, like, random places? So, like, you know Yes.

Okay. There you go.

and you like studying a thing. The origin of this was that at some point, I read an interview with some IOI gold medalist. Uh-huh.

And it's like, what do you do in your spare time? It's just like, I like looking at maps. And I was like, I feel so seen.

Like, I just love like, scrolling out. Was like, oh, Canada is so big. Where's Baffin Island?

I don't know. I love it. Yeah.

Anyway, so the traits of p 99 p 99 is obsessive. Right? Like, there's just like you'll you'll find traits of that.

We do an interview at at at at Turbo Puffer or, like, multiple interviews that just try to screen for some of these things.

The P99 is very clear about, hey, Turbo Puffer, you can't run a high transaction workload on Turbo Puffer. It's like the right latency is a hundred milliseconds. That's a clear trade off.

I think the p 99 is very good at articulating the trade offs in every decision, which is exactly what the map is in your case. Right? Yeah.

It's yeah. My my my world. My world.

if you sit down and do the napkin math, right, where you're just like, okay, like, if I have a 100 machines, they have this many terabytes of disk, they have this bandwidth, whatever. Right? And you sit down and you just do the, like, high school napkin math on is how many QPS we should be able to drive to it.

Similar to how I did the Vibe pricing, right? If you can sit down and do that and then you observe the real system and you see, oh, we're off by like 10x. Bending trajectory to your will is like just making the software get closer and closer to that first principle line.

The P99 might even be able to cross the line by finding even more optimizations than from first principle. So bending the software to your will is about that, right? Like, a hundred millisecond p 99 to to s three, I mean, now you're talking like someone really high agency that like goes to Seattle, finds the s three team, and it's like, how are we gonna make this 10?

You know? Like, it's that that that's not quite what we talk about. Right?

But yeah. What's the future Turbo Puffer? Turbo Puffer started out act one of Turbo Puffer was vector search.

That was all we did to begin with. Act two of Turbo Puffer is is and was full text search. TurboPuffer today has a fairly start of the state of the art full text search engine.

We beat Lucene on some queries. In particular, very long queries that we've optimized for because those are the text search queries we see today. They're generated by LLMs or augmented by LLMs, and we see them on web scale datasets.

Right? Like someone searching for a very long text string on all of Common Crawl. We beat Lucene on some of those benchmarks, And we expect to continue to beat Lucene on more and more queries.

That's the performance and scale. TurboFuffer does phenomenally now at full text search performance and scale. What we work on now is more and more features for full text search.

People expect a lot of features with full text search. And full text search is still very valuable, right? If you go in and you press command K and you search for SI, an embedding based search might be like, oh, this is something agreeable because that's C, that's yes in Spanish, right?

that's the prefix of maybe a document of like, you know, these are all the reasons I hate Simon. Right? Like, this this is like that's a completely different So that augmentation to like how the human brain works on mapping like data to user is very important, but it's a lot of features.

That feature grind is what we're firmly on. And you will see us just adding to the change log every month, just more and more full text search features. So we're fully compatible.

And we're seeing people move from some of the traditional search engine onto Turbo Puffer for that. That's a big focus of Turbo Puffer this year. The other the other focus of of Turbo Puffer this year is just on scale.

We're seeing more and more companies that wanna search basically Common Crawl level types of datasets, both internally accompanied and externally time, at like, Corey, like a 100,000,000,000 vectors or a 100,000,000,000 documents at once. This is tricky, and we wanna make it cheaper, and we wanna make it faster. That's a big focus for TurboPuffer this year.

That's, you know, we just released ANN v three, which we talked about before. We're working on ANN v v four, and we also have planned what we're gonna do with ANN v five. Right?

And then on full text search, we're working on a lot of these features will be like FTS v three, but it will all roll out incrementally. Those are some of the really big features. And then the other thing is our dashboard.

Have any of you ever logged into the TurboHofer dashboard? There's not very much there. It almost looks like if a founder two years ago just sat down and wrote enough dashboard that there was at least something there, and then other people just sort of added stuff on for the next two like the the following two years, and then at some point SSO and other things to just catch up.

And it may or may not be what happened. But adding like I want PHP my admin back. Do you guys remember?

It was so good. Right?

just say that if you wanna build a big database company, the database over time has to implement more or less every query plan. Because when you have your data in a database, you expect it to over time, not just search, but also, Hey, I wanna aggregate this column. I wanna join this data, all of that.

But when you're a startup, your only moat is really just focus. So you have to lay out this apps, and you have to not get overeager. And I think we've seen some of our peers get very overeager and overextend themselves.

And what I keep telling the team, I was just having breakfast this morning with our CTO and chief architect, and we were talking about like, what we're most likely to regret at the end of the year is having tried to do too much. And so act three candidates could be a bunch of simpler OLAP queries. It could be lending ourselves a little bit more into see some people who wanna do traces and logging and things like that.

Some very simple use cases. Could be that. It could be maybe some time series.

Some people are trying to do that. There's lots of different things that you can do with TurboPuffer. But for now, if you're trying to do not search on TurboPuffer as the primary use case, you probably shouldn't.

But we see some customers that are like, oh, at some point, Cursor moved like 20 terabytes of Postgres data into TurboPuffer because it's like, it's die it's there. It works. And these particular query plans we know work well, and so they just moved it all to defer sharding.

So we look for patterns like that in what future acts of Turbo Puffer are going to be before firmly doubling down on them. But we wouldn't if today, if you're using TurboPuffer, it should be because search is very important to you. And then we might do a lot of auxiliary queries to that, but that should not be the main reason to go to TurboPuffer at this point in time.

Yeah. You didn't mention one thing I was looking for was graph type queries, like graph database, graph queries. Can you basically trivially replicate this with what you already have?

We see some people doing that. Right. Because you have parallel queries and it's it's the same thing.

Exactly. So we see some people doing that. Right?

Like at the under like, TerraBuffer is just a k v. Right? And then we expose things on top of it.

So we are seeing people do that. And I think our roadmap is very much just the database that connects AI to a very large amount of data is what the path is to do that in the right order, which is what a good startup is around. What is the order to do things in?

Our customers are p 99, and they will tell us what they care most about next. And so some of them are doing graphs now, and if they need more graph database features, they'll be banking on our door, and we'll prioritize accordingly. Tea.

Alright. Give us the tea. This you you you kindly gifted us your favorite tea.

caffeine, I think. And especially when I'm on a trip like this to San Francisco, I consume a lot of caffeine, but this is my preferred caffeine. It's this green tea.

I have an air table with 200 teas that I've tried over time over the past like fifteen years, and this one is my favorite. Now, you drink a tea, there's different there's like six different types of tea. I like green tea.

In particular, I generally prefer Chinese green tea, and I don't really like Japanese green tea. But this little prefecture somewhere in Japan has specialized in like, they're like Japanese, but doing it the Chinese way, and it's just phenomenal. But then the interesting thing about the tea world is that all of the different like, you can find this particular tea.

There's probably, you know, hundreds of places that sell it, but they all go to a different family. Right? On whatever mountain that they have these like Camellia sinensis bush bushes on.

And this woman, Japanese woman in Toronto from the green tea shop, I don't know. She just like, has found a really good family because that's the best one. The best time of year to get this is in a few months when they do the spring harvest.

Now it's like kind of old. It's just like I love the spring for the fresh tea. So I hope you enjoy it, but it's not the right time of year.

Yeah. But that's great. Awesome.

Well, as we were saying, we have instant hot water at Colonel. So MET lover can come by MET. I I have a little tea kit where I bring a where I bring like a little thermometer to, like a little thermal works thermometer.

Last Friday, when we do demos, I have this thing where if there's not enough demos, then I fill the remaining time talking about something completely ridiculous as an incentive for people to actually demo. And last night time, I spent twenty minutes walking through my air table and going through my entire tea travel kit, including the the the temperature monitor. Because like, yeah, you will show up.

There's only a boiler. You can't get it to the right. Yeah.

You know, you need this at 80 degrees. But anyway, yeah. Sorry.

Yeah. We have a we have electric kettle with the temperature thing at home. Yeah.

I would watch You should start a company YouTube, but it doesn't have anything about search.

and like other rants. I don't think I could talk, but something that I started doing do you you two know Sam Lambert of Of course. Landed Scale?

Of course. Very outspoken guy. I love the guy.

And we just Like last week, we just went on X Live and just sat and like shut the shit for like an hour. And I think we'll probably do that again. Yes, we'll probably come up there.

Well, I don't know what we'll call it. Maybe P99 Live or the P99 Pod or something like that.

Yeah. Thank you so much for having me. It was a pleasure.