The Programmable Data Center — Evan Conrad, SF Compute

NVIDIA is obviously, like, the 800 pound gorilla in the room. Yeah. Why don't they...

Why didn't they make SF Compute? So our impression is that NVIDIA can't make it, but benefits tremendously from an existing. The reason why they can't make it is because they would then not be independent enough for people to trade on it.

And, also, congrats on AIE. It it seemed to, y'all are rolling.

I'm aware of this. Yeah. Yeah.

Yeah.

And we started there in the technical stack. And a way to think about it is it's like you're in a hotel and you're trying to sell that hotel room, and then somebody asks you, Hey, like, are you gonna clean my room? And then you say, Oh, I guess I probably also need the hotel room as well, or, like, I need the, the cleaning closet.

And so you go get access to the cleaning closet, and then they want room service, and so you get access to the kitchen. And then, you know, they want, security, and so you get access to the security closet, and you get this, and you do that, and you do this. And at a certain point, you're just running the whole hotel.

And the moment you're running the whole hotel, you say, oh, wait a second. Instead, my supplier on this market, is somebody that looks closer to a fund, or a cloud that has access in our case, a cloud that has access to the data center, and we're gonna level up. And so what we've spent the last several years doing is building the entire operating capacity of a cloud, in order to offer in order to actually deliver, the compute at a reliable level.

We call that physical settlement. We're taking a term from... The futures market.

From futures markets, from other markets, where in, you know, oil or soybeans or something, physical settlement means you get delivered the oil and the soybeans. Yes. But in some markets, like in oil and also in power, there's like a delivery network.

In oil, you have to have pipelines. If you would like the oil to be delivered to you, there has to be a way for it to get to you. Yes.

And and for people interested in financial history, during COVID, oil went negative Yeah.

Yes. There's a bunch of weirdnesses like this in other markets as well. So in the power markets and the grid, there's parts of the orders that you fill out, the buyers and sellers, that are there because the boiler takes a while to cool off.

And you can, you know, stop the boiler, and then it will you can start it back up again within some period of time, and so that has to get factored into the order book. This genre thing is the same thing that happens in compute markets. So we co designed physical settlement and the order book at the same time, and that meant that we get to actually handle the real constraints of compute, which might include something like, oh, I'm okay with being interrupted, but only if there's, like, storage on the thing that I can recover from really quickly.

So for example, well, how how mature or deep or liquid is the compute market today? Not very mature or deep or liquid. Okay.

So what we've been doing is first, we had to we had to go get a bunch of supply online in the first place, which just takes a while. And then we had to build the entire platform in order to do this. And we just now got that working.

And so now we move capacity over onto the order book, which makes it liquid. And so 5% of our total capacity is on the order book. We're consistently moving more and more, and then we're bringing on more capacity that's all on the order book.

The nice part is that, maybe I should have said that the volume on the actual order book is small, but the, the liquidity on the order book is high. So for that small amount of capacity that you have, you have, like, very concentrated liquidity. So a sort of standard metric, I would say, for someone who is using this is a customer who buys capacity from us and they spend $100 to buy that capacity will make back $25 on average from selling it back to somebody else.

And that's the number that we or this is one of the metrics that we try to track because, we want the liquidity to be useful to the commercial participant. And you can measure liquidity in a bunch of different ways. You can say like, Oh, I just want tons of makers or something.

And you can measure it in, like, high minded ways. We try to measure it by, like, is this solving the problem, for the customer?

the CFTC or whatever. What's the status of those things? So I think standardizing the contracts in general is still pretty far away.

Yeah. And the what we were trying to do at the time was... Or basically, what we learned over the last several years was we just needed to operate the whole thing.

Otherwise, you cannot standardize the thing. And so to get closer to something that can be standardized, you first need to make all the operations of the thing the same. Right.

To give an example of what can go be different is a single operator can offer two different types of virtual machines. One type of virtual machine, might have a 40% performance degradation versus another types of virtual machine. And 40% is a lot.

When you say performance degradation, are you talking like MFU type things or, like, amount of burn in that you've experienced or what? It will affect... So in this particular case, I'm talking about GPU kernel launches and how long it takes for that to happen, which is a process that happens quite frequently when you're doing sort of either training or inference.

Okay. The, that will basically...

And so, like, people are trying to... Well, you have to establish the equivalence of those things. Correct.

Yes. And I'm wondering if they they have been established. So what we did instead, was we created an order book that had multiple different constraints.

And you can say, I wanna have, you know, something that's between this and that. It runs from this time to this time. It's like x to y.

It's a bunch of different properties that you can attach to your order, and then we match that against sell orders that have those properties Mhmm. In a big matching engine. And the reason why we did it this way is it allows us to generate liquidity without having to standardize before you know what the technology can do.

And because the technology was shifting so quickly, we wanted to basically be able to do that. The trade off of this is that you have less concentrated liquidity, and so the prices are going to have... There's gonna be more spreads between the buy and the sell.

Yeah. It sucks. Yeah.

But on the other hand, it means there's lots of arbitrage to make if you can do that correctly. So if you have a product that can use lots of different types of capacity, then you might really like SF Compute because you will be able to use lots of different types of capacity and you'll make more money.

So a bunch of different things. One is dimensions categories that I will say is like, is there a workload running on it right now or is it capacity in the future? So is it capacity right now?

Capacity right now, in order for you to sell that, you have to interrupt your current workload. So you have to think, do I care if my current workload is interrupted or not? Capacity that's sold that's starting a month from now, you can buy and sell that all the time, And so you get, liquidity on that really easily.

There is some concern like, Oh, no. Will I have capacity next month? But it's very different than if you're interrupting a workload right now.

The way that the interruption happens matters. So an example is if you're on a cluster and the cluster doesn't have as amazing storage as you want it to have, so an example is like, do you have a Weka or a Vast system under the hood, and do you offer that, and like, what's the size of that? This changes how you respond to, being preempted or removed off the cluster.

Meanwhile, the location of that storage matters. So if you're able to move that data from one cluster to another quickly, then maybe you might be able to spread across multiple different clusters, and you can be interrupted over here and then pop back up over there. To make that work, you also need, like, a a fast transit path between one cluster or another.

Typically, this is implemented with a WAN, but most of these clusters don't necessarily have, like A wide area network? Yeah, a wide area network. So, basically cables that connect the clusters together.

A private Internet is another way to think about it. A lot of clusters that are being spun up today are like they're not attached to traditional CPU clusters were next to the customer for latency reasons. In GPUs, they're next to the power, wherever the, like, you know, you put the data center next to the power rather than next to the customer.

This means that a lot of stuff is getting put in places that previously they weren't expecting data centers there, and so you may not have, as, excellent of a transit path, from one to the other. So this this starts to matter. Other examples are weird shit where we theoretically support but don't actually, like, this is a thing that we're thinking about but haven't done, basically.

The early GB two hundreds broke. The early H one hundreds broke. And so if you were buying H one hundreds or you're buying GB two hundreds, you might really want the ones that were deployed after a certain time because you're not getting the first JITI ones off the rack.

And so if you're a super sophisticated customer, you may be willing to pay more, because for that. Oh, so it's it's about, like, what cohort they were they were produced Yeah. And they get better over time?

So, like, the first ones off the rack maybe had some fault or some problem. Right.

Yeah. So I'll I'll give an example. You're describing burn in.

Yes. So when you get a cluster, a good cloud, what it does, is it tries to kill all the GPUs, before they ever get in the hands of the customer. And then, the GPUs that are broken or the nodes that are broken, you get RMA'd.

Yes. You basically go and request You return it. Yeah.

You return it and you get a new one and there's a lead time to that, but whatever. You'll get Yeah. You'll get some back.

And then the customer never has to experience this. The thing I'm referring to is that, like, in early h one hundreds, there were, like, firmware bugs that may not even be related to the specific... Like, they may not be NVIDIA.

They might be, like, Super Micro has a firmware bug in the in the box that you bought, and it applies to all of them.

To answer your first question Yeah. Three to five years. Wow.

We're starting to see month by month or quarter by quarter or the houses? Are you asking what... How the cash flow works?

The the the contracts, like, fee... The futures contracts. Right?

Like, again, like, I'm a former con... Contracts derivatives trader. In the in the market itself Yeah.

Like, is in the broader market Yeah. Clusters are financed three to five years. Okay.

So the payments are over Totally get that period of there is also, like, blocks of capacity in our order book, which can extend any arbitrary time. Yeah. There's some stuff that we have to do, for, the cash flow on that.

Right now, what we do is we just ask people to pay entirely upfront for a block. We don't ask them to do that for a lease for a longer term contract. This gets into something that I definitely need a whiteboard to explain.

what's more interesting to you? Real time or long longer out? So we care more about longer out because we're trying to build a forwards curve that has a lot of liquidity on it.

The... These are kind of the same thing and kind of not. So what we do is we...

Since, say, you can buy a... An hour out. You can buy right now to the next hour, or you can buy right now to the next three years and anywhere in between.

That's wild. I know. And so the...

You know, or fine, The... There's a certain difference that happens that's dependent for each customer. Basically, like, if I have a workload that is that's batch inference, basically, where I can be interrupted at any point, I don't really care, then my ability to be cut off is probably like, I need an hour of time to know that I'm gonna be cut off, and then I'm good.

Or I can even be notified in, you know, a minute or two. But if I have, live inference customers who are expecting me to be online, then you can interrupt me, but in a month, maybe two months, maybe three months. Yeah.

And then I can adapt to that, capacity over that period of time. But you cannot interrupt me in a minute. And so those two things are priced differently, and the reason why they're priced differently is because what is the price you would pay in order to interrupt your batch job?

It's very different than the price you would pay to interrupt your live job. If you have a live job, you're you are probably paying an SLA when you break for your customer. Does that make sense?

Yeah. I see. So this is actually the market for breaking leases.

Are the market for breaking leases, basically. Oh my god.

is in power markets, there's actually two markets. There's, like, the day ahead market, and there's the live market. Compute kind of has the same thing, where it has capacity that is coming online, or capacity that's about to, arrive to you, and then capacity that is currently in use.

And these have different price dynamics.

Is compute further out priced...

Do Oh, this is an ex... Both at different times. I find it fascinating that you don't know.

Yeah. I mean, both... It depends on who you are.

Okay. So I'll make the case. Wait.

So you're asking, is compute right now? Same unit of compute today?

Like, I think this will make intuitive sense. It's basically, like, if I have milk and it is about to expire, I will lower the price on it. Yes.

And so that's basically what we see. So it's it's the NVIDIA release cycle, basically. Right?

Well, we're more saying, like, if there's a block of compute and it's approaching now, it can't be sold in the past. And so it has to drop its price in order to clear at some price. Yeah.

And so initially, you see this, like, drop where, stuff that's sold right now, anything that's on the market right now, is something that hasn't been sold, and so therefore, the price will drop. Mhmm. And then it goes up, to where it wants to be.

Mhmm. And then maybe compute in the future is worth less, because people don't want... Like, people would like to compute now, and so they're sort of paying for the shorter lead time.

Mhmm. And they pay less for the compute that's off into the future. This is what I think a normal healthy curve looks like.

Yeah. The place where I'm like, oh, man, no shit, is that, right now, if you took a B 300 and you tried to run inference on it, you can make $40 an hour or so selling that Any model. Yeah.

Any model, basically. And so that's if you use it at a 100%. And maybe if you only have, like, 50% utilization or something, so, like, $25 an hour.

Yeah. So $25 an hour. Meanwhile, you can buy v three hundreds at, like, 5 to $6 an hour.

One of these numbers has to change. Yeah. It's a huge arbitrage.

It's a huge arbitrage. This in your your talks a lot. Yeah.

So a bunch of people are basically looking at this and thinking, if I think prices are going to go up, I might be willing to pay more than what I previously thought in the future, because I might think, okay, I can buy a five right now. But if I can't buy anymore, then maybe I will be willing to buy even more in the future just to lock it in if it's gonna come online later. So we see the other side as well.

Kinda depends on who you are. Cool.

I think part of this is because of things like DRAM, and part of this is because of, like, larger NVLink domains Yeah. And things like that. So there's just no way to abstract all this to, like, some generalized unit of compute that you can sort mush around and and whatever you like.

I think, like, when you were saying oil... Even oil isn't oil. Oil is actually just, a bunch of different types of oil.

Yeah. Like jet fuel or car fuel or whatever. I think it's the same thing.

You're gonna get a bunch of different types.

Yes. So we've been able to reduce the, the the problems to a certain level. But Yeah.

It it... Like, job is, like, dimensionality reduction. Yes.

We reduce a lot of the dimensions, but nonetheless, there are still dimensions. Yeah. Like, it's still gonna exist.

Do you do you know how many SKUs you have?

We actually made SKUs like a a concept that... And the goal is to make it so that somebody else can make their own SKU. So if I wanna buy Blackwells, I can say, I'm willing to buy Blackwells at $6 an hour or less.

Uh-huh. And you can match me against GB three hundreds or you can match me against B three hundreds. And I don't care which Blackwell as long as it's less than $6 an hour.

But you just say you care about memory and other stuff. Totally. But not every customer does.

So some customers, like, just can work on black holes, whatever it is. Some customers are super hyper specific, and they need this exact thing. And they're gonna pay more for that because the market's gonna charge them more.

Okay.

With the caveat that having a named query is a new thing we're doing, but the query system as a concept is how the order book works right now. Okay. Interesting.

Mainly just the requirement for physical settlement. Yeah. We think that if you...

We we spent three years learning why you need physical settlement. And, our entire conversation today is, that it is these things are not as fungible as you think. We're gonna try and make them fungible.

That's the industry's job to do that. But without that, you run a lot of risks. If you try to build, too much financialization before you've the industry has had time to build out the, the physical settlement, then I think what you have is you have a market that's very easy to manipulate and a market that has a lot of basis risk, where, the price you may be hedging against actually has nothing to do with the price that, the market...

Yeah. So to be clear, we think, cash settlement as a future concept is a great idea. Yeah.

We just think it should be derived from physical settlement. Yeah. Because, as you said, it's a side bet, And it would be kind of like if you...

I mean, be kind of like if you made a cash settled future before you made the physical settlement.

And lots of those go very poorly. Like, it would be kind of bad if you made a, it would be bad if you started selling, the the derivative on a stock that doesn't exist yet.

Our concern at the moment is Is this 12 banks? Yeah. Yes.

Well, our concern is there's a very similar problem, where you can manipulate an index. Our impression is why you can manipulate that index is because a lot of the indexes are based on a very small amount of data that has, not a lot of correlation with the actual price of buying a new reserve contract. That's what we're concerned about.

Oh. So we're big fans of Silicon Data and Computexchange. I think they were also very early to this, and I think they are doing, like, a reasonable shot at the a path that we think, that we'll likely end up being partners with.

the general gist is they're doing cash settlement. We're doing physical settlement. We think both can exist and should exist.

Mhmm. And that they're sort of mutually beneficial to each other, and so totally fine. So I think they are one of the earliest in the space, and I think they are, clearly took the bet, well before anybody else did.

And I think that's, like, very honorable to do.

the categories of folks that we've seen is there's folks that are trying to set better standards. So seminalysis, I think, is working to try and create better standards. There's folks that are trying to do, residual value insurance and so on.

There's a number of different companies in this space. I don't know any of them super well, but, I know know this is a concept, and I think that's important. There are folks that are trying to do, deposit, lending where, like, they'll front your credit risk for the cloud.

do all the things a cloud does without being the cloud, as in we don't own the hardware. The reason why we don't own the hardware is because if we did, we would be able to manipulate the forwards market. So we have to be Switzerland.

We have to be independent, which creates a lot of sort of strange things about the company. But in order to do physical settlement, you have to do site visits. You have to visit the data center.

You have to have to design the bill of materials and ensure that you have to basically do like supply chain consulting, where you have relationships with different OEMs that ensure that things get delivered on time. Then you have to, do cluster architecture, so you're sort of designing the cluster itself. Then you're, rolling out racks or you're working with partners to roll out racks and supervising that.

when you buy a node from a cloud, there's the bare metal. Yeah. But there's all this shit around the bare metal.

Yeah. Yeah. And in order to make that bare metal node work, you have to operate everything else.

Then we... Mhmm. Because I just like...

We're we're the ones that do that. You're you're... Yeah.

So we offer supercomputing as a service. And what this means is folks come to us and say, you know, I've got x, where x is I've got, you know, x megawatts here. And maybe sometimes they mean I've got a data center, that's like I would like to turn into a GPU cluster.

Like, my options are I could sell the GPU cluster... Or I could sell the data center to somebody, or I could become the cloud. And then we go to them and say, great.

We will turn you into the cloud. We're the supercomputing part. You're the data center clerk.

Let's partner together, and then we will sell and all make money. Sometimes it's people who have something that could turn into a data center, in which case we're gonna basically project manage that up to the layer, and then we will work with those we'll work with those folks to turn their powered land or whatever they have, into a data center by working with partners, and then we take over at the supercomputing Okay. Requires us to have some sort of data center expertise in house, but we don't self perform builds.

So...

like Palantir for compute, because it's like these... Each time, it's this new rodeo. Yeah.

But... And we, like, we we deeply partner with folks on both sides. Yeah.

And primarily The US?

So when... So we're deploying clusters all around the world. Yeah.

And when we do that, we're working with people who themselves are trying to be clouds all around the world. They look like sovereign compute players. And our pitch to sovereign compute is amazing.

Let's help you set up a cloud in your own country, and we will bring the San Francisco offtake. We'll bring the San Francisco expertise. Uh-huh.

We'll set up the clusters. That's our whole business.

Nobody I can name. Really? Basically.

Wow.

Yeah. So our impression is that NVIDIA can't make it, but benefits tremendously from it existing. Reason why they can't make it is because they would then not be independent enough for people to trade on it.

If you don't trade on it, then you don't have enough liquidity for lenders to finance against the forwards market. Yeah. This is also why Crusoe and CoreWeave and Lambda, why the hyperscalers can't make it, why Base 10 and Fireworks and Modal can't make it because they have too much demand on that side.

This is a thing that benefits the entire industry because it makes the financing costs go down. But nobody can make it unless they give up the ability to do all the big inference money and all the big cloud deals. You have to kill your business in order to make this.

We closed several $100,000,000 of volume. We have a significant number of new clusters on the way. Yeah.

And then the things that I think I'm most excited about, is we've shipped a whole bunch of new features, that make SF Compute very viable for platform as a service companies. An example is, we allow customers to subdivide their InfiniBand fabric without having to ask us. So many clouds, when you go to buy them, they will give you one InfiniBand fabric.

And then if you want to re redraw the lines of that InfiniBand fabric, you need to contact them and they will then do something for you. We now can do that on the fly for customers, which I think is really great, because it interfaces with the market really well, where you want to be able to redraw things in order to scale up and down your clusters. Would you describe this as like you're virtualizing the data center and making it programmable?

Yeah. We're virtualizing data center and making it programmable. Yeah.

So all the stuff that you would have gotten if you had owned the hardware, we're trying to virtualize that. Yeah.

we would love to work with you in order to build that GPU cluster. We will take over and we will, do the supercomputing portions of that and ensure the things go right. We own the SLA.

And then if it turns out that that was a bad idea that you need out of that cluster, we can immediately sell that cluster to somebody else so it doesn't hit your balance sheet.