Reiner Pope – Chip design from the bottom up

So I'll start with sort of the very smallest fundamental unit of of chip design, and we'll sort of build up into what an overall, like, actual production chip, what are the components of that. Yep. At the very bottom level of a chip, the primitives that we work with are logic gates, which are very simple things like and or not.

And then these are connected together by wires that have to be laid out physically as metal traces on a chip. The main function that AI chips want to compute is multiplication of matrices, and really inside that is the fundamental primitive is multiply accumulative, just like of pairs of numbers. So, we're going to demonstrate what that calculation looks like by hand, and then infer what a circuit would look like for that.

It'll turn out to be easiest if I do multiplication accumulative, something like a four bit number with another four bit number. And then we're going to the actual clearest primitive is actually multiply accumulate. So there's a multiply these two terms and then we're going to add in, so product of these two terms and then we're going to add in an eight bit number.

you know, whatever computation happens inside a computer? Yeah. So there there's a few reasons for this.

It it's a little bit more efficient, but the the reason it's natural for AI chips is that if you look what's happening during a matrix multiply, the what is matrix multiply? In very short, it is there's a for loop over I and over j and over k of output I k plus equals to input I j times other input j k. And so multiply accumulate happens at every single step of a matrix multiply.

Makes sense. And then the other observation is that the precision will almost always be higher in the accumulation step than in the multiplication step. This is maybe specific to AI chips, but you're you're multiplying low precision numbers, but then when you accumulate, errors accumulate quickly, and so you need more precision here.

So this is why we've chosen to do a four bit multiplication and an eight bit addition.

maybe that that part is like less intuitive to me, but it's maybe the same principle? It it is really the same principle. I guess the sort of I mean, I guess the the separate principle is that as you are summing up this number, you are summing up a whole bunch of numbers, and so you've got a lot of rounding errors accumulating.

Whereas in this case, there's, like, there's there's only one multiplication in that chain, so there's not a lot of rounding errors accumulating in the multiplication.

j So, j many

I mean, as a human, we would probably separate it into two, but we can sort of do it all in one using long multiplication. So the multiplication term first. We're going to multiply this number, this four bit number here, by every single bit position in the other four bit number.

So we write that out. First one zero zero one multiplied by this bit position. That is this number itself.

Then shift it across by one, we're multiplying by zero. That gives us an all zeros number. Shifted across even one more to multiply by this one, we get one zero zero one.

And then finally for this last last bit position, we get an all zeros number again. So this this sort of gives us a bunch of terms that we're gonna have to add for the for the modification. And then while we're doing that summation of this, we might as well add in the actual accumulated term as well.

So we just copy that directly across. So so this is the sum. It's a it's a five way sum that we're going to want to commute compute.

So firstly, what what logic gates did it take us to even get to this intermediate step? We needed to produce all 16 of these partial products. Mhmm.

How do I produce one of these partial products? So let's take this this number one for example here. It is one.

So what was it how do we produce this number by multiplying this number by this one over here? We can actually produce that by an AND gate. This number is one if both this bit is one and this bit is one.

Yep. If either of them is zero, then the multiplication of one times one of zero times anything is zero. So to produce all of this stuff, we ended up consuming 16 and gates.

Mhmm. Or in the general case, if I were doing a, like, a p bit multiply times a q bit multiply, then this will be like p times q, many ands. Exactly.

Finally I sum them. Actually most of the work is gonna happen in the summing. And so let me describe sort of the other logic gate that we use here.

AND is almost the simplest logic gate that exists on a chip, it's it's it's almost the smallest. At the other extreme, typically the very largest logic gate that you'll use is something called a full adder. And what this does is sort of it does coming from software, you might think that a full adder, it adds 32 bit numbers together.

In this case, it just adds three single bit numbers together. And so you can think of it as like adding zero, one, and one together. Now when I add these together, the result can be zero, one, two, or three.

So I can express that in binary using just two bits. So as input, it has three bits, and as output, it has two bits, which in this case, the number two in binary is one zero. So this is also known as a three to two compressor because it takes three bits of input and produces two bits of output.

The sorry. The the the three inputs are all all bits that are in the same sort of bit position, like three bits that are in a column here. Yeah.

Yep. And then the two outputs, I have sort of drawn them vertically here and horizontally here to kind of match this vertical versus horizontal layout here, which is expressing that things that are in the same column are in the same bit position. Yeah.

Whereas things that are in adjacent columns, like this is a carry out, whereas this was the sum. So if the inputs in the full adder are, let's say, like, one zero one, then the output would still be one zero. If it was one one one, it'd be one one.

If was zero zero zero, it'd be zero zero. If was, like, zero one zero, it'd still be zero one. So yeah, it's just counting essentially the number of things and expressing that in binary.

So this circuit actually can sort of capture what we as humans naturally do when we're doing summing along the column. So I can show that sort of, I'll show sort of one iteration of using the full adder to sum. The way I sum here is going be a little bit unnatural for humans.

Humans, we would sort of sum along the column and then remember the carry. But instead of remembering the carry, we'll actually just explicitly write it out. So in this, we proceed from the rightmost column towards the left.

On the rightmost column we sum the one and the one. And that produces like a zero here and a carry of one. So we've sort of used this full adder circuit on this pair of bits and produced a pair of bits as output.

Now we can do the same thing with this column. We've got a column of one, two, three, four numbers. And so maybe we'll like take the first three of them, run a full adder on them, and that gives us a zero and a zero as output.

So the sum of these is zero zero. So that's the full adder applied to all of these bits. As I've used up bits, I'm gonna sort of just cross them out to indicate that I've handled them.

Let's just keep going a little bit more. So we'll go here. I take these three numbers, I add them.

That gives me a one and a zero. I've dealt with these three numbers. And now I take one, two, and I can even take these three numbers, for example, right now, and add them, and that gives me a one and a zero, and I've dealt with these numbers.

So I can sort of like, the way I should view this is that I have this whole grid of numbers that need to be added. I'm going to just keep applying full adders to all the bits that are here, constantly removing three numbers from a column and then writing out two numbers as output. Keep going with this over and over and over and again until I eventually get like some just one single number coming out here.

Something like that. This is probably the wrong sum. So this approach that I've described here, this is called a data multiplier.

And this is sort of like the standard for how you do area efficient multipliers using full adders. Let's try and quantify the circuit size of this just so we have got a sense of how big things are so we can compare to them later. How many full adders did I use?

I started with how many numbers? I have the 16 partial products, which is the product of all of these terms with all of these terms, plus the eight terms that I'm adding here. So I started off with 24 bits, and then I produced eight bits on the output eventually.

And in every step I was sort of crossing off three numbers and writing two numbers out a result. And so every single use of a full adder eliminates one of the bits here. And so how many full adders?

It must be the twenty four minuteus the eight. So there were 16 full adders in this circuit.

p plus q equals p times q. That's right. So I think this explains sort of, or at least hints at, the second reason why we chose to do a multiply accumulate.

First reason being that's actually what shows up in matrix multiplication, but second reason being it gave us this very slick p times q, very simple algebra. Mhmm. So we've sort of described, like, this whole procedure, every single atomic step that I took here becomes a logic gate, and then sort of the wires connected together.

Like, when I had these three inputs that that I salvaged to produce these two outputs, like, if I think of mapping this to a physical device, there would be a wire that runs sort of connecting all three of these things together into a logic gate that produced this output. Okay. So this is the the main primitive at different bit widths that that is that is inside an an AI chip.

Basically, can you can you fudge them? As drawn, they're actually not particularly fungible. This is actually one of the main choices you have to make when designing a chip, which is how much of f p four, how much of f p eight do I have?

And then sometimes I'll make that consideration from the point of view of what do I think is the customer requirement. Another way to take an angle on that is to say what is the power budget for equalize the power budget between FP four and FP eight.

Why Like, is the ratio exactly 2x? Yeah. Exactly.

Yeah. So, part of it is I mean, surely that won't be exactly equivalent to die area. There's a data movement reason, actually, and we'll maybe come back to this when we sort of look through how it goes into and out of memories.

quadratic. It's quadratic in fact. Yeah.

in fact NVIDIA made a change. Historically up until b 100 or b 200, every time you have the bit precision, you double the flop count. That ratio is exactly like for the reason you said, because of this quadratic scaling, that ratio is actually slightly wrong.

It should be like an even bigger you should get an even bigger speed up than you might otherwise think. NVIDIA's product specs have sort of started acknowledging that in b 300 and beyond, where the FP four is three times faster than the FP eight. Though it should be four x.

Yeah. What I've shown here is the simplest case of integer multiply. When you're dealing with floating point as you do in f b four and f b eight, there's this other term which is the exponent that just complicates So what can we see already from this?

Think the big observation you've made is that there's this quadratic scaling with bit width, which is very effective and is the single reason why low precision arithmetic has worked so well for neural nets. But the other thing we're going to do now is we're going to compare the area spent on the multiplication itself with all of the circuitry that is around it. So we'll walk back in time a little bit and see how did GPUs prior to Tensor Cores work, which is the same way as the way that CPUs worked, fact.

So which is like, where do we stick this multiply accumulate unit? Generically, I'll describe like a CUDA core or a CPU. You'll have some register file, which stores some number of entries.

Maybe maybe it's like eight eight entries of of, like, in this case, I guess, four bit numbers, but typically, like, 32 bit numbers or something like that of which are just numbers. So this is the like, inside the CUDA core, I'll have some register file and some depth, and then I will have my multiply accumulate circuit, multiply and accumulate circuit. And what it's gonna do, it's gonna, like, it's going to take three arbitrary registers from this register file, perform the multiply accumulate, and then write back to the register file.

So it's going to maybe write to this one, but was able to read from this one, this one, and another random one. So it'll take three inputs like this. So this is the core data path of many processes.

Most processes look like this. You've got some set of registers and then you've got some set of logic units or ALUs. We want to analyze the cost of the data movement from the register file to the ALU and back.

So ultimately there's going to be some circuit that says, well, I don't always have to select this guy. I might select any of the registers at any point in time. And so a first question is, how can I build a circuit?

The circuit that I'm going to look for is a MUX. So in this case, it's going to have eight inputs, one from each entry of the register file. And it's going have one output, which is actually producing this output.

And then what is the cost of this thing? All we have to build it out of is and and or, and so how do we build it? We do the dumbest thing possible.

We form a mask saying, okay, when we want to read the third entry, we're going to end every single entry with either one or zero based on whether that's what we want to read, and then we're going to or all of them together. Okay. Just to make sure I understand the basics.

What the MUX is doing is it's just like selecting. Just selecting. Just selecting an input.

Yeah. So, like, invisible to software is like you say I want input number three, that means there's a MUX here. Yep.

And so, like, what is the cost of this MUX? So an n input mux operating on p bits. Well, I'm going to so I have n n rows, that's this eight rows, and I've got, like, each row is p bits wide.

Well, I have to and every single bit. So I get n times p many and gates. Every single input, I have to say, am I gonna like mask it out or not?

And then and then I'm gonna or them all together. And so there's gonna be like n minus one times p many or gates, which is saying, I've got all of these different things, almost all of them are zeros, but I need to sort of collapse them down into into like, from my eight options down into one option.

a sort of like in and of itself, a quite complicated circuit. Yeah. I mean, this is the first step of all of the hidden data movement costs that are in the shop.

Yeah. Yeah. And so, like, the thing like, we're we're just gonna, like, compare, like, I have to pay this cost, and I've got one max here, and then in fact I have two more copies of that for each of the three inputs to my multiply accumulate operation.

And so I have this cost which is like like three times n times p and gates over here compared to this p times q, like, sort of gates in the actual circuit that I that is doing the thing I care about. And if we plug in actual numbers, like this n being eight, like, I get, like, 24 times p gates over just in the data movement compared to if q is four, four times p gates just in the adder multiply adder. And sorry, where is the three coming from?

Three different inputs here. Got it. Okay.

So we'll take some inputs. We'll we'll have maybe like we'll just do a a two way mux. So we've got two different numbers.

We've got these two inputs, and then we have a so these are the inputs that are being selected between, and then we have a selector. Which says which can either be like, I want this one or it could be, I want the other one. So this is a one hot encoding.

So this is what we all start with. And then the output we want to produce, let's focus on this case. So this is the actual input we got.

We just want to produce this guy as the result. And so very laboriously what we do is we and this bit with all of these, and so that produces, like, and in this bit with this row. And likewise, we and this bit with this row that produces all zeros.

So this was the there's four here. There's four ands here. And then finally, we just OR these two together.

And this gives a one, we OR these two together, this gives a one. We OR these two together, it gives a zero. We all these two together, and gives a one.

And so this is the four ors. So, like, this actually ends up looking a little bit like addition, in fact. Like, we did exactly the set of same same set of ands here.

Sort of we've added all of these things together, but then instead of collapsing it by using these full adder circuits, just get a very simple collapsing with or gates.

So so yeah. So this was this was with n equals two inputs. Ah, great.

In the general case, we will have n, sorry, this is n rows, and then we'll have p bits per row. So that gives us the the n pi n times p many and gates. So this circuit I've described here, almost all of the cost, like, seven eighths of the cost is in the reading and writing the register file, and only a tiny fraction of the cost is in the LogicUnit itself.

So this is the problem to solve. Essentially was the state of play prior to the Volta generation of NVIDIA GPUs. This kind of thing is what was inside the CUDA cores, and and this sort of problem statement is what motivated introduction of tensor cores, which are more generically called systolic arrays.

So if we think about how we're gonna solve this problem, we're spending almost all of our circuit area on something that we just really don't care about and is hidden to the software programmer, and the thing that we actually care about is not much of the area. Well, make this one bigger somehow while keeping this at the same size. That's the goal.

So the evolution was we had baked this much into hardware in this stage. This single line is a multiply accumulate, this single thing was baked into hardware. The idea of a systolic array is to sort of go two levels of loop up and bake this entire loop out here into hardware.

And so the idea being that if we have a much bigger granularity fixed function piece of logic, maybe the taxes we pay on the input and output are much smaller.

That's right. So there's two effects that we're gonna take advantage of here. One is just that we can do more stuff before per every trip through our register file.

Mhmm. And then the other thing we're gonna take advantage of is, in fact, in some of this loop we can take advantage of, for example, some certain things staying fixed. Let's sort of Visually, we're going to look at this matrix multiplication.

So, portion of the loop corresponds to a matrix vector multiplication, in fact. So, we'll take a matrix and multiply it by a vector. So how do we do this?

We take every column gets multiplied by the vector and then sum. So we're going to sum sort of along columns. So this zero and three gets multiplied by the three and seven and gets summed, And then the one and two gets multiplied by the three and seven and gets summed.

So there is a multiplier accumulate associated with every single one of these entries in the matrix. So we'll just draw out these four multiplier accumulates.

Yeah. So the addition so really there's only one addition per dot product but like But we'd to start with initialization of zero. Yeah.

Yeah. So what we're gonna aim for is to have we want to have quadratically more compute. We do.

We have. We have got sort of x times y as much compute as as we had before. But we're gonna want to somehow aim for having only x times as much, like, communication.

And this is sort of the intention so that we get this advantage term going as y. So we've laid down the multiplications. We're going to want to bring in a vector of size two, so that sort of already is in line with our comms target.

That's fine. However, we need to somehow manage the communication of this matrix, which which which does which exceeds our budget of of x. And so the idea is that in an AI context, this matrix is actually going to stay fixed for a long period of time.

And so instead of bringing it in from the outside, so we've got some register file sitting over here, We don't want to have the amount of stuff coming out of this register file. This is the term that we want to go sort of as x in some sense. We don't want to bring this full matrix in from the register file every cycle because we don't have enough.

That would cost too much in terms of wiring from the register file. And so instead we're gonna store our key trick is that this matrix can be stored locally to the systolic array. And so we'll store these numbers zero, one, two, and three just like a gate called a register that like physically stores these numbers and we're gonna reuse these numbers over and over again for a for a large number of different different vectors.

in before you get some value out of it. That's right. That's right.

So just to complete the picture here of concretely how that looks, I swapped the three and the two here. Three and two. So just like this zero and three is going to multiply by the three and the seven, and so we're going to form a dot product sort of along columns here.

So somehow we're going to feed a three and a seven in here. These participate in sort of this feeds into this multiplication and also feeds into this multiplication. Likewise the three feeds into here and also into here.

And then we're gonna sum sort of along here with like starting at the top of a column we feed in zeros and then coming out the bottom we get results coming out. So sort of just to visually see what we've got, there's a dot product that is performed along columns in a matrix and that sort of maps exactly to what is done spatially in the systolic array here. So this is one dot product summed vertically and this is a second dot product also summed vertically.

And then what is the data that needs to go into and out of the register file? We have x amount of data that's coming out here on the output, and we also have sort of this data coming from the input, x amount of data from the input. And so with respect to the input and output vectors at least, we've sort of met our goal of having only x as much data going in and out of the register file.

This leaves open the question of, like, I said that the weight matrix is stored locally in the systolic array, how did it get there in the first place? It's sort of like at some point you need to boot your chip and populate this data, and so where did that come from? The trick is just we just do it very slowly.

So we we very slowly trickle feed it in into the systolic array. The sort of the simplest strategy is that we we sort of run this daisy chain that says like feed a number into here, and then and then on the next clock cycle, it'll move down to the next entry with the systolic array. And so we can do that in every in every column in parallel, and that gives us sort of this is also gonna come from here, and this is gonna be another factor of approximately x units of bandwidth coming in.

Can you just would you mind repeating that sentence one more time? So we're sort of like, we know that we're going to be bringing in numbers only rarely into the matrix. And so we just want to come up with any construction at all such that the amount of wiring that actually feeds into sort of crosses this boundary of this systolic array, like this boundary right here, just we want to keep that bounded to x and not go And as x so a particularly simple strategy is that we sort of bring in a number into the top row of the systolic array.

That's what we can do in one clock cycle. And then for y consecutive clock cycles, we're gonna be bringing in the top row every time and then sort of shift all of the other rows down by one. And that keeps the wiring that needs to come from this expensive register file only down to a factor of x rather than x one.

I see. Okay.

Yeah. This this shows up sort of all the way up and down the stack. This is sort of close to the bottom, sort of like to to the gates.

There's sort of a a version that's maybe even closer to the gates of just like even the precision of number format that you choose to use. We saw that same effect. There's like a square cube law or a like squared versus linear term going on both in just purely the precision of this ALU, but then also in terms of the size of the matrix.

Yeah.

So this unit is sort of the next bigger unit we had, like the multiplication circuit, and then on top of that, we have a, like, a pretty large systolic array. I drew it as two by two, but in, like, for example, older TPUs, they were described as a 128 by a 128 of of this circuit shown here. And this circuit ends up being this is the the most efficient known mechanism for circuit for implementing a matrix multiply.

I see.

sizing decisions. And so already in what we've drawn so far, so AI chips all have this circuit in it. They have a systolic array, and then somewhere near it, a register file providing inputs and outputs.

The two sort of even within this scope sizing questions that you have are how big should I make my systolic array, and how big should I make the register file? And then the trade off for the size of the systolic array, actually, these two questions are coupled, is one way to think of it is to say, I'm gonna, like, have a budget for how much what percentage of my chip area I want to spend on data movement. So maybe I just say that I want this to be 10% and the systolic array to be 90%.

And then sort of I can size my register file. Bigger register files are more flexible. They allow me to run sort of more I can get more application level performance out.

Yeah. And what is the clock cycle of a chip? So I guess at baseline it's sort of worth observing that chips are incredibly, incredibly parallel.

Right? You've got 100,000,000,000 transistors in a A key thing that you need to do whenever you have massive parallelism is you need to synchronize between the different parallel units. In software, typically, have these very expensive synchronization methods like a mutex.

So one thread will finish what it's doing. It will ink, like it will grab a lock somewhere stored in memory and then notify the other thread that it's done. On chips, take a very different approach and say that every nanosecond or so, all circuitry in the chip will kind of pause for a moment and then synchronize every so it actually synchronizes every single nanosecond or so.

And so that is the clock cycle. The entire chip, typically all in sort of one fell swoop, goes in lockstep to the next operation that happens. And so what this looks like in circuitry is that you will have it's typically drawn so the clock is sort of mediated by registers, which are these storage devices that we've drawn elsewhere.

And the way to think of it is that I have I have some storage which is storing like a bit, which might be zero or one. And then I have some sort of cloud of logic which maybe is like this systolic array or this multiplier or something like that. And then I've got some and that's gonna produce some output.

So my inputs, I've got a bunch of inputs feeding to this cloud of logic. And then eventually later there's gonna be some output register that this writes to. There is a global clock signal which drives all of these registers, and it says, at a certain instance in time when the clock strikes, whatever value happens to be on this wire at that instant, that's what's going to get stored in so the challenge here is I would like to have my clock speed run as fast as possible because if I can run at two GHz I can get twice as many operations done per second than if I run at one GHz.

But what that ends up meaning is that I'm very sensitive to the delay through this cloud of logic, because any computation that is gonna happen in here needs to sort of finish before the next clock cycle hits. Right.

or is it just like, no. Either it's gonna finish by clock cycle or not? Yeah.

In standard chip design, you you margin it such that I mean, is a probability, but it's like many, many standard deviations, like way standard deviations out such that for all intents and purposes, is a reliable part. It it will always meet the clock. There are some weird exceptions to that.

There are clock domain crossings where you go from one clock to the other clock, and then you actually do have to reason about this Interesting. The main path, you just like you margin it such that you'll get there like 25% of the clock cycle in advance.

that is like too long such that it makes the whole clock cycle of the entire chip longer than it has to be. Yeah. So this is actually a huge part of the work of designing a chip actually is inserting them.

So it is done in a combination of manually and automatically. So just to show the very sort of dumb version of what you can do here, you can take this logic and split it in half. And so say, actually instead of just one cloud of logic, I'm gonna have two smaller clouds of logic, which do the same thing, but split them up by a register.

Right. Feeding in like this. And this is, like like if you split it, like, in the middle, you can hit twice the clock frequency.

That's great. You get twice the performance at the cost of this extra register, and so at the cost of some more storage.

Yeah. So taking that analogy, the the the the thing that you need to be mindful of is if I've got two different paths through some logic. So I have to do a computation like f here, and then computation g here, and then they're gonna come and meet for computation h somewhere here.

And so there's gonna be manufacturing variance here. In some chips, F will take a little longer, maybe in some chips G will take a little bit longer. And so if I've got some signal that's propagating through here, and the result from f and g have to sort of meet up at h, what can the the thing that can go wrong is that f can get there early and it meets like the previous value of g or the next value of g or something like that.

And h needs to know when to start. Exactly. Yes.

it it slows down the whole ship's clock cycle. That's right. This optimization that I that I showed here, this is just the this is sort of pipeline register insertion, it's called.

Yeah. We've inserted in the middle of the pipeline register here. This is a sort of pure trade off between clock speed and area.

This is the easy case. There is a harder case too, which is sort of drawn it as a pipeline of logic here, but in other cases you may have some calculation which actually feeds back in on itself. So it runs some function f and then writes back to itself like this.

So, for example, this might be this addition, like you've got some number that you're adding into every clock cycle, and so this could be like a plus where adding in some number every clock cycle. So this little circuit, essentially, it's just gonna sum all of the numbers that are presented on on different clock cycles. And the challenge is if this plus takes too long, what can I do if I, like, split it in if I try and put a pipeline register, like, right in the middle of it, like like here in the middle of it, this will end up changing the computation that's done?

Instead of forming a running sum of everything that comes here, I will actually have two different running sums. I'll end up having a running sum of the even numbers and a running sum of the odd numbers.

As you just demonstrated. Yeah. It took a whole lot of work to do a summation.

So, can take the early parts of that work and then stick a register in middle and then take the late parts of that work. Got it. Okay.

Yeah.

the chip architect set the clock cycle. So just for for one example, the primitives you get from TSMC are on the order of like AND gates or full adders. They depends a lot on voltage and frequent and which library you choose and so on.

But generally, you can typically have about like 10 or 20 or 30 of these in a clock cycle sequentially. So these parameters are very, very fast, like ten picoseconds or something like that. And so as a logic designer, in principle if you literally just had register and then AND gate in a loop like that, you could get an insanely fast clock speed, like more than four or five, six gigahertz, something like that.

But if you take this really simple circuit and you look at the area you're spending here, this is called one gate equivalent in size, so unit of one in area, and this thing is unit of eight in area or something like that. And so this is just, again, almost all of your cost has been this synchronization or communication cost compared to the actual logic. And so this would be a case where you've gone too far.

You've made your clock speed really, really fast at the cost of spending almost all of your area on pipeline registers. Interesting.

or throughput rather. Yeah. The it hurts your throughput in fact because like like the the throughput of your chip you can think of as the product of how much I can get done per clock cycle, which is based on this area efficiency thing times how many clocks I get per second.

have deterministic clock cycles for like high high frequency trading? Yeah. So I mean, firstly, like, let's consider the sort of the business case for an FPGA versus an ASIC.

FPGAs and ASICs use largely the same conceptual model, which is that I have a series of gates built from ANDs, ORs, six ORs, those very small primitives, connected together with a fixed clock cycle, and connected together with wires that are running in a fixed clock cycle. So anything you can express in an FPGA, you can express in an ASIC too, and it'll be about an order of magnitude cheaper and better energy efficiency on an ASIC than an FPGA. The trade off is that the first FPGA costs you $10,000 whereas the first ASIC you make costs you $30,000,000 because it requires an entire tape out.

So the business use case for an FPGA would be that I want something that has this very deterministic latency fast runtime and high parallelism, but, you know, I I'm gonna change it very frequently. Can change what I do every month or something like that, and so then I don't wanna pay the tape tape out cost every time. Now how does an FPGA actually implement it sort of like it emulates the ASIC programming model, but in fixed piece of hardware.

And so how does that work actually? So what it has at the base is it's got the two components we just talked about. It's got these registers as storage devices, and then it's got these, these are called LUTs, lookup tables, which actually provide all of the gates.

And then we're gonna see even the sort of the third component. We then have sort of a swarm of these registers and lots. And all of these are available and then they are connected by sort of this big set of sort of MUXes.

So in front of every single one of these, we've got something like one of these MUXes, which selects one input from everywhere else, sort of selecting from all of these different things. We've got a whole bunch of different options feeding into all of these things. So so what what what this allows is essentially a when I program my FPGA, I can say that I'm gonna take all of these components and I'm gonna sort of superimpose on top of this a particular wiring which, like, goes through this lot and then feeds into this lot and then goes to this register and then feeds into this lot or something like that.

So what I've drawn in orange is you FPGA means Field Programmable Gate Array. The orange is what has been programmed in the field, whereas the white is all of the wires that must exist in the FPGA in order to make the device in the first place. What does it mean to be programmed into a field?

Like programmed in the field. So like the device is being deployed in a data center, it's sitting in the field and then you can come and program. Not field as an electric field.

No. Field as in like out there in the world field. Okay.

how yeah. How? Like like, where where are the wires that made that happen?

Guess this yeah. So I I I got a little bit, like, lazy in drawing all of these. Every single device here has a MUX sitting in front of it, which can select from all of the, like, nearby, circuits that are available.

And so the actual configuration of the FPGA is like amounts to it, it is the MUX control. So in this MUX here we have the data inputs, and then we have the control that selects. And so there's a little storage device sitting next to every single one of these MUXes saying this is where you're gonna source your input from.

Got it. And so programming it consists of configuring every single one of these Muxes. So that makes sense.

So what is happening inside of the lookup table? Yeah. So the purpose of the lookup table so it's gonna also have a little bit of control feeding, telling it what to do as well.

The purpose of the lookup table is to function to be able to configurably take the role of an AND gate, OR gate, XOR, any of those different things. So there's many ways you could consider doing that. The way it is done in sort of traditional FPGAs is to say it will support will be a lookup table will be it will have four bits of input, one bit of output.

How many different functions are there from four bits to one bit? There are 16 different functions. And so you can actually just tabulate this as 16 different numbers.

You've got a table of zero one one one zero zero one, 16 entries. And so what it does is this table is stored in this blue configuration bit, and then it views these four bits as binary, looks up the relevant row of the table, and emits that bit.

take as input. Those are, like, two input functions. Yeah.

So sometimes we have, like, more complicated like, a three input function would be a three way XOR Right. Or a four way XOR. And in this case, how many it just depends how how big it is.

Typical size for LUTs is four input, which is sort of just a sweet spot between there's another computer communication trade off like here. Like, if it if it has too few inputs, then you need to use more LUTs. Yeah.

If it if it has too many. But basically, the lookup table is like a truth table. It's a truth table.

Yeah. And with a truth table, can program in any gate you want. That's right.

Yeah. And so it's a lookup table that just thinks like a programmable gate. That's right.

And so, I mean, one of the things you can do here is you can see why where the rule of thumb that an FPGA is like an order of magnitude more expensive than an ASIC comes from, is to count how many gates would be inside this lookup table. Yeah. So we can view this lookup table essentially as one of these MUXes.

And so it is a MUX with, has to select between 16 different values. And so it is a MUX with sort of n equals 16 options, p equals one bits. And so what we saw way earlier is that this circuit costs like n times p many gates.

And so it costs like n times p equals 16 and gates and also 16 oars.

Yeah. Exactly. The mux.

The mux is the core of The mux that goes into the lookup table. So the lookup table itself, can think of as being actually a big mux that selects from all 16 rows down to one output. Yeah.

Okay. See. That is the lookup table.

But the way you've drawn it here, there's a MUX and then a lookup table. It's it's MUXes all the way down. So, I mean, there's a there is a second MUX that is inside here.

This MUX is is this MUX. Got it. Okay.

still a four bit value. Yeah. So I've selected four bits from the soup.

Right. And then and then I use those four bits to select which entry in the lookup table Yeah. I am gonna use.

Right. Okay.

Those four bits go into the second mux, which is inside the lookup table. So actually, would say in yeah. In this case, these registers are single bit registers.

So if there are eight, like, eight nearby registers and lookup tables, then I have eight bits total coming in nearby. I select from eight down to four different values, so there's actually like four different MUXes, one associated with each of these input, like little marks associated with each of these input bits. Each of them is selecting one out of eight.

And where are those eight coming from?

Right? Yeah. So there's there's sort of like you get flexibility in a local grain thing.

There's a sort of nearby neighborhood where you can select from, but then more grossly, like more coarsely, longer distant connections, they they form an opinion on that. Right. Yeah.

And the reason it's 10 x slower is why? So if you look at the cost of, like, building this lookup table, it's, like, 32 gates. Yep.

And then it can give me the equivalent of, like, what's one an interesting thing I can do here? I can do a four way AND gate. And so that's like, I am using 32 gates of lookup table to sort of implement, like, a four way AND means, like, what is a four way AND?

I would do, like, AND, AND, and then AND of AND. So to implement like, this is a a socket that I could implement in an ASIC directly using these three and gates. But using a lot, I can also implement it, but it's gonna take, like, these 32 gates instead of three.

which is just to, like, write out the gate. Yeah. Like, to to, like, place down the polysilicon and the and the wires and so on.

Yeah. Yeah. Interesting.

So you can actually design a CPU that has deterministic latency as well. And in fact, like, the the processors that are inside a lot of AI chips actually also have deterministic latency too. Grok has advertised this.

TPUs have that in the core as well. The challenge is getting sort of deterministic latency and high speed at the same time. And so where does the nondeterminism in latency come from?

Nondeterministic latency comes from specific design choices in a CPU. It's actually possible to remove those design choices and make a CPU that has deterministic latency. Those are not very attractive in the market, and so people don't make those CPUs anymore.

But actually, in some sense, deterministic latency is maybe a sort of a simpler designing starting point, and then and then, like, some chip designers have added things in to to be nondeterministic. To take a concrete example of that, probably the most important example is on a CPU just like the CPU cache itself. So in a CPU, have the the CPU this is this is the CPU die itself, and then there's a memory off on the side.

This is the DDR memory off on the side. And then you have a cache system here. Inside it is the cache.

That sort of remembers recent accesses to DDR and stores them. And so I'm running through my CPU instructions, every time I have an instruction that accesses memory, it first checks in cache, was was the data stored in cache, and then if not, it goes it fetches out to DDR. Yep.

This is a huge optimization. The cache is like two orders of magnitude faster than DDR. If you never if you never use the cache, basically all programs would run a 100 times slower.

So the presence of a cache is absolutely necessary for a CPU to to run at reasonable speed, but whether or not you get a cache hit is dependent on the ambient environment of the CPU, like what other programs are running, what has run recently, what is the random number generator inside the cache system doing. And so that is a big source of nondeterminism the runtime of a CPU. So this is sort of the memory system for a CPU.

The big thing that you can do differently is instead of having the hardware say, I'm gonna read memory and then decide the hardware decides whether or not it comes from cache or not, you can actually bake this decision into software. So a different design philosophy is to, and you see this in maybe for example TPUs. The the TPU instead has I mean, I'll draw the same diagram, but I'll call it a scratch pad.

And so the the main difference is so this would be like a TPU and then like HBM, in this case, rather than DDR, but it's still an off chip memory. And instead of the software saying first access memory and then the hardware decides, you've got some instructions that go here. This is like one kind of instruction, And then a totally different kind of instruction that goes to HBM.

And so this style is generically known as Scratchpad instead of cache. The key distinction being that you have one kind of instruction that says read or write Scratchpad and a totally different instruction that says read or write HBM. So is is Scratchpad being the cache?

Yeah. This this thing here is the is the Scratchpad.

I think it's a fair way to describe CPUs. Like, just the amount of parallelism. Like, on a CPU, the amount of parallelism you get is about a 100 cores times maybe, like, 16 way vector unit.

die area of a CPU if there's as opposed to If there's so few cores, am I spending all the there. Yeah. The cores are just much bigger and more complicated.

So, I so I guess we should compare a CPU core which takes up one hundredth of a die I mean, to a LUT. Like, a LUT is just only these 16 gates. Right.

So, like, it's clear why there are so many more LUTs in an FPGA than cores in a CPU. But then sort of maybe the the like, why are there more CUDA cores, for example, than than than CPU cores? Think would be like, why what's the difference between a CPU and a GPU or something like that would would would be a a big difference.

Inside the CPU, you have so one big use of the sort of the top unit uses of area inside a CPU are the cache. Very little is actually the ALUs, like mostly it's like these register files rather than the logic units. And then both of these things have equivalents in GPU, and so that's not a big difference.

But the thing that does not have an equivalent in a GPU is the sort of this branch predictor. And so there is a whole big area in the CPU which is sort of just a whole bunch of predictors that are saying when will my next branch be and where's the branch target for that. And so stripping a lot of that out as well as sort of making these register files tighter in a sense is driving a lot of where the GPU gains are And what is the purpose of the branch predictor to execute both branches at once, or what does it do?

So the issue is that when I've got a series of instructions like like instructions instructions instructions instructions, I if I have a branch like here, if this instruction is branch, The the actual processing step of processing an instruction takes a really long amount of time. It takes, like, maybe five nanoseconds or something like that. So, like, the time to actually notice that I've got a branch and then, like, evaluate the boolean whether it's true, and then update the program counter to the new target, and then read from the instruction memory for that.

That could take actually five nanoseconds to finish. And so in reality, this may finish somewhere down here. I don't want to but, like, I want to run a clock speed that is much faster than what five nanoseconds allows.

Like, five nanoseconds is 200 megahertz clock speed. I would like to run at one or two gigahertz or something like that. And so I need to run other instructions while the branch is is being evaluated.

And so I, like, I really just wanna keep running the following instructions that happened after me. But that might have been wrong. If the branch ended up being taken, then I need to know that instead of evaluating these instructions, actually need to jump to wherever the target is and run these instructions instead.

And so the purpose of the branch predictor is genuinely to predict based on before you even get to the destruction to be like five cycles earlier to predict there was gonna be a branch that's gonna happen.

is exactly the co location in some sense of the memory. That's right. That's right.

Yeah. Yes. Maybe maybe that actually isn't a big difference.

Clock speed is quite high on a chip because that drives higher throughput. When we compare a GPU running some workload, it's running batch size thousand or something like that, whereas the brain is not running batch size thousand. There's only one of me.

And so you could sort of imagine saying, well, take a GPU and instead of running it at gigahertz, run it at megahertz or something like that. And that would start to look maybe a little bit more like sort of equivalent things that you're talking about in the brain. There is, in the way that silicon works, that does not give you an 1000x advantage in energy efficiency.

So what it ends up looking like is you like, you sort can of just end up running this circuit once to stabilization, and then it'll sit idle for a long period of time. It doesn't consume a lot of energy while it's sitting idle because most of the energy is consumed in sort of toggling bits from zero to one and back. So actually, let's let's talk about the energy consumption of of a circuit like this.

The way to think of a bit being stored is you've actually deposited some charge in in a capacitor somewhere, sitting somewhere in in the chip implicitly. So it it becomes charged when it is bit becomes a one, and then it becomes discharged when it next goes to a zero. Yeah.

And that cycle of, like, charging the capacitor and then dumping that charge out to ground, that is where the energy is consumed. This is called the dynamic or switching power. This is most of the energy consumption of a chip.

There is some other energy consumption just coming from the fact that insulators aren't perfect insulators, but we'll just discard Most of the energy consumption actually comes from just the charging and discharging of, like toggling from zero to one and back to zero. Right. So if you run a chip much slower and you only clock it once every thousand clock cycles or something, you will have a thousand times fewer transitions.

It'll be about a thousand times less energy consumption, but but like but not a substantial advantage in energy efficiency.

Yeah. So I mean I think there's sort of a high level organization principle that is different. And then there's sort of inside the cores what are different.

But we'll look sort of outside the at the high level, so we'll take a GPU and a TPU and what does sort of the top level block structure look like. If you think of this as the whole chip in each case, the organization of the GPU is mostly a bunch of almost identical units, which are these these are the SMs. And then they've got an L2 memory in the middle, and then a bunch more of these SMs on the bottom.

And so there's sort of this fairly regular grid of cores. And then, like, if we look at a at a TPU in in comparison, you you end up with much coarser grained units of logic. And so you you end up with something like some large number of maybe like maybe just a few matrix units.

These are the these are the big, like, systolic arrays. And then in the middle, you've got some vector unit. And then you've got your matrix units at the bottom.

So now, sort of like matrix units with a vector unit in the middle, sort of this is the whole TPU chip. You can sort of think of scaling this thing down into a really tiny unit with a smaller matrix unit, smaller vector unit, and that is sort of what an SM is. So sort of at a very high level point of view, the GPU has a lot of tiny tiny TPUs sort of tiled across the whole Oh, interesting.

registers and work schedulers and things like that? Why don't we just like make a huge thing and like amortize those costs across the whole thing? And, I mean, I think this shows up in how large you can grow things.

We've we've sort of seen this theme, like, especially with the systolic array where largest systolic array amortizes the register file costs better. Yeah. This sort of design allows you to have larger systolic arrays.

This whereas the sort of GPU design constrains you to having small units of everything. There is a trade off, however. There ends up being, because of this sort of coarse grained separation of things there, you need to move a lot of data from the vector unit to the matrix units.

And so, like, you need to move a lot of data through a sort of, like, lines of parameter here. Whereas if you sort of look at the equivalent thing here, you've got vector units everywhere, and you need to move data through this line, through this line, through this line, through this line, through line, through this line.

Which, I mean, is also a saving. Like, it's an energy setting. So so data ends up moving like, if you can operate entirely within s an SM Yeah.

The data movement is much smaller. But then the moment you want to operate across SMs That's like, it it becomes sort of more more complicated and expensive.

Yeah. We've talked publicly about something which we call a suspendable systolic array, which in some sense you can think of as big systolic arrays that can be small systolic arrays as well. Cool.