VAEs Are Energy-Based Models? [Dr. Jeff Beck]

Geometric deep learning is a big part of the stack if for no other reason than when we talk about modeling the physical world. That means incorporating the symmetries that exist in the physical world. We're highly motivated to employ a lot of those methods and techniques.

world is translation invariant. The world is rotation. Well, not really because there's gravity.

But in principle. There is a principle axis, but it's certainly rotationally invariant in the x y plane. Yeah.

And if you wanna have a good model of the world as it actually is, it should incorporate those features. And of course, you can discover it in a brute force y way, but the mathematician in me really wants to build the symmetries in. And fortunately, we've got a lot of great tools that were developed over the last several years that can do that.

What's your view on agency? If I'm being like an FEP purist, I have to sort of say like, oh, well, there's no difference between an agent and an object in a very real way. Or at least there's nothing structurally distinct between how we model an agent and how we model an object.

It's really just a question of degrees, right? An agent is a really sophisticated object, right? It has internal states that represent things over very long time scales.

It has sophisticated policies that are context dependent, which is basically saying really long timescales again, and things like that. Yeah.

your definition of an agent is something that executes a policy, then anything is an agent, right? A rock is an agent, right? A policy is an input output relationship.

When many people talk about agents, they're a additional elements that I think have a lot to do with how the policy is computed. So for example, when we think of how the difference between us and really amoebas, we often cite things like planning, counterfactual reasoning, goal oriented behavior. We're specifying things that are specific mean, that are all related to how it is we compute our policies.

There are latent variables that represent policies that are compatible with well reinforcement learning. And that's the defining characteristic of an agent. But you could very easily just say from an outside perspective, if you can't look at how someone or something is doing the computations, if the only thing you observe is the policy, does that mean that you can never conclude that something's an agent?

And I would say no. You'd still like to be able to conclude that this is an agent, even though the only thing I ever get to measure is its policy. But do you think we should have some notion of the strength of an agent?

The strength of an agent. Or how is this like a measure of agency? Is that what you Yes.

Yeah. So I mean, I think you could use notions of transfer entropy and things like that in order to estimate the timetable for which something is incorporating information, or the degree to which it exhibits a context dependent behavior and things like that. And that would be a pretty good measure.

Now, is it normative? No, it's not. But it is a measure.

And you could use things like that. But at that point, you're really just talking, again, about policy sophistication, not does it have a reward function. Like, is it actually executing planning?

Yeah.

part of the mass of things going on around them. They are just obviously disconnected from the locality. So the trick is that, Okay, so I've got this agent, and I know exactly what it does.

It takes into account information. Internally, it rolls out a whole bunch of future consequences of various different actions or plans that it could take. It selects the best one, and then it executes it.

So all of those variables that occurred inside, from the outside perspective, it just looked like a function transformation. Unless I'm somehow going in and recording and somehow demonstrating the fact that the manner in which it is calculating its policy involved doing those rollouts, I wouldn't be able to show that it's actually doing those rollouts. I would just be able to conclude it has a really sophisticated policy.

So can you conclude that something isn't so the question is how do you identify something that is actually doing planning? I think that's a really hard question as opposed to having an incredibly sophisticated policy.

It does. So suppose I coded it up so it was doing all of that planning. It gets its inputs to some crazy massive Monte Carlo tree search, picks the best policy possible, and then executes it.

Now, you don't observe any of that. Right? Because you know what's going on, you could say, oh, well, it's clearly executing.

It's doing planning and counterfactual reasoning. It's going on. Like, look, there it is.

Because you coded it, so you know it's doing it. But if you're looking at it from the outside, right, if you don't know what's happening inside, all you have access to is, oh, here's the action that it did given this long series of inputs. And so it's really hard to identify what, you know, something as an agent per se from the outside, you kind of have to know what's going on inside.

This by the way is why I don't think that like, you know, these sort of prediction based approaches to like AI are necessary, you know, you could sort of say, well, it's not really doing anything even remotely agentic unless it's executing and doing planning counterfactual reasoning. So like your chess program is like, oh, clearly it's doing some planning and counterfactual reasoning because you know it's doing it.

Yeah. So in my mind, that is the fundamental feature of an agent. If you can show that it's engaged in planning counterfactual reasoning, and then it's definitely an agent.

My argument is just simply that that's hard to do unless you crack it open and see what's going on inside. Now, you could take a pragmatic view and say, well, if the simplest computational model of the behavior, model it as if it was doing planning and counterfactual reasoning, then you can draw an implicit conclusion that, oh, yes, well, I may as well say it's an agent. And that's kind of the approach that I've taken.

So one of the things that comes out of the physics discovery algorithm is that you apply it to agents, and what do you get? Well, you get a model. Now bear in mind, I called them all objects before, and I didn't change anything to make it special to an actual agent.

But what I do have the ability to do because of the model is I can look at the internal states associated with that object that I want to call an agent and look at how sophisticated it is. And that degree of sophistication is what allows me to say, oh, well, I'm going to go ahead and say that and I like the whole idea. It's a great idea.

Let's have a metric. Right? And I'm sure it would be something that would effectively be transfer entropy or something like that.

But we have this metric on, well, how sophisticated were the internal states that were necessary in order to generate this output? And if it's above some threshold, we'll call it an agent. I don't like thresholds.

But, you know, we just sort of say a degree of agency, a degree of sophistication.

it's as if it's the cause of its own actions. Yes. And that's why this as if phrase comes up a lot, right?

I mean, it's important to remember that no matter how clever your model is, and no matter how clever your approach is, and how clever the words are that you use to describe it, a lot of this stuff is as if, right? This is the best model, right? This is why, like, repeat this over and over again, grind it into the students, right?

Is that science is about like prediction and data compression and like nothing else. And the same thing is going on here, right? Just looking at behavior, you'll never know for sure in any meaningful way, like whether or not it's just doing a function transformation or whether it's engaged in planning and counterfactual reasoning.

But if your best model of it, if you sort of say, well, I tried to model as a function transformation, but God damn it had a lot of parameters. But then I tried to model it as something that was just doing Monte Carlo tree search on the inside and giving the answer. And that had 40 parameters.

And it's like, well, that's the model I'm going go with. And now I'm going to call it an agent. If we had a physical agent in the real world that was doing all of this planning and so on, would that have some kind of primacy to the computer simulation of agents that were doing all of this planning?

Oh, is this like if I uploaded my brain onto a computer and didn't connect it to the world, would it still be thinking, even though it's doing all of those things? Is that the idea here? Or am I the way That works.

So yeah, let's say a high fidelity computer simulation of Jeff.

Oh, I wasn't expecting to say that. Because I'm the agent. And if you upload it, no, I don't know.

So if you do a high fidelity computer simulation and you put it in my body, then I think I would have to say it's an agent. Yeah. Right?

If it's doing exactly the same I mean, is like the standard. It's doing exactly the same calculation from a purely phenomenal auto perspective. It's the same.

It's indistinguishable. Okay. So agents need to be physical.

So I do believe that an agent needs to be physical. Absolutely. I believe you can have a model of agency and not have an agent.

Right? You can put that model in a computer and run it and make predictions as to what an agent would do. And it might even be 100% correct, but I still wouldn't call it an agent.

But again, this is like getting into philosophy. And philosophy frustrates the Bayesian because philosophy is not probabilistic. Right?

Philosophy is really about drawing clear lines and distinctions. And in my world, those don't really exist. Right?

There's everything has an error bar. There isn't a clear delineation between an object and an agent. From this modeling perspective, it's really just a question of degrees.

And philosophy is terrible at handling questions of degree.

forced me at the point of a gun to put a measure on agency, it'd probably look a lot like that. Yes.

then you have written an energy function in weight space, and following you it to its energetic minimum. The advantage of using an energy based taking an energy based approach as opposed to taking, say, a straight up function approximation approach is that an energy based model comes with something that's kind of like an inductive prior. An energy based model if you're doing function approximation, you're basically saying, there's any mapping from x to y.

X is my inputs, y. But any mapping is out there. I just want to figure out what it is.

Now in an energy based model, you're effectively placing constraints on what that input output relationship can be. I like thinking about the distinction between an energy based model and a traditional sort of feed forward neural network has to do with where your cost function is applied. So in traditional neural network, you take in your inputs, you got your outputs, and the cost function is just a function of the inputs and the outputs.

And the only thing that you're optimizing is the weights. In an energy based model, there's another thing that your cost function operates on. And that's something one of the internal states of your model.

And as a result, in order to figure out what the best approach is, you actually have to do two minimizations. One that finds the energetic minimum associated with part of the cost function that operates on the internal states, like the hidden nodes of your network. And then one that is the prediction.

That is your effective prediction error. This is very much consistent with the approach that a Bayesian would take. You a prior probability distribution, which gives you an energy function over every single latent variable in your model.

And you are optimizing with respect to all of them. So you take a probabilistic approach. Examples of this are like a variational autoencoder.

A variational autoencoder, I think, is the best example of the most commonly used energy based model out there. Why? Because you have an encoder network.

You have a decoder network. Right? And your cost function is based on the difference between inputs and outputs, right?

So that's just like a yeah, it's fine. That's still a regular but it also is how Gaussian. And it well, it depends on what flavor of VA.

But you also have some part of your cost function is a function of the actual internal representation. In a traditional VAE, it's how Gaussian is it. You want that internal representation to be as Gaussian as possible.

If it's a VQ VAE, then it's a mixture of Gaussians. But it's still a cost function that is applied on the internal states as well as on the inputs and outputs. Very cool.

So a VAE is a fairly canonical example of an energy based model.

right? I mean, yeah, you're treating some of the weights of your model as if they're latent variables, right? Because when you show a new input, you're allowed to change some of the weights without looking at the output, And so what are you doing?

Well, you're treating the weights as latents. Now I think that which makes it a great trick, in my opinion. It's like, oh, great.

Yeah, they're moving in the direction of energy based models. I love it. The only thing I don't like about test time training is the vast majority of the training that is done.

So in a traditional energy based model, you always find the minimum with respect to the latent variables, these extra weights, which in the case of test time training is the subset of weights that you're allowed to change during test time. When you do the training for a traditional energy based model, you're allowed to make those changes throughout the entire course of training. The way that we're often doing test time training these days is we just do regular old neural network learning.

And then finally, when we get to the deployment phase, then we suddenly turn on these additional latents, which are basically some of the weights of the network, And we do an additional bit of learning at that point. This seems monument. Now, again, not an expert here, but this seems unwise to me.

And the reason it seems unwise is because you didn't train the original network with that on. You trained it in a completely supervised way. Yes.

Now I'm sure that people are aware of this, and it's been addressed in the literature. But I'm not personally aware of that. I don't think that's how it's used in practice, Supra.

We should also introduce this term transduction.

Yes. Yes. Tell me about that.

My take on it is that an energy based model and a Bayesian model have a lot in common, right? In many ways, like energy, I mean, well, in physics, right? Energy is log probability.

Now, of course, there's the normalization factor that you don't need to worry about if just minimizing energy. And so the difference between, which is sort of like, in a Bayesian framework, that's like saying, well, I'm not actually going to treat some of these latent variables in a probabilistic way. I'm just going to do maximum or map estimation on some of my variables and just be Okay with that.

And that's one way to interpret the relationship between an energy based model and a properly Bayesian model. There's a happy medium here though, right? And the happy medium is you can still treat it as if it's, you you don't have to just minimize the energy function, but you can calculate the curvature down there too, do a Laplace approximation and call yourself a Bayesian again, right?

Yes, there is more computation involved, but we've got a lot of great tricks for making that totally tractable.

Regularization term, I think, the short answer, right? No. So there's an and and if you're being very, very, very pedantic, the difference between an energy based you know, minimizing energy and minimizing free energy is that free energy has this additional entropy penalty term.

Now, if you're just doing maximum likelihood estimation, if you're minimizing your energy function with respect to some particular just well, let's pretend we're only worried about one variable. And I'm just gonna, like, get a point estimate and call it a day. Do, like, you know, some kind of map estimation to get to get that that one thing.

There's not that big of a difference, right? Because there is no probability distribution over the latent that allows you to compute that regularization term. But that's the only difference.

It's are you regularizing or not, is, I think, the easiest way to think about it. So Lakun is a big advocate of JEPA.

joint embedding and prediction architecture. There we go. So what's the joint embedding bit about?

Well, the joint embedding bit about is, well, I'm going to take my inputs, I'm going take my outputs, and I'm going to embed them in some space. And then I'm going to learn a prediction between the two embeddings. And that's a great idea.

It's a great idea because it has some of the flavor of what we would like to get out of our models. In many situations, I should be very particular about this. In many situations, we're not interested in predicting every single pixel on the image.

We want to get maybe something that's a little more gestalt, a little more high level, a more conceptual understanding of what's going on. And so emphasizing the goal of predicting every single pixel, which is what's typically done in generative modeling right now, might lose some of the power, the abstractive power of some of the networks. And so the whole point of JEPPA, as I understand it I'm sure there are other points is that you're gonna compress your inputs and compress your outputs and then do all the learning in this compressed space.

Love it, right? Science is about prediction and data compression. Let's make that compression explicit on the front end and the back end.

The downside of this approach is that it doesn't work out of the box, right? Because it's very easy to find a compression or an embedding of the inputs and an embedding of the outputs for which prediction is perfect, which is to basically make both of them zero. And so you have to do some other thing.

Other tricks need to be employed in order to make it work. Yes. Yes, I remember Lecoom was talking about this.

contrastive thing Yeah. This, by the way, is what he should have won the Nobel Prize for, In my opinion. Yes.

Because the whole point of the wake sleep algorithm and contrastive divergence was that, oh, it's actually biologically plausible.

There's an old school way of accomplishing the same thing. And that is to do all of your it's called pre processing. And this is something that a lot of people do.

You take your data. In fact, we do this all the time with vision language models. So we want to use an LLM, we want to predict images.

So what do we do? Well, the first thing we have to do is tokenize the image. Right?

And so what do we do? We run a VA. Then we do the preprocessing.

And we do it by the preprocessing step is completely independent from the actual algorithm that's going to be tasked with solving the problem of interest. And that's not something that we necessarily have to stick with, right? It would be very nice if there a way of again, jointly we're getting right back to JEP again.

What we'd like to do is we'd like to choose our preprocessing algorithm in a manner that not a priori, not do it first. We'd like to choose the preprocessor that works the best in this space. And I think that that's the ultimate motivation for a lot of this work is what's the right embedding.

One of my favorite tricks, course, I pre process the VAEs all the time. In fact, time someone hands me a new neural data set, the first thing I do, I'm not ashamed to admit, I run PCA on it and pass it through a VAE and then sort of take a look, right? It's the first thing you do with your data because it gives you a good idea of what the signal to noise ratio is in the data set itself.

Yes. And then what do I do? I subsequently do most of my analysis in that discovered embedding space.

And I don't see a huge problem with that from a purely pragmatic perspective. But it's certainly cleaner to have a single algorithm and approach and not just be stringing these sort of things together in an ad hoc way. Doing PCA PCA is a really great example of this.

There's a failure mode for principal component analysis, which is actually really common in neural data. Because principal component analysis basically says, well, where's the most variability? Okay, I'm worried about that.

And then all the stuff that's not varying very much, I'm just going to throw it away. Dimensions in which there's low variability are not important. Well, turns out that in neural data, the dimensions in which there's very little variability are some of the most important dimensions.

And so pre processing with PCA runs a risk of throwing out the most valuable information in your data set. Yes. And so there's a lot of wisdom in jointly fitting your preprocessing model as well as your inference and prediction model.

so that it compresses the information, but it also maintains enough fidelity to work broadly for different things. Yes. And that is a laudable goal, right?

And I certainly share it, right? The last thing you want to do is, I mean, you know, fortunately, like networks are so big, we don't really run the risk of overfitting as much as we used to. But the last thing you wanna do is train your network to toss information that you might need down the road.

That said, like the vast majority of what the brain does, just like these neural networks, is decide what information is currently task irrelevant. But that's all the more reason to do things in a self supervised or unsupervised way. Right?

That sounds like another philosophical. So yes is my answer. There will always be something left over in the sense that this has been the trajectory things have been going for a really long time.

It's sort of like we get algorithms that do amazing new cool things. And then someone comes along and says, yeah, but it can't build me it can't pull a rabbit out of a hat. And then, of course, what does someone do?

Oh, They figure out the new training protocol, a slightly different architecture, or they just train it to pull rabbits out of hats, and then suddenly it can. And then someone proposes a new challenge and a new challenge and a new challenge. And it's always this game of like one upmanship.

So the question becomes, well, what's the point at which there are no more new challenges? And I'm not entirely certain we're ever gonna get there, right? It may very well be the case that we get, you know, these sort of algorithms that are capable of replicating the complete suite of human behaviors.

And then someone will come up with some criticism like, yeah, but it's not really doing x. It's just faking it. This is just the direction things go because people really do think they're important.

self improving intelligence is a valid one? Yes. I do think that it is.

So I think that one of the most critical missing elements right now is some form of continual learning. At the end of the day, you really want an algorithm that doesn't just learn on the training set and then just gets deployed. You want something that runs around in the world and comes across things that it doesn't understand, and then is able to build append its model in some sense.

Right? So this is like and there are some approaches to this all based on Bayesian nonparametrics and Dirichlet process priors and stuff like that, where you see something that's surprising or unique or different, something you didn't expect. And it causes you to say, need to turn learning on because I got to figure this out.

That is an absolutely critical element that we need to be developing. We are developing that. And it turns out that that's one of the nice things about this sort of object centered physics discovery thing is because it's object centered, if it comes across a new situation that it does not understand, it is capable of instantiating a completely brand new object just to explain this new situation.

Something that would be closer to true artificial intelligence than what we currently have would be capable of building models on the fly to deal with new situations, to taking things that it knows about and combining them in new and different ways. There are approaches that have some of that aspect to it, like GFlowNets from like Benji stuff is a great example of something that, at least in principle, is a generative model of generative models. It's sort of like, oh, I might actually need a new node.

It's time to create a new latent variable because the current set's just not cutting the mustard anymore. Those are things that I think are hallmarks of true intelligence. I don't want to ever make the statement, as soon as it's got that, it's truly intelligent.

I will never, ever, ever say that. But I do think that that is a critical component that needs to be present, right? Is the ability to generate new models on the fly to deal with novel situations and data.

Most of that, as well as the ability to combine old models, previous models, in new and interesting ways. This is actually how the brain evolved. We started out with really simple brains.

And there were different regions. And they solved different problems. And what eventually happened as we evolved is that these different regions of the brain learned to communicate with each other in new ways.

And through that communication, acquired new abilities and then eventually evolved into new capabilities and things like that. I often like to point out to the I think olfaction is like the sense that's not studied nearly enough. It's an incredibly old part of the brain.

And arguably, it's the first part of the brain that evolved the ability to do proper associative processing. Unlike visual space, where there's translation symmetries and all that sort of stuff and things are smooth, Olfactory space, that does not exist, right? It's really, really, really combinatorial and complicated.

And the part of the brain that evolved to solve the olfactory problem arguably is the part that evolved into our frontal cortex. Don't quote me on that. There's a lot of disagreement there.

That's just my take. But it certainly has a lot of the features that we associate with associative cortex, right? Wow, I just said got like six uses three different uses of the word associate in that sentence.

But I think you see what I mean, right? It was all about taking old capabilities, combining simple models and modules to create something that was more complex. And then over time, so that was what made the brain work.

It was all about taking little things that worked and combining them in new and different ways in order to evolve effectively an emergent properties, emergent computational abilities, and an emergent understanding of the world in which we live. And I do think that when we get to the point where we start really saying, oh, this is actually truly intelligent, it's going to have that feature. It's going to have the ability to have a it's going have a modular description of the world, and it's going to have the ability to combine those modules in a way that creates a more sophisticated understanding.

It's like Legos, right? The Lego bricks all connect in certain ways, and I can build all sorts of new and amazing things that were never built before out of them. That's the capability that we have.

And that's the essence of creativity.

Think that's absolutely correct, right? So the specialization is great. In fact, I would argue that specialization is how we got all of this, Right?

And this was I'm pointing at London in case you there was some confusion there. Right? It was it was really about, you know, the interconnected, highly specialized intelligences that are people and their ability to learn how to work together that, you know, gave rise to the technological revolution.

The brain is the same way, right? In my view, it's highly specialized little modules or agents that are capable of being repurposed, reused, capable of communicating with one another in order to solve really complicated problems. But there's always a benefit to specialization.

I don't believe in like AGI. AGI seems like a bit of a misnomer to me. What we really want is not artificial general intelligence.

We collective specialized intelligences.

largely focused on summarizing vast troves of data and looking for correlations that are present in it. I think the next major milestone in this trajectory is experimental design, not just, oh, well, here's some correlations you may not have seen because they're really small. And this is what computers are good at.

They're really good at identifying small but highly relevant correlations. And the next step, of course, is constructing a system that tests these hypotheses explicitly and generates the experiments that will identify fill that in the gaps of our knowledge. And all of this, I believe, can, in fact, be automated in a very sensible way.

I don't see any major obstacles to automating empirical inquiry other than we probably want to place some safety constraints when we start letting them work when we start letting the AIs run the labs. Because you never know. You always have this AI where it's like, well, know, the most effective experiment to determine if this is correct is to set off a nuke.

And that would be bad. Yes. So pure empirical inquiry does run risks like that.

But I think that that's not the biggest issue. I think what we need to do is we just need to have a nice concise framework for saying, oh, look. I'll give you an example.

So we had this problem that popped up a while back. A gentleman we were talking to is you've got robots, and the robot sees something it's never seen before. And so a robot is running around.

It comes across like a beach ball. Never seen a beach ball in its entire life. And what you'd like is you'd like the robot to know how to figure out that it's a beach ball and to figure out what its properties are.

And if you tell the robot, if you see something new, just stop. Then that's no good. What you really wanna do is you wanna figure out a relatively non invasive procedure for the robot to like poke, do what a child would do.

What does a kid do when they see a beach ball? They run up and they poke it and they say, oh, right, yeah. And then it moved And experiments it with its environment for the purposes of identifying the properties of the objects that exist in it.

Now, I do think we probably want to test this out virtually before it's deployed in the real world because you never know. It might very well be that the optimal experiment is to run up and kick it as hard as you possibly can. And we certainly want to avoid that.

But something along those lines, a robot that is able to test the theories that it has about how things work in an online way and learn from those results in an online way is definitely part of the goal.

Mean, offloading your thinking onto a machine, which is something that AI allows, is potentially a big problem. I don't really want to have a situation where humans are reduced to like value function selectors. They're just basically going, oh, no, I don't like that outcome.

Do this instead. I do want to see a future where we have an AI that actually improves our understanding of the world. And simply automating everything runs the risk that you specified, right?

It runs the risk of people becoming couch potatoes that just watch TV and occasionally say, yeah, these chips are no good. That seems like a bad outcome to me. I worry less about that, I think, than some because people are remarkably adaptable.

I mean, they have all these arguments about like, oh, this new technology comes along, and it's going to completely destroy this way of life. And that's going to be awful for people. And it is maybe in the short term.

I think of tractors. Or just go back how many hundred years do you have to go back when 99% of people were involved in agriculture? And now it's like, what, two?

I consider that a solid improvement because it allowed the rest of us to it allowed us to do a bunch of other things that we find more satisfying, that are more interesting. It allowed us to I can spend some time reading a book, don't have to labor in the fields all day. That's the future that I sort of see.

And that's the future that I hope for is one in which all of these artificial agents running around and doing things autonomously are there to free us up to pursue more interesting, to improve ourselves in more interesting ways. But at the end of the day, it's just another technology. At least initially, it'll just be another technology like the tractor.

Now, one hundred years from now, who knows? What will the value of work be if the AIs can do everything and there's nothing left for us to do? I don't think that it will ever be the case that the AIs can do everything.

Like I said, the future I worry about is one where sole role of people is sitting around making sure the AIs aren't going rogue and things like that, which I don't consider a good outcome. I would really like to see human improvement. I envision a future of, I don't know, this is like cybernetic transhumanism, if I'm going to go sci fi on this, right?

Where the technology and us evolve together in a way that's beneficial for both. That's the goal. Are there these dystopic possibilities where like, oh, well, what are humans in a world where, well, what are they?

What humans in a world where everything can be done by a robot? Yeah. You know, that's a good question.

And at the end of the day, right, they end up just becoming reward function selectors. They end up just sort of saying, oh, I don't like this, and I do like that. And they're basically I mean, you end up with this it is another nightmare scenario.

I don't like talking about these dystopian futures because honestly, I think people are too clever. And I think people are too motivated. And people are too interested in how the world really works.

And people are too interested in actually understanding things that they will never stop, that AI will become a partner, not an adversary or a crutch. And that's what I think will happen. But that's a statement more about my belief about humans than it is about my belief about the development of AI.

I am a techno optimist, if you will, not a pessimist. I I believe that we will find a way to adapt to an ever changing world as we have done for millions of years, including one that includes technology that alleviates most of our labors.

Starting a business can seem like a daunting task. Unless you have a partner like Shopify. They have the tools you need to start and grow your business.

From designing a website, to marketing, to selling, and beyond, Shopify can help with everything you need. There's a reason millions of companies like Mattel, Heinz, and Allbirds continue to trust and use them. With Shopify on your side, turn your big business idea into sign up for your $1 per month trial at shopify.

com/ special offer.

How should they make sense of all of this? We are now well outside my area of expertise. So I'm just gonna say that before I say anything else.

AI is developing very quickly, but I am much more concerned about what people will do with the new technology than I am with what the technology will do all by itself. I don't have this big concern about I don't really believe that Skynet's gonna take over, the internet's gonna suddenly become conscious and kill us all, right? In part because AI is not that advanced, but also because we are still in the position where we specify the goals of the system.

And that will likely continue for a very long time. And it will always be the case that these systems are subject to review. We will always keep an eye on them.

They will always at least initially be released in relatively restricted domains and where we're keeping a close eye on what it is that they are and are not doing. So I don't worry too much about the going rogue. I worry a lot more about somebody building it's sort of like a virus, which we already have to deal with.

Like somebody builds some insane virus and takes down the internet. I'm more worried about malicious human actors than I am malicious AI actors because at the end of the day, all of these algorithms, they simply do what they are told. We train them.

We tell them, here's your objective function. As long as we are specifying the objective function and we understand the objective function, we're probably going to be Okay. I think the safest way to deal with AI concerns is to tell people, hey, look, this AI is just doing what we told it to.

We set it up to make really good predictions and to achieve these outcomes. Now, is it dangerous to specify these outcomes without being very, very, very careful? Yes, it is.

This is the whole, hey, Skynet, end world hunger, and it kills all humans. That is a real possibility. But whose fault was that?

The fault was the person who was very, very naively specified their goals. There are, in fact, relatively straightforward ways to specify the reward function that don't run that risk nearly as badly. And the best one is so are you familiar with maximum entropy inverse reinforcement learning?

I like to call it active inference because it's really similar. And so there, what you're doing is you're basically observing someone's policy, and then you're trying to do a maximum entropy model you're doing maximum entropy model on the reward function itself. At the end of the day, what ends up happening when you do this is this is why it's basically just like active inference.

You get a reward function. So you have some organism or whatever, and you're trying to do this for it. And it's got some stationary distribution over actions and outcomes.

It's inputs and outputs of the stationary distribution. That becomes your reward function. Not directly, there's some math involved.

But basically, your reward function is a function of the steady state distributions over actions and outcomes. So we could do this. We could take the current manner in which humans are making decisions.

And we could write down, what is the current estimate of the stationary distribution of reactions and outcomes? So this would include things like everyone's getting this number of people are going hungry, all of the stats that describe the inputs and outputs to our policy decision making. And then we could just ask an AI, your reward function is the one that results in the same outcome that we currently have on average.

And it would execute it. And to the extent that it works, would ultimately result in an AI algorithm that just sort of is like mimicking human behavior, or at least achieving the same outcome that we were achieving before. Now here's the safe way to improve the situation.

You don't say end world hunger. You perturb that distribution over outcomes, and just over outcomes a little bit. And then you evaluate the consequences.

It's all you're doing.