Why Dwarkesh is Wrong about Computer Use + How OpenAI shipped its Jev competitor in 1 Week

There were a bunch of computer use announcements that I think are worth thinking about. We have, Dots, which is the new, sort of personal assistant product, and, that has some really exciting computer use features. There's g p d 6.

1 Soul, which is this amazing new model, that I think is particularly great for computer use because of, sort of the cost and and speed, advantages. I think I think we shared that it's, a fifth of the cost as Astra and a seventh of the cost if you're looking at computer use specifically, which is really amazing. Sorry.

There were so many things I'm trying to sort through it. In the API. Agents API, which now has computer use in it, which is really cool because now developers can build on the same computer use that is is part of Codex and and ChatGPT.

And then there were some demos of our existing computer use features like app shots where you can take the context of something you're doing on your computer and bring it into Codex and ChatGPT really fast, and and then, like, native computer use on your Mac where Roman had it taking screenshots of his app automatically and he could do other things on his computer while computer use was using his applications.

Like, basically, like, is Kubernetes using Decisions API, or are they, like, kinda separate? So what's... It really cool about the Decisions API is it, you know, it it has all these new capabilities.

It does inference in parallel. It doesn't have reasoning. It's a smaller model than the ones we use for computer use.

And so those those capabilities make it really fast. Yeah. They also make it a little bit less good at doing like long horizon sort of sophisticated tasks.

And so I think it's... I would say it's still an open area of research for how we like bring those approaches together. But yeah, I'm really excited to see what people build with the Decisions API.

One of the interesting things is DOTS now have attached personal computers.

Dots are really cool product because each dot has access to its own Linux virtual computer in the cloud, which is different from our other products. You know, traditionally, we've have access to a browser in the cloud or it has access to your own computer, but now you get your own entire Linux computer in the cloud, and so we can run full desktop applications and it can also use a web browser. And so, yeah, you know, I think the powerful thing about computer use and the reason why I think it's so exciting is because it makes it so that the agent can do anything you as a as a person can do because all the software in the world was designed for humans and now agents can use that same software and you can delegate to the agent.

So, yeah, like anything that you would do on a computer, you can ask a dot to do. Yeah. I think what particularly useful is going to really depend on who the end user is and what's valuable in their life.

But yeah, I would just start by thinking about like what are the things that you spend time on and how could you delegate those to an agent?

Yeah. Don't know. For me, something I did recently, I've been working on...

I I subscribed to a meal prep service because I was trying to like eat healthy, you know? And I really like this meal prep service I found because it lets me customize the meals I order to, like, a high degree of granularity. So I can say, like, I want this many grams of chicken and this many grams of of of rice, but it was so complicated.

It took me two hours to do an order, and I found that I could ask ComputerEuse to do it for me, and it did it in fifteen minutes. So I actually saved two hours. It both did it eight times faster than I could, and it saved me two hours on on GP 6.

1 Soul. So those are the kinds of tasks that I feel like are are really powerful.

Anyway, so... I've heard that from our developer experience. They they use it with YouTube a lot.

Yeah. It's really awesome.

Is it model? Is it DOTS? Is it harness?

So all the history plus, what what really just changed in today's announcements? Yeah. On the through line, I guess I've always been excited about automation and helping people automate tasks because then you can, like, save time in your life and focus on things that are more important to you than, like, operating a computer very intricately.

And so, yeah, that that was why we worked on some of those products. I was at Apple before. We we made a company called Sky.

We ended up joining OpenAI, which is really exciting. And I think something that was... And almost like you have to hack around Apple until Apple was like, fine.

Like, we'll just hire you and you can just work on the inside. Right? Like...

It was it was a cool place to get to work. What was really interesting looking back at Sky is we were we were working on computer use there as well and the models were so much less capable. And now the models, just in the last one year, have become extraordinarily capable of computer use.

I think the biggest delta that I see is before they could, like, reliably start tasks, but then they would run into problems, and now they're really good at debugging. They're really good at trying again, introspecting what is and isn't working. And I think we've also brought the computer use...

The the computer use field itself has moved forward. I think we're using more techniques. Now computer use often writes code.

So if you actually look at it in Codecs and you expand the tool calls manually, you can see that it's not just doing one action at a time. It's actually writing JavaScript code that it executes... That that the computer executes to perform sometimes many actions at once, which is a great, you know, speed up and great capability.

We use more accessibility sort of multimodal interfaces, so the model may use screenshots, it may use accessibility, it may use playwright, it can use a lot of different mechanisms based on the task at hand. And then, yeah, the the the model acceleration has been has been just amazing. So, yeah, what's different today, I think we're making computers better all the time.

So I think just, like, one day's difference is is probably a little bit less consequential than, like, even the past month or the past two months. But, I think the the computers in dot is really exciting as well as the the new model that we came out with.

what were the changes with the new model? We actually have a bunch of different ways of measuring it, some of which are on different permutations and configurations of the harness. It's a bit of a complicated story because, know, our production products have, you know, some more safety checks and, you know, those are configured differently based on the the the needs of the of the task at hand.

So there's a lot of ways to to measure it. But I think regardless of how we measure it, we find pretty consistent gains and those gains are sometimes in the harness and sometimes in the model. And, yeah, I was really excited by this result that GPT 6.

products had to spend a lot of time, like, scrolling, you know? So it would, like, take a screenshot. It would try to do something.

It would be like, oh, I gotta, like, scroll down to the next page of results, and then it would take a screenshot, then it would try to do something, would scroll down again. And so I think with with accessibility and other... And and direct access to the DOM and other things like that, now the language model can actually see, like, an entire page or an entire application.

It can write code that can do multiple steps at once, and so I think those have been probably the biggest single moments. There's, like, a lot of tiny ones that are less exciting in comparison, but actually, we do find also that a lot of speed improvements are driven by, like, a lot of little paper cuts that we gotta go in and introspect.

in in in a very optimal representation. Yeah. Yeah.

Exactly. Yeah. It's kind of fun, actually.

If you wanna be, like, really nerdy about it, you can go into Codex, take an app shot by hitting the two command keys. So you grab the content from whatever app you're working with, bring it into into the Codex or Chattypici chat. And then if you click on the attachment and you click on this, like, little tiny button in the top right, you can see the raw text and you see the raw accessibility representation.

And, yeah, we've put a lot of work into... Just dumping everything out. Yeah.

Dumping it out, but also making it token efficient, doing it efficiently, there's, like, a bit of an art to it.

the screenshot doesn't include where the link goes. It doesn't include... You know, maybe you take a screenshot of your calendar, the event titles are truncated, you know.

But when you take an app shot, it gives, like, the language model, like, full context about everything, and that lets it just sort of, like, do do much more. Yeah.

you know, the team's accomplished over the past couple months is that now computer use is, like, faster at accomplishing tasks than, like, the average human probably in in most cases. And I think that the next frontier is to have computer use be, like, literally superhuman in its performance, where it actually is as fast or faster at using software than, like, expert computer users like us. And I think that'll be really consequential and exciting when that happens because I think we'll be able to, all of a sudden, build products that provide just much more real time experiences, and I think it'll also...

Lowering the barrier to entry of... Or the activation energy, I suppose, of using GPUs, I think, will make us start to default to doing certain things in agents that we've become accustomed to doing manually, and I think that's exciting also because it'll save us a ton of time. And I think there's a...

You know, there are a lot of different little paper cuts and bottlenecks that are sort of standing in the way of that. I think that there is... Yeah, there's things on the model side, there's things on the inference side, there's things on the harness side, there's things in representation.

You know, we find that as computer use gets faster, we're increasingly bottlenecked by just, like, the speed of doing an operation. Like, for example, you know, a nontrivial amount of time in our benchmarks of computer use tasks is actually, like, let's say you're automating a task on doordash.com.

Like, a lot of the time is actually waiting for doordash.com itself to load, you know? Yeah.

Then you just write a wait and then you execute the wait. Yeah. Totally.

And and you wanna get... Actually, it's it's actually really important that you get that... Like, you want as little delay as possible between when it finally finishes loading and when you go and trigger the LM to the next action, which is actually itself a statistical science.

Like an event driven way maybe to do that. Possible you want it to JavaScript be event has driven load and JavaScript has load events for... The web browser has load events for web navigation, but there's other types of events that actually really can't be event driven, so there's a lot of complexity.

The one that comes to mind is, like, chatting with customer service. Yeah. Supplies could take thirty seconds, could take three minutes.

Oh, right. I have dealt with so many bots with Codex.

First of all, I'm just really excited that we brought computer use into the agents API. I think this is really great because, obviously, a lot of developers are building applications that wanna be able to work with third party websites and and and and services. And so computer use has this universality to it.

It can work with anything. So now all of a sudden, developers can build using the same computer use implementation that we're building on. I think there's great work to be done if you want to build your own computer use harness, but it's hard.

And also, we train our models on our computer use harness, so there is advantage to using the one that's in distribution for the model. There actually might be a speed and cost and and and accuracy advantage. So I think it's really great for people to get to build on top of that.

And, yeah, you know, I think kind of to to the point that you were making, like, I think we're all sort of still in the process and maybe, like, some of us are ahead of many people in the world of, like, getting comfortable with this technology and trusting it. And so I think it's incumbent on us to sort of build that trust over time by making sure we're building things that are reliable, by building the right kinds of of safety checks, by asking for the user's consent before doing something consequential, like making a payment, by asking, you know, maybe depending on the application, making sure you're only letting it access the websites or applications that it actually needs for the task. So that's, I think, something important to to think about.

and one that we see a lot in the wild, is computer use letting the agent actually test the software that the agent has built, which is far more consequential than it sounds because traditionally, you know, you'd build something in codex and then the... And the codex builds it for you and then you have to test it and you are now like QA for the agent. Right?

So with computer use, you can complete the develop... The software development life cycle where the agent can build software, it can test it. So I have a lot of fun, you know, building stuff, having the agent test it.

By the time it comes to me, it's already working. I have extra fun because sometimes I'm, like, developing computer use itself, and so now I have a computer use agent that's using my computer use agent that's using something else.