-: Hey. So in this video we're gonna learn about
Anthropics computer use
and how to set up their example code inside of GitHub.
Anthropics made a fine tuned version of Claude
and its other models,
where it can click on specific elements within the desktop
using an age agentic-based system.
Claude can now use computers.
This is really impactful,
and we'll see this in the next couple of years
where LLMs will be able to use entire computer systems,
anywhere from coding editors, text editors, browsers.
This is gonna give a range
and plethora of opportunities for software engineers
for automating tasks that were traditionally hard to do
because they didn't have a REST API,
and building web scrapers by hand is quite brittle
and often bespoke work.
We'll need to visit github.com,
go to /anthropics,
and go to /anthropic/quizstarts,
and then you'll see
there's a range of different quickstarts here.
The one that we are interested in using
is the computer-use-demo.
My suggestion is to git clone this repository
onto your local desktop,
and then we can go into the computer-use-demo,
and then we'll have a look at the setups for that.
My suggestion would be to install Docker on your machine,
and then what we're gonna do
is then export the Anthropic API key,
and then we're gonna run this Docker run command.
My API key is already exposed as this environment variable,
so I'm just gonna copy this Docker run command,
and then what that's gonna do
is it's going to download all the images
and it's going to start setting those up.
It will be installing a Linux virtual machine,
and we can see that if we look inside of the Docker file.
They actually use a Debian based image,
and then they install a range of different things
such as some UI requirements.
They install Python.
They also install some network tools.
And also, they provide some other things,
such as the libraries such as Mozilla,
and also installing Office as well.
And as well as that, then they also do some other things
to make sure that Python is set up correctly,
and they make sure that the relevant screen height and width
is set to 768 pixels and has a width of 1,024 pixels.
After a while, that should then initialize
and you can go to localhost 8080.
And then for example, in here on the left hand side,
you've got a Streamlet app, which will accept commands.
So I can say open Office,
and then what it's gonna do is it's gonna run those commands
against a tool functioning agent,
and it's decided that it's going to pick
the image of a screenshot.
And then after that, you can see that the Libre Calc icon
is on the desktop.
And then you've got here
where it's actually got this tip of the day.
And then it's gonna go on and mouse move onto coordinate,
and then it's decided to click on that,
and it's clicked on that okay button.
So you can see it's able to load things like Office,
and this is really, really great
because we can actually utilize this.
What it's also decided to do,
it's decided to left click
and it's gone into the top left column here.
So again, we do things like
close this Office application and open a browser,
and go to Google and search for cats.
And so, it's basically deciding what next steps to do
by taking a screenshot every one or two seconds,
and then what it does is it uses a range of different tools.
So it can use keyboard, it can use mouse movements,
and all of this is done inside of the GitHub repository.
And now it's going and it's initializing Firefox.
And after it's initialized Firefox,
it's then decided that it wants to left click,
and it's gonna then enter into the search address.
The tool use from the computer is to then type in text,
and it's gone to Google, typed in text,
and then it's come back with cats.
My one caveat here is you might get rate limited.
So depending upon your Anthropic key,
this might be something that you experience.
But the computer use is a demo.
It's in beta, so it's very experimental,
but it's definitely a direction that is happening.
And within the next couple of years,
I can see this being a way
that you can interact directly with computers
so that you can automate repetitive tasks
where you have data across different types of applications
that are potentially on that machine.
Cool. Alright, I'll see you in the next one.