-: Hey, welcome.
And in this video we're gonna have a look
at how you can use Python web sockets
via the OpenAI Realtime API,
to make a AI chat bot,
which you can communicate with
by giving it a call.
Now there's a couple of prerequisites.
You'll need a Twilio account,
a Twilio number, an OpenAI account.
You'll need OpenAI Realtime access,
and you'll also need a tool called ngrok.
And ngrok is a useful tool
because it allows you
to forward the webhook requests
that you're gonna get
from Twilio to a local number
for development testing.
So go to ngrok.com and make sure
that you download this.
It's actually free to use,
so you can just use that for free.
And after you've got
that binary installed on your machine,
then what you need to do is
go into the repo that will be
inside of our GitHub repository.
And then what you're gonna need to do is
go into the EMV doc example
and make a copy of that and make a .emv file.
And then make sure that you add your OpenAI API key.
I'm not gonna add one
because mine already technically exists.
I've already got one on the terminal.
Now this will load the environment variable
and it will use the OpenAI API key
and we are going to load a fast API server on port 5050.
Now I've put a system message here that is
for a charter surveying practice
and you could change the system prompt depending upon
what you want the chat agent to do.
You can also change the voice
and the types of events that we listen to and respond to.
Now the impactful thing happens
inside of this /incoming-call,
we actually have a a request, we make a voice response
and then what we then do is
we then connect to the media stream.
So we've almost got a web socket connection
directly to the media stream.
This will then accept that web web socket connection
to the client.
Then we will set a new web socket connection up to OpenAI.
So you've got two web socket connections here.
We'll then initialize a OpenAI session
and then we've got two functions here
that we'll both receive from Twilio and send to Twilio.
And so the send to Twilio will actually
be getting the information from the OpenAI's web socket
and it will be getting the audio deltas
and sending that directly to the web socket.
And the web socket will then receive that
and it will then send that directly to the Twilio.
And the way that that works is basically we are gonna be
adding in these audio chunks
and then we're sending them back.
And then that's basically how it works.
And so you've also got an ability
to truncate some of the messages if someone speaks.
And we send these via the web socket using Twilio
and then we basically clear those as well.
Now the send mark is quite important.
The send mark is how we send information directly to Twilio.
So if I scroll up here,
you'll see they're actually sending some information
and the information that we have here
also contains the audio deltas.
So that marks the end of the conversation.
And that's basically it.
So how do we run this?
First you need a Twilio number.
You need a Twilio account.
And then you will also then need to
basically go to this active number section
and buy a number.
Once you have a number,
it will appear in the active number section.
Then after that, then what you can do is
you can click into your number.
And this is where we then need
to configure the number using webhooks.
So for example, when I do ngrok on the terminal
and I do this and I'm doing HTTP 5050,
it will give me a public endpoint that I can then expose
so that we can share that with the webhook.
And then we're basically gonna put that there
and then we're just gonna put the /incoming-call,
which will basically accept the call connection.
Then we're gonna click save configuration.
Then the next thing you now need to do
is you just need to start your Python server.
So make sure you go into the Python and type Python main.py.
And then what you, what you now have is a fast API server
that's running on ports 5050 locally,
but we've actually proxied that in to using ngrok.
So we've got this publicly accessible URL,
which is then gonna port into 5050 on the local host.
So now when we actually do phone conversations
and we call this number,
we're actually gonna be interacting
directly with the Fast API server that you have locally.
Now this is a really great experience.
I really recommend trying this out
and giving the number a couple of calls
and I've done this in the past with my uncle
and we were sort of playing around with
how we could improve the prompt.
And we found that basically by listening to the calls
and like interacting with it, we were just iterating
through on this system message prompt
and trying to figure out all of the different ways
that it was going wrong.
Just be aware that there is a token per minute
of around 20,000
so you might hit rate limits if you're on the usage tier
of one at this point in time.
But hopefully this gives you a good indication
as to how you can start using this.
So you've got your fast API set up,
we have a proxy with ngrok
and we've added that to the phone configuration.
And then we've forwarded on the...
Added on the extra /incoming-call
so that this will then receive the call.
We get a voice response,
and then we also connect to the media stream.
And the media stream then handles the connections
between OpenAI and Twilio.
Cool, I'll see you the next one.