Welcome!
In this tutorial, we are going to build a personal assistant that can make phone calls for you.
So let's get started.
You have already seen this AI voice assistant built in and integrated into an annotation workflow in
one of the demos, but this time we are going to do something slightly different.
First we are going to watch the full demo.
Then I'm going to give you an overview of the setup and explain how it works, note by note.
And instead of building it completely from scratch, we are going to use the template provided in the
resources of this lesson and I will show you exactly what you need to change to make it work for you.
This way you will learn how to take advantage of the templates we provide, and also how to adapt blueprints
to fit your own needs.
Because personally, I believe that the best way to learn any ten after you've got the basics is by
modifying, recreating, and rebuilding templates so that the supervisor agent from your workflow can
connect to it and take action.
So now let's watch the demo.
Hi everyone.
Now I'm going to show you my personal assistant agent in action.
Hi, Emily.
Can you please call Mark and ask if we can meet tomorrow at 7 a.m. in the office to discuss the onboarding
of two new clients.
And if that works for him, send him a confirmation email and also update my calendar.
So I should receive a call in a second from my personal assistant.
So the main agent is looking for Mark's mobile number in the contacts data stored in a Google sheet,
and is calling a subagents.
Yep.
Hi, I'm Emily, Damien's personal assistant.
Hi, Emily.
How can I help you?
Hi, Mark.
Damien asks if you can meet tomorrow at 7 a.m. in the office to discuss the onboarding of two new clients.
Yes.
No problem.
That works for me.
Great.
I'll inform Damien.
Thank you.
Can you ask him if I need to prepare anything?
I will check with Damien and get back to you shortly.
Thank you.
Bye bye.
Yes.
So now the supervisor agent is calling the Subagents.
So email and calendar agents.
And I should get a confirmation email and have a meeting booked in my calendar.
So let's check my inbox.
Yeah.
That's great.
And my calendar.
Yeah.
That's awesome.
All right.
So now I'm going to walk you through each node and explain everything.
I communicate with Emily via telegram because it's the easiest way and it's also free.
But you can easily integrate with other apps such as WhatsApp, slack, etc. as you can see here in
telegram, I got a confirmation from my agent summarizing everything.
She also passed on what I asked her as Mark confirmed that she sent an email to Mark, updated my calendar,
and even added a link to make it easy to access.
Apart from the three subagents so phone call, calendar and email agents, I've added a vector database
as another tool for Emily.
So in short, Emily is trained on Google Docs stored in my Google Drive.
So this is my company's knowledge base.
So if I ask her, for example, what is adaptive AI?
She knows exactly what's included in that document and can answer straight away.
This means she can remind me about my agency SOPs past project details, training materials and much
more.
So she is very helpful.
You can add more tools to Emily like want to do research or analyze competitors or create reports.
The very first step is this trigger node which listens for incoming events.
So every time I send a voice or text message to Emily, the workflow is activated thanks to that node.
So when a new event is detected, the trigger captures it and pass it to it along to the rest of the
workflow.
So any interaction with Emily on Telegram automatically starts the process.
Great.
Now, with this in place, Emily is officially ready to listen to us on telegram.
So next, once our telegram trigger has captured an event, this node determines content type.
So acts as a decision maker and analyzes the incoming messages to figure out whether it's a text, voice
note or something else.
So this node uses simple rules to check for key properties in the coming data.
For example, if the message contains a text field, we know it's a regular text message, and if it
has a file ID in the voice field, it's an audio message and so on.
So based on these rules, the workflow branches out.
Text messages go one way.
Audio another.
And errors like unsupported content types take a separate path.
This ensures Emily can handle every input.
For example, if it's a voice message, the workflow prepares to download and transcribe it.
If it's text, it moves forward to process the message.
Search, so every type of input is handled in the right way.
Now when someone sends a voice message download voice file node, takes the file ID from the incoming
audio message and known as the actual audio directly from the telegram.
Great.
So now when the voice message is ready for the next phase of the workflow, the file is prepared so
it can be transcribed into text and the AI agent can understand its content.
This node uses OpenAI's whisper model to to do the transcription, and this model works great even if
there is an accent or some background noise.
So the node takes the audio file, processes it, and spits out plain text.
For example, if I send a voice message saying what's on my calendar tomorrow, the note turns that
into a text message.
This text is what Emily uses to figure out what you want her to do next.
So now let's move on.
The next note.
Combine content so you have the input and it can be text or a transcribed voice message right.
So this note pull everything together.
Add some labels.
So it combines it into a single variable called combined message.
This way no matter how the message was sent, Emily processed it in the same way.
So it figures out the type of message, like if it's a text query or a voice message, and tags it with
message type.
So if the message was --, it adds a little label in source type to know that now that everything's
sorted, we are ready to send it over to Emily's brain.
So the main agent.
Now, Emily processes the message and decides what needs to be done.
So basically, Emily works as a as a supervisor agent and delegates tasks to the three sub agents working
under her.
So first make phone call agent.
If you ask Emily to make a call, she passes the details like the name, phone number and what to say
to this agent to handle it.
Next calendar agent.
So for anything related to your schedule like checking availability, adding, or checking events,
Emily hands it over to this agent.
And the last one?
Female agent.
This one handles emails.
So whether it's sending, replying or setting up drafts.
But Emily is smart, so she she always checks the contacts data, a Google sheet in Google Drive to
make sure the email address is valid before sending anything.
Now, really important part of this agent and how it works is the prompt.
So it's basically the set of instructions that tells Emily how to handle different tasks.
So the prompt is super specific and even includes variables like the current date and others.
So Emily always has the right context for what she is doing.
So as you can see, it's very detailed.
For example, it reminds her to check the contact list.
Before sending an email, or to make sure phone numbers are formatted correctly before making a call,
such as adding a country code.
So once again, Emily works as the supervisor agent and delegates tasks to the three sub agents, and
each of these sub agents is included in separate workflows.
So this keeps everything modular and easy to manage.
Like in object oriented programming.
So let's start with a make phone call agent.
This agent uses VPI, which is a voice API platform to make the calls.
When Emily needs to handle a phone call request, she gathers all the necessary details like the name
of the person and the type of contact, like if it's a friend or business, and specific instructions
for the call, such as ask if they are available to me tomorrow.
So once Emily has all this information, she passes it to the make one call agent, which then triggers
a workflow called make a phone call path.
So this is the make a phone call agent.
This agent uses the API to handle the actual call.
And after the call is made, the response from the workflow comes back to Emily.
So she knows what happened and can continue with the task.
So this setup ensures that each Subagent focuses on its specific job, while Emily oversees everything
and makes sure it all works together as expected.
So next let's look at the calendar agent.
So this is our calendar agent.
So when Emily handles anything related to scheduling, like checking availability or adding events,
she passes the details to this agent.
Basically, this separate workflow is dedicated to managing your calendar.
It processes the request, updates or retrieves the event information and sends the results back to
Emily.
From there, Emily uses the response to provide you with the updates or confirmations.
Now the email agent.
So Emily first checks the contacts data, a Google sheet to make sure the recipient's email address
is valid.
And once verified, the email agent triggers a separate workflow to handle the email task.
So the workflow sends the email and then returns a confirmation and response back to Emily.
Now let's look at the knowledge base node.
So this node is connected to find Convector store, which allows Emily to access and retrieve specific
information stored in database.
So you can store all sorts of documents in pinecone.
This could be Google Docs, spreadsheets like Google Sheets.
Company policies, etc. so you can give Emily access to, for example, frequently ask questions, client
details, SOPs, training materials, or any other knowledge base.
So Emily can retrieve this content and fetch accurate answers by pulling directly from your stored information.
For example, if you ask Emily something about your company or any stored knowledge, this node searches
through the vector database for the most relevant information and sends it back, and the response is
then passed to Emily.
And based on that, she can provide a relevant answer.
So the pinecone vector star node is where all the knowledge is kept.
It stores the information in a way that makes it easy to search and find later.
So all the knowledge is converted into numerical representations called embeddings.
So I can understand it.
Embeddings are small, easy to search versions of your data that can help large language models find
exactly what they need.
And when Emily sends something, this node searches through the embeddings to find the best match and
sends the results to the knowledge base back.
So from there, it's formatted and passed to Emily so she can use it to answer a question.
So this node generates embeddings for the data stored in a Python vector store.
And whenever you add new information to the database, this node processes it, creating embeddings
that make it easy to search and retrieve later.
So it's a key part of keeping the knowledge base up to date.
Now let's talk about the window buffer memory node.
This one.
This node helps Emily keep track of the context during conversations by storing short term memory.
It allows Emily to remember recent interactions so she can handle follow up questions without losing
track of what you are talking about.
For example, if you ask what's what's on my calendar tomorrow?
And then follow up with can you add a meeting at 3:00 pm?
Emily can connect those requests and respond.
However, this is not a persistent memory like the Pinecone Vector store.
Once the session ends, the memory from this note is cleared.
So it's designed for short term use within the same conversation.
Now, this OpenAI chat model is connected directly to Emily and acts as her brain.
So when Emily needs to process a request like understanding a message or deciding what to do next,
this node uses OpenAI large language model to interpret the input and provide a response.
For example, if you ask Emily to send an email, this node helps analyze the message and figure out
the context and decide what to do next.
I've got a quick five minute tutorial on my channel showing how to connect all OpenAI models to your
agents.
So be sure to check it out.
While this OpenAI chat model works with the knowledge base.
When the pinecone Vector store finds the information Emily needs, this node helps it into a clear response
she can share with you.
For example, if you ask about company policies, this node takes the data from pinecone, processes
it, and turns it into an easy to understand answer.
And in this setup, both openly chat model nodes.
This one and this one use GPT for Omni.
So it's very cheap to run.
I managed to achieve this with a complex prompt.
Now this is the response to me node.
So after Emily completes a task or finds an answer, this node sends a response back to you on telegram.
Now go ahead and download all the files from the resources section of this lesson.
Once you've got them, import the first file called your personal assistant into a new workflow.
If you miss the lesson on how to import workflows into an A-10, no worries.
Just simply go to just simply create a new workflow.
Go to these three dots and import from file.
You should see this workflow.
And now the first thing you'll need to do is connect your credentials for some of the nodes in this
setup.
So especially all the telegram nodes.
So if you have watched the previous lessons, you should already know how to connect your credentials
like telegram, OpenAI and any others using this workflow.
In most cases, Naa10 should auto detect and update those for you as long as you have set them up before.
But if anything's missing or not working, just go to the Setup credentials section in this course and
you will find short tutorials showing exactly how to connect each specific service step by step.
So the first node we need to connect is this node to listen for incoming events.
So make sure this node is connected to your telegram account.
Then if we have successfully connected your telegram simply double click on all of the telegram nodes.
So N810 will automatically refresh the credentials for you.
Make sure you connect the same account.
And you have an output node.
So respond message.
All right.
Great.
Now make sure your OpenAI account is also connected.
So we can use the whisper model to transcribe the recording.
Yeah great.
And also here is a brain for our supervisor agent.
All right.
Awesome.
Also disciplinary note to use the vector database.
Great.
Now what we need to do we need to download these three sub workflows, import them separately and then
connect it to these nodes.
So our personal assistant.
So supervisor agent will be able to use them as tools.
First let's import the phone call agent workflow.
All right.
Now let's rename it to.
Agent.
And save it.
Now let's go back to our main workflow.
Alright.
Great.
Now we should be able to choose this workflow in the Execute workflow node.
So is this one for Focal Agent.
So workflow from list.
And we call it.
Agent workflow.
Alright great.
Now you have to do the same thing with the two sub workflows.
So I'm going to create a workflow.
Import the calendar agent.
Do the same thing.
Import from file.
Calendar.
Agent.
Alright great.
Now just double click on this node.
If you have connected your open account area it will be automatically updated.
The same for Google Calendar.
Make sure you are connected to the correct Google Calendar account.
Okay, now let's rename this workflow.
Let's call it Personal
Assistant.
And hit save.
Now let's go back to our workflow.
And let's search for the workflow we just created in this execute workflow node.
Is this one personal assistant calendar node tool.
So create a new workflow.
Import profile.
Updated credentials and make sure you are connected to the Gmail account you want to use for this setup.
Let's call it.
Personal.
Email tool.
Itself.
And let's go back to our workflow.
And the last step workflow.
So email agent.
We call this workflow personal.
Tool by perfect itself.
Now the next step is to create a Google Sheet with contacts data.
So please go create a Google Sheet.
And mine looks like that.
So I have a name, column email address and phone number.
So our supervisor agent will be able to look through this Google sheet and and then find the phone number
we want to call to.
Alright.
Perfect.
Now this is really important.
Also make sure you are connected to the correct Google sheet account.
So our supervisor agent will be able to retrieve this contacts.
Then also make sure you you choose the correct document from the list.
So mine is contact database.
This is how I name this Google sheet.
All right.
Now let's move on.
Now let's start configuring our agent.
So this node agent is really important because it's one of the, uh, is the one responsible for making
phone calls.
So let's open it.
And let's scroll down.
As you can see here, I added a little bit of code, but don't worry.
This is just a simple code snippet.
So this node triggers a separate workflow that's dedicated to handling the phone call using the API.
And we already imported this workflow and connect it to our supervisor agent.
So here we send it a JSON object with all the important details such as who to call, the phone number,
what to say and how the voice should sound.
So, for example, Tom here friendly as a friendly.
So these are dynamic variables which will be passed to our AI voice agent on platform.
So it passes this info to the phone call agent workflow, which makes the actual call through the API
and sends the response back.
So this node acts like a bridge between your supervisor agent and the AI voice agent on VPI.
So it delegates the task and waits for the result and lets the rest of the automation continue based
on what happened in the call.
Perfect.
Let's close it for now.
Oh, actually.
You can change the callback response because for now it's.
I'm not sure, but I'll check with Damien and get back to you.
So in case our agent is not.
Sure how to respond to a specific question that in the call it will say this so you can change it to
your name.
All right.
Awesome.
All right.
Now let's jump into VPI and create our AI voice agent, the one that will actually make phone calls
for us.
So we are going to set up a new advanced agent, which means it will be triggered by our workflow to
call someone exactly by this workflow, say what we tell it to say and send the response back.
Now, in order to create your account on VPI, please use the link included in the PDF called Essential
Tools for this course.
You will find it in the resources section of this lesson.
So by signing up through that link, you will get 1003 minutes to use with your agents on VPI, which
is more than enough to build, test and run a lot of AI voice agents Once you have signed up and logged
in, you will be ready to create your agent and connect it to your workflow.
So let's move on.
Once you create your account on VPI and then log in, you will see this dashboard.
So now on the left sidebar click on assistance.
So here you will build your first agent.
Now let's create assistant.
So hit that button.
And give it a name.
For example agent a voice agent.
Anything.
And eight n and we want to start from a blank template.
So create assistant.
Alright.
Great.
You will land on this dashboard.
And let me quickly walk you through what you are seeing.
So at the top you will see the cost per minute and the latency which shows how fast your agent responds.
So for most use cases, these defaults are totally fine.
And below that, under the model section you will choose the brain of your assistant.
So here we are using OpenAI as the provider and GPT four as the model, which is very powerful.
And I recommend to use this model for the best results, but you can easily switch to other, more affordable
models.
For now, let's stick with GPT four for now.
The first message is what your agent says when a call starts.
So, as you can see, the first message that the assistant will say, this can also be a URL to containerize
audio file.
So something like hi.
I'm Emily.
Personal assistant.
And the most important part is the system prompt.
This is where you tell your agent how to act, what tone to use, and how to handle different situations.
And we also will be using um, dynamic variables we defined in the workflow.
Now what I'm going to do I'm going to use the prompt I've added to the resources section of this lesson.
So the prompt and copy it to the agent system prompt, we just we just created on the platform.
All right.
So as a role we have your Emily a voice personal assistant for Damien.
And now remember the code snippet we have in our phone call agent sub workflow.
So.
So in this node that JSON defines all the key variables needed to make the call, like who does supervisor
agent is calling, what the purpose of the call is, and how he should speak to the supervisor.
Agent passes those variables to this node, which then triggers a separate workflow.
So this one that handles the actual call on VPI.
Now these fields are automatically filled by N810 at runtime based on what you ask supervisor to do.
So now we are going to use those same dynamic variables in our system prompt to make our agent sound
smart and personal on every call.
So let's go back to our system prompt.
So by using these variables in both your workflow.
So in phone call agent node and your prompt you create a Are pretty dynamic and intelligent agent that
responds to exactly what the user asked.
So you will be speaking with and we are using a variable with a type.
And the purpose of this call is.
So we use the call purpose dynamic variable and so on.
And as additional nodes we want to keep responses in a specific response style.
We want to maintain a tone throughout the conversation.
We want to keep pauses for natural flow.
Start by introducing yourself and stating the purpose.
Be concise to the point.
If unsure about something, say and we want to use this fallback response.
So when you check the workflow.
You can see we want our voice agent to say something like I'm not sure, but I will check with them
and get back to you.
Do not mention actions outside the call.
Like emails, calendar updates, etc. do not repeat yourself and be very polite and friendly.
Now, just to make it clear, this is just the schema.
So the template and when the workflow runs and it will automatically fill in most of these fields using
data pulled from earlier steps like the contact lookup or the telegram message.
But you can set defaults here, like I've done with the tone and the speaking style and the fallback
response.
So every time a call is made, the agent knows to speak in a friendly tone.
Keep responses concise and pause briefly between sentences, while things like first name, phone number,
and instructions will be dynamically filled based on the context of your request.
Now we are back in Vaapi.
Just briefly.
Here you can change the transcriber so you can switch to a different provider.
But I like Deepgram program also can change the language and model.
And here invoice section you can change the provider of of the voice.
So you can switch to 11 labs and others OpenAI for example.
And here you can pick your favorite voice right here you can test your voice agent and talk to it to
check if you like the voice, for example.
Now let's actually look at how to set up a phone number in Wapi so your agent can actually make calls.
So click on phone numbers.
Create phone number.
Now if you are based in the US you can use the free Wapi number.
So just enter area code and click create.
And these numbers are completely free.
And you can have up to ten of them in your account.
So if you are outside of the US like me, you will need to use the Import Twilio tab.
So you we need to go to Twilio.
Create an account, purchase a number and then import it here.
So once it's connected, you need to assign it to your voice agent in the phone number section like
I've done with this one.
And I'm going to show you how to create an account on Twilio and purchase a number in a separate tutorial.
So please go and find the tutorial in this section of the course.
So once you buy the phone number on Twilio and the phone numbers in for you section, scroll down.
You should see account info.
Just copy this account ID.
And paste it right here.
The same for auth token.
Copy it.
Paste it here.
And your telephone number.
Then simply click on import from Twilio and that's it.
Alright, great.
Once you successfully import the number from Twilio, you have to assign your voice agent you just created
right here.
Alright, now let's go back to our workflow.
Alright, perfect.
Now we've got the full setup complete.
We have built the domain and workflow, connected the three sub workflows for email, calendar and phone
calls, and linked everything to our AI agent in Waapi.
There is just one more important part I want to walk you through, and that's the system prompt for
our supervisor agent.
So let's have a look.
Let's expand it.
Now this prompt is what tells the supervisor agent Exactly.
So the role is your.
Emily.
That means personal assistant.
Your primary role is to delegate tasks to the appropriate tool.
You don't complete tasks yourself.
Instead, you ensure that each request is routed correctly with all necessary details passed to the
right tool for execution.
Now, we listed all the tools.
So phone call agent which initiates phone calls by following provided instructions ensuring clear and
effective communication contacts data.
So for any task involving communication calls, email scheduling, always retrieve and verify contact
details using the contact data tool before passing them to the relevant agent.
So we want our supervisor agent to use contacts Data Google Sheet to retrieve all the contact details
to perform all the tasks.
Email agents.
So use this for handling all email related actions such as sending messages, setting up replies, or
for forwarding emails.
Only send emails to verified email The addresses found in the contacts list.
Always sign emails as Damian and never use placeholders like your name.
So of course you can modify this prompt.
You can change it to your name.
If an email address for a recipient cannot be found in the contact list, do not attempt to send the
email or use placeholder addresses like example.com calendar agent.
Use this for scheduling, updating, or managing calendar events.
And now very important how to use the phone call agent.
If the user wants to make a phone call to get the phone call agent tool.
Ensure the following details are passed to the calling agent.
First name so the name of the person or business being called.
Type so the contact type, instructions, phone number, call purpose, response style, tone, etc..
So these are the exact same variables we saw earlier in the phone call agent note.
And that's because the supervisor agent is responsible for passing those values to the tool.
When it's time to make the call.
So the calling agent already understands it is a personal assistant making the call, so only provide
essential details.
Avoid unnecessary contacts or explanations.
And here we are providing examples for all the variables.
You also have rules and best practices.
So always respond in English.
Always delegate tasks using the correct tool and never complete them manually.
And so on.
And when a task requires using one or more of these tools, make sure to identify which tool is most
appropriate.
Pass along the relevant details and execute the actions needed to complete the task.
Your goal is to be proactive, precise, and organized in managing these resources to provide a smooth
experience for the user.
And lastly, we also provide.
Here is the current time and date.
All right.
Awesome.
Let's close it for now.