And in this video, you're going to learn about how you can do streaming inside of Lang chain.
Streaming is a concept that allows you to get the output of a large language model, token by token,
rather than waiting for the entire response.
Firstly, we've installed all the different packages such as lang chain, lang chain, OpenAI, lang
chain community, and then we've also added on a section here for adding in your API key.
What we're now going to have to do is we're then going to have to import the chat OpenAI package from
Lang chain.
I'm going to start my Jupyter notebook.
And then also we're going to import lang chain core dot messages import human message.
Next we're going to set up the chat model.
So I'm going to do model is equal to chat OpenAI.
And then what we're going to do is instead of invoking it we're also going to have to add a parameter
to the chat OpenAI.
So we'll turn on streaming is equal to true.
And then what you'll see is then instead of doing dot invoke we're going to do dot stream.
Now what does dot stream give you.
Well it gives you chunks back.
You can see it returns a iterator of base message chunk right.
So it's going to return an iterator.
So what we can then do is to for chunk in model dot stream print chunk dot content An easier way to
do this would probably be to flush the input stream.
So we'll just change this and say flush the input stream.
And here you go.
That's exactly what we're looking for.
Right.
So you can see here that the content will come in.
You can also adapt this to work with the human message or the system message or the AI message classes
that you previously learned about.
So for example, we can do for chunk in model dot stream.
And then we will pass a list of messages.
And we're going to have a human message which says what is the capital of the moon?
We all know there is no capital of the moon, but just bear with me.
Then what we're doing is we're then going to do exactly what we had here.
So printing a chunk of content, we're then flushing the input as well.
So you'll see here you do have to pass a list if you're providing message history in the stream.
And what you'll see here is that we actually get the output exactly what we had like before, but we're
now able to pass in different types of messages.
In the next video, you want to learn about how you can extract structured information out using a large
language model with a concept called an output parser inside of.