‫All right, now lets actually work with streams.
‫And again, starting by creating a new file.
‫All right. Now lets say that for some reason in
‫our application, we need to read a large text file from
‫the file system, and then send it to the client.
‫So how would we do that?
‫Well there are multiple ways and we're going to explore
‫a few of them starting with the most basic one
‫and moving all the way to the best way of doing this.
‫Okay, so the first thing is to require the file system
‫package just like before, and also the HTTP module.
‫Let me now show you a nice trick,
‫so lets actually create a server like this.
‫So, require the HTTP package, and then from here we can
‫right away call createServer. So just like this.
‫Okay, so the result of requiring HTTP, is the
‫HTTP object and on there we can then use createServer.
‫Just like this, and then save it into a variable.
‫So that is yet another way of creating a new server
‫with writing even a bit less code.
‫Okay and so just like before lets now list to the request
‫event, and specify our callback here.
‫So, the first solution that we're going to use is the
‫easiest and most straight-forward one.
‫Which is to simply read the file into a variable,
‫and then once that's done, send it to the client
‫in the way that we already know how to do it.
‫So that one's very simple, and let me write a comment
‫here for that. So solution one, and so
‫fs.readFile then our test file again.
‫And then once that's ready, we call our callback function
‫in which we have access to the data.
‫But first lets handle the error.
‫So for example in case that the file does not exist,
‫and in that case we simply log the error to the console.
‫But otherwise, we simply send this data back to the client.
‫So we use the response object just like we did before
‫many times, and then .end and the data.
‫So, give it a save, and so that is the first
‫and the simplest solution, right?
‫But before we can test this, we need to actually also
‫start a sever, right? So we use listen to do that.
‫And the port, the same one as before, then localhost
‫and our callback function.
‫Okay, so lets see if this actually works.
‫And of course not, because I actually didn't even start
‫the application.
‫So now it says listening, and now lets see
‫what happens here.
‫And here we go. So this file is huge, it has
‫Node.js is the best written in it like 10000 times
‫or something, and so it takes a lot of time
‫until it loads entirely. And we're not really interested in
‫loading everything, so lets stop this here,
‫and go back to our code. So this works just fine, the
‫solution one that we have here in this case,
‫but the problem is that with this solution,
‫node will actually have to load the entire file
‫into memory, because only after that's ready,
‫it can then send that data. Now this is a problem
‫when the file is big, and also when there are a ton
‫of requests hitting your server. Because the node process
‫will very quickly run out of resources and your app will
‫quit working, everything will crash, and your users
‫will not be happy believe me. So this solution here
‫does work when we're just creating something small
‫locally for just ourselves, for example.
‫But in a production-ready application, you cannot use
‫a piece of code like this. Okay? So lets move on to
‫our second solution. And in that solution,
‫we will actually use streams.
‫Okay, so lets comment out this part,
‫and move on to solution number two.
‫And the idea here is that we actually don't really
‫need to read this data from the file into a variable, right?
‫We don't need this variable. So instead of reading
‫the data into a variable, and having to store that variable
‫into memory, we will just create a readable stream.
‫Then as we receive each chunk of data,
‫we send it to the client as a response
‫which is a writable stream remember.
‫So let me now show you how we can use streams.
‫So we create a variable, lets call it readable here,
‫and then from the file system again, we use createReadStream
‫Then this of course small. And now the name of the file
‫that we're trying to read. So that is again, test-file.txt.
‫Okay, perfect. So this now creates a stream from
‫the data that is in this text file, which we can then
‫consume piece by piece. So chunk by chunk.
‫And how do we do that? Well, remember from the last
‫lecture, that each time there is a new piece of data
‫that we can consume, a readable stream emits the
‫data event. Okay, and so we can listen to that,
‫just like we learned in the events lecture.
‫So readable.on, data, and then our callback function.
‫And in our callback function, we have access to
‫that piece of data, so to that chunk.
‫Let me call it chunk here in our callback function
‫and so now we can handle this piece of data.
‫And what we're going to do with this piece of data,
‫with this chunk, is to actually write it to a
‫writable stream, which is the response.
‫So, res.write, this chunk. Okay?
‫So again remember that this response is a
‫writable stream. So just as I mentioned in the previous
‫video, right? And so we can now use the write method
‫to send every single piece of data into that stream.
‫Okay, and with this effectively we're streaming
‫the content from the file right to the client.
‫Okay, you understand the difference?
‫So before, we write everything at once into a variable,
‫and once that was ready, we then sent that entire piece
‫of data back to the client. But in this situation,
‫with the stream, it is different.
‫We're effectively streaming the file, so we read one piece
‫of the file, and as soon as that's available,
‫we send it right to the client, using the write method
‫of the respond stream. Then when the next piece is available
‫then that piece will be sent, and all the way until
‫the entire file is read and streamed to the client.
‫Okay, so now just to finish, we also have to handle
‫the event when all the data is read.
‫So when the stream is basically finished reading the data
‫from the file.
‫So on that case, the end event will be emitted,
‫and so as soon as that happens what we're going to do
‫is to do res.end, okay? And we used this one before,
‫so calling end on the response, we did that before, right?
‫And now, it makes actually more sense, because again,
‫the response is also a stream, and the end method
‫signals that no more data will be written to this
‫writable stream, okay?
‫So up here all we did was actually use res.end with the
‫data in it. So we did no streaming, all we did was
‫in the end to send some data.
‫Now in this case, we're not passing anything
‫into this end method, because we already sent all the data
‫using res.write, chunk by chunk,
‫and so when the readable stream is finished reading
‫its file, well all we have to do is to then signal
‫that we're ready using res.end like this, okay?
‫So we always need to use this data and this end event here
‫one after another like this. Because otherwise,
‫the response will actually never really be sent
‫to the client. Okay, so without this piece here,
‫this entire solution would not really work, okay?
‫So lets now close this one, start again,
‫and read again, and so you see that its actually working
‫again, okay? Now this time without the problems we had
‫with the first solution. Lets stop this here, and go back
‫to our code, because I want to show you now,
‫that there is another event that we can listen to
‫on a readable stream, which is the error event.
‫So readable.on('error') and in this callback function
‫we have access to the error object.
‫Okay so in this case we will log this error to the console,
‫and then send the result of file not found.
‫And we can also set the status code to an error,
‫so 500 for example. So usually it's automatically set to 200
‫which means okay, but in this case we have a server error,
‫which means that we want to send back the code 500.
‫All right, so lets now...
‫Actually I need to quit this again.
‫So lets see what happens now.
‫Oh we already have an error right here,
‫res.status is not a function. And yeah,
‫actually its statusCode. So this is the way that
‫you write it in express, so I'm already used to writing
‫express so much, so express is a node.js framework
‫which we're going to use in the rest of the course
‫and in express you do it like this, and so that was
‫the problem. So I'm already a bit too used
‫to writing express.
‫So lets go back, and yeah so now we see file not found,
‫and if we open up the dev tools, you see our 500 error code
‫that we just sent before, and if we go to our network tab,
‫lets try that again, you have the status code here as well.
‫So just like we saw in one of the previous videos in another
‫section actually. So this is how we can inspect this
‫kind of stuff in the network tab.
‫All right, so lets fix it here, give it a save,
‫and okay so this again works just perfectly
‫but there still is a problem with this approach
‫that I just showed you. And the problem is that our
‫readable stream, so the one that we're using to read
‫the file from the disk, is much much faster than actually
‫sending the result with the response writable stream
‫over the network. And this will overwhelm the response
‫stream, which cannot handle all this incoming data so fast.
‫And this problem is called backpressure.
‫And it's a real problem that can happen in real situations.
‫So in this case, backpressure happens when the response
‫cannot send the data nearly as fast as it is receiving it
‫from the file, does that make sense?
‫So we have to fix that solution,
‫and come up with an even better one.
‫And so, we will create solution three,
‫and that is actually the final one,
‫Okay? So no more solutions than three.
‫So the secret here is to actually use that pipe operator
‫that I mentioned in the last video, okay?
‫So the pipe operator is available on all readable streams,
‫and it allows us to pipe the output of a readable stream
‫right into the input of a writable stream, okay?
‫And that will then fix the problem of backpressure
‫because it will automatically handle the speed basically
‫of the data coming in, and the speed of the data going out.
‫Okay, so lets actually get our readable stream here.
‫Okay, so that is our readable stream, and now
‫all we have to do is to take our readable stream,
‫use the pipe method on it, and then put in a writable stream
‫and that is the response. And that's actually it.
‫That's all we have to do for this solution, okay?
‫So it always works like this, let me write it out here
‫as a comment. So we need a readable source basically, okay
‫so again this is just to explain it to you,
‫then we can use the pipe on it, and here we will have to put
‫a writable destination. So this is where our data
‫comes from, and it has to be a readable stream,
‫and that data we can then pipe into a writable destination.
‫So in this case our destination is the response, okay?
‫Now this stream here can actually be a duplex or a
‫transform stream as well, but what matters is that
‫we can write to the stream. And response of course,
‫is such a stream. So we can of course write to the response
‫that will be sent to the client, okay?
‫So the pipe operator automatically solves the problem
‫of backpressure that we had previously.
‫And it's also actually a much more elegant and
‫straight-forward solution. So what we did here before
‫was just to show you all the ways in which we can
‫use the stream methods and events in order to create this
‫kind of solution and of course they have many use cases,
‫but in a problem like this, the pipe operator is actually
‫the best solution. Behind the scenes it does something like
‫this here actually, but again in a
‫much more straight-forward way for us to write,
‫because it handles all the stuff internally
‫behind the scenes. So I believe that pipe here is actually
‫the easiest way of consuming and writing streams, unless
‫of course, as I mentioned, we need a more customized
‫solutions. And then, we have to use these more complicated
‫tools like these events and methods that
‫I showed you before. All right, so just to finish,
‫lets quit the process here, start it again,
‫and of course, see if it still works, which it does.
‫And so, our work is done here. So we're finished with this
‫lecture, and ready to go straight to the next one.