OK, so now that we have a prompt health service pushing data to I like service and we've set up a like
data source in California and we can read it in the Explore tab Loki and shows log browser.
We can continue when you open Loki.
It already has some information from the Loki data source about the kind of information that it's already
collecting.
Now the information that we're seeing here has come from our prompt configuration.
If we look at the prompt tail configuration from the last video where we download it, install the prompter
binary, we create this one scrape config.
Here we named system.
It targets the local server.
It has a job called VAR logs and it has a path variable.
So it's scanning everything in the VAR log folder.
Star Lock.
That's a wild card for anything log.
So VAR log stare, we can see that here.
Our job is VAR logs.
This file name is also created, and that's because the scraped config path property here is showing
us all the file names it is found that follow that pattern.
OK, now I can see both those options there because I've got them highlighted here so I can highlight
it or highlighted again.
Same thing the job I can make it active or not.
If I my job active and then select VAR logs, it then shows me here this string here job equals VAR
log.
That string is called a log stream selector.
That's one of the required things we need when we're doing a logical query in this window here.
So very quickly we can just show logs and has put it into their job equals VAR logs is showing me everything
in the job, VAR logs in the last one hour.
So if I just scroll up, these are all the log lines from all the files that that scrape config has
collected.
I make it look at each of those individually and open it up, and we can see it has two labels file,
name and job.
So looking at job equals file logs, every one of these has job VAR logs.
The file name will be different.
For example, that's VAR log or slope, where as that is VAR log syslog.
OK, so that's a log stream selector.
So if we go back to log browser and data like that press phoneme press VAR Log Auth log, that's also
a log stream selector.
There is just a different one.
I show logs.
It's showing me all the logs it has or the labeled file name equals VAR log of log.
So if I open those up, you'll see the label phoneme equals VAR log or log.
So all these lines here all have thought I'm VAR local log and that's the log stream selector.
Now we can do more with logs, streams like that we can select to log streams at the same time.
So I go back to the log browser and all select auth and sites at the same time.
So if we look at the log stream selector now, there are a few differences.
There is a tilt there indicating it's using a rejects pattern equals rejects that is a port character.
So VAR Log Auth Log or VAR log syslog slots, show logs.
It's now showing me all the lines from both of those files so that one is VAR log syslog.
I go further down.
-- VAR log syslog and there's a VAR log or slot that was selected before going to my documentation
website.
Log stream selectors now inside looks, frame selectors out operators.
OK, and this is an example of using a rejects.
There are several of them here.
There's a course which is the most common one you'll see.
For example, job equals VAR log, so they're not equals down here.
I've got fault.
I'm not equal to VAR.
Log syslog also rejects matches, so we've seen that we've searched for a rejects VAR log or log or
VAR log syslog and rejects does not match.
So all these examples here we can type those by hand into the search query.
For example, if I was to delete that and create a curly brace, it's given me the available labels
that it knows about in the last one hour only have two labels.
So there's no point in me changing that to 24 hours, for example, because I still only have two to
choose from.
But if I just selected for, I'm there.
It then says Equals and then it shows me what values I can search for and phoneme.
That's everything that I have from tail scrape config figures found and put into the likely service
for us.
So for naming a DB and package manager log, so I could run that query.
And that's everything it has in the last 24 hours.
We can say that something was stored Lipsy bin, for example.
OK, so now let's say I wanted to search for all of those two ways we have of doing that job equals
VAR logs because just so happens, I have one scrape config and the job name was bollocks.
Or I could say phoneme equals using a rejects.
That's the title character they dot plus like that ended off with a curly brace press shift enter and
that will return all the file names, for example VAR log syslog.
But look at that one, that's VAR log syslog.
Most of the time it's VAR log syslog.
Sure, there's some auth ones there, but let's say I didn't want one of those comma I write file name
again, not equals to syslog, for example, and then shift enter or press run query.
OK, so it's given me all the file names, auth logs in there.
We'll find the Debian Package Manager logs as well, but we won't find anything in there with the label
phoneme, VAR log syslog or slog or slog or slog.
Anyway, that is outlined here, so you can read more about the log stream.
Select operators now looking at filter expressions, filter expressions allow us to filter what's returned
from the log stream selector.
Even more so, let's go back to job equals VAR logs.
So everything that I can find where the label job equals bar logs, the job, VAR logs, I can say I
really want the jobs with the word error in them.
So some of those who have error was quite hard to see.
So let's level equals error there.
So I can say pop equals error.
So everything in here now has the word error in it now for this filter expression here.
It doesn't work to actually write equals equals like that or just a single equals needs to be a pipe
equals.
The pipe is quite an ambiguous character to use, but that's just how it does because pipe is the same
as writing or in a rejects expression.
But that's how you use that filter expression if you want to include everything with error.
Then you use those two characters there in older versions of local.
You could just do that.
But that now shows an arrow.
But the more recent versions that says Give me everything with the word error in it.
OK, so we can say, give me everything that doesn't have error in it.
So for that, we can use the not character not equals the error.
So everything return doesn't have the word error in it somewhere.
Now moving on, we could say we want to use the rejects in our query.
So in that case would use the pipe and then the rejects tilt.
And we could say, give me everything with error or info.
Now that pipe is used twice here, there, in there in this expression, that means give me everything
that matches the rejects and in the rejects.
That just means the or so error or in fact, so we'll see everything with error or info.
So shift enter.
If I scroll down, it's error Iraq info error info so we can find error or info in those results that
say I didn't want error or info, I could say not read GICS error or info.
OK, so everything returns.
Doesn't have error and it doesn't have info written anywhere in there.
We can do more than that.
Sometimes you might get a line with error and info in it, so you can say give me everything with error,
but not if it contains info.
So we're getting error there, and none of those lines also contain info, which is quite hard to find.
And this was also anyway, but just showing that it can be done.
Also, we could do another one.
Invalid user rejects invalid user will find everything with invalid user, followed by bob or radius,
so there may be a few those.
And so there's none of those.
If I check in the last two days, there are two occurrences of that.
There was invalid user followed by bulb and invalid user, followed by Radix.
I could just say give me everything with invalid user data, and we can see all the attempts where someone's
tried to log into my server in all the usernames that they're using their IP addresses.
This is pretty normal for a server on the internet, though, so if they were logging web server logs,
we could also search for status equals for a three or statistical support base, right?
That would be rich, except that if you know, rejects, rejects can become quite sophisticated and
quite long.
So we'll see some more rejections later.
OK, now look at scalar vectors and series of scalar vectors.
OK, so the data returns so far are returned as streams of log lines.
OK, so log lines many, many local ones and we can look at them individually how Gryffindor tries to
break them up.
But that's the line as it's written in a text file.
We can't really grasp that, despite the fact that Gravano in the Explore section is creating a graph
of that.
So, for example, I just put job equals VAR logs.
It's drawing us a graph and it's colored it in some way corrupted into common information like info,
error and unknown.
If we look at the tool tip there now, if we use the log visualization in the Gryffindor dashboards,
it won't show us a graph.
It will just show us the log information.
So I'll quickly demonstrate that.
So I'll copy that.
Go into create dashboard, add a new empty panel, select.
The locks option just there.
Select low key and base your local query, and so there it is, click out of it.
So there are bonds.
That's the information I was saying in the log panel.
There's no graph there, but if you wanted to see a graph, we can do that.
And that is by converting our log lines into scalar vectors or a series of scalable vectors.
Okay.
So what we do is we wrap our query into a function that somehow counts our data.
So the first one count over time shows the total count of log lines for a time range going back into
the Explore tab.
I've got that and I'll do that query again.
Curly brace job VAR logs count over time bracket and the range one minute.
OK.
So that has taken out log lines and created two Skyla vectors, so it's done a count of VAR logs where
the file name label equals VAR log syslog or so.
Another one at the bottom here with a phoneme label, says VAR Log Auth log in the last one hour.
There are only two log files being written if I change that for last 24 hours.
Then there are four log files that it can create scalar vectors for.
It's quite hard to say there is some blue dots down here that would be for the dropper agent update
log, and there's a yellow one just over here, which I'll zoom in to Nat to log Davian Package Manager
log just down there.
I can image that even more zoom out.
So that is taking our data, our log lines and converting it into a graph by doing a count on the information
that it's getting back for the log stream selector just there.
Now this range parameter here we need to put in.
That's a range of how far back it should count every time it creates one of these scales for us and
graph.
So for example, if I look at this value just here for VAR log syslog, it says 16 in the last one minute
that they're that range.
There are 16 lines in VAR log syslog at that point there.
In the last one minute, there are 21 occurrences of a log line in VAR log systems, and that's what
the one minute is.
I could say give me that for one hour shift.
Enter to update that.
Okay, so now the graph is a little different.
Zoom out.
We can see here that at this point here in the last one hour, there are one thousand three hundred
twenty one lines in the VAR log syslog for.
I could even do one second.
There's not many Detroit ten seconds ago, so the last ten seconds at this point here in the VAR log
syslog, there were 11 occurrences go back to one minute.
There are 21 occurrences in the last one minute there.
That's what the range property is about.
When you use these functions that cover your logs.
The scale vectors does not write is very similar to count over time, except it's showing us the rate
per second.
So once again, if I was to look at that value there, there is a rate of zero point six six seven log
lines per second in the last one minute at that point.
So that's how you read that.
We can also do a bytes over time count shift into via Zoom into that section there.
That is because sometimes a log line might be very long and contain a lot of bytes.
You may want to know something like that, and we can also get the rights of the bytes per second as
well when we're converting our query into a scalable vector.
We could also limit it to using a filter expression.
For example, a copy that shift into job equals bar logs that contain error, but we're counting over
time with the right of one hour in that time, right?
So much nicer to last 12 hours.
Last 24 hours.
So count over time VAR logs that contain the word error at this point here, for the last one hour,
there were 360 entries.
OK, now aggregate functions.
Now what we've been looking at so far are series of vector scales.
So I'm getting two series here.
So when doing the count over time job because VAR logs for one hour, for example, it's giving me a
new series broken up by phone.
I'm there, so I'm doing job because violence, but because each of my log lines has two labels in it,
developing a file name.
It's given me four different series and you can see in the different colors.
Now I can do a total of all of those, which would then become a single, scalable vector by wrapping
that into an aggregate function such as sum.
So sum is one of them shift.
And so it's now giving me the total of all of them.
It's no longer broken that up into the different series depended on the name of the label.
It's now done.
Some can't over time is now given me one.
There are other ones as well.
We can get the maximum of count over time or the minimum count over time.
There are several of them, their average standard deviation of standard variance account, which is
count the number of elements.
So count.
OK, so here there was to a three to three I here there was four file name series that contained log
lines with label job logs and then down here got a bottom K and top K.
Now they won't convert all our series into a single Skylar Vector like these other ones to above here.
These will give us only two series or three series of the bottom values, depending on what we use the
K or the top value dependent of what we use the case.
So an example over here there are four series return because I can see that when I use count.
So if I zoom into that, I really want to know where the values were highest.
So I can say Top K two comma, and it's only going to show me the top two series now, even though there
are four series to choose from.
OK, so let's just show me the top two.
I want to see what the bottom two series were in that collection there.
So bottom to show me the bottom two series.
So at that point, there were only two series that contained log lines being phoneme, VAR log syslog
and full name also.
But at this point, there are other fault names being written.
That's a packager and drop of agent update.
So that's the use of top and bottom k excite.
Give me the bottom three.
Now got one of three or four, which is just going to give me all of them anyway.
OK, so there are some examples there that you can use.
And of course, you can also filter those further down.
So give me everything with error or everything with the info, etc. Now not only that, let's say l
can't over time, VAR logs one minute.
For example, returns four different series based on the labels in the information, so there are two
labels here.
Job because logs and fall name may course whatever the name of the phone name was, we could have a
third label, which I'll show you in the next video and we can choose what we're going to group by.
So, for example, I'm saying some give me the sum of everything.
As one, I don't really want one anymore.
I want to split them up again, and I'll use my phone name.
So it's broken up again.
So that's essentially exactly the same response as what was returned by that.
But I'm saying explicitly to a group of by file name.
Now this is good in those cases where you have more than two labels in your log lines, I only have
two, so I'm not really changing in the query.
Much so it's quite a useless query that one.
Let's just say I had a label called host, for example, wicked group by host.
And not only that wicked group, multiple log streams and cetera, you can read my documentation for
examples if you want those kinds of things.
Now, comparison operators is not the thing we can do with the aggregate functions there.
For example, show me the count over time of job, VAR logs one minute where it's greater than four
presenter.
OK, so that's given me a total bang, the sum where the value was greater than four.
So none of those values are less than four, but we could have less than four if we wanted to.
There's nothing there.
What about less than 10?
I think there was some 20, I guess a few less than 20.
So same things greater than greater than our equals, not equals.
There's a few examples logical operators and we could say, give me something where the numbers are
greater than four or less than equal to one.
Copy that.
Put that in there.
Some kind of a time of job.
VAR logs one minute credit and four or less than an equal to one.
So everything's over for anyway.
So this example or change up to 24 hours.
So another example is between 100 and 200.
Copy that.
There's not very many values between 100 and 200 anyway.
Operator order, that's the order of how operators are processed.
That's common in computer programming.
If you don't wrap your operations in brackets, it will default to a particular order.
Example PAM does is an acronym that you can use to understand that will process parentheses first.
If you don't have that, it will then move on to processing exponents, multiplication divisions, additions,
subtractions in that order.
Anyway, some examples there will say that that there are no parentheses in there, so it's processing
the exponent first in the multiplication and division and also the modulus, then the addition.
And that's the same equation, but with parentheses wrapped around everything.
So that's a particular order.
There we go.
OK.
Local, you can read more about local here, the official documentation, it's very versatile and it
can do anything to it.
OK, now become more useful when we can a dashboard later.
Now, in the next video, I'm going to set up from table service on another server down here, showing
you that you can set up multiple prompt health services or pointing to the same wiki service that Gryffindor
is pointing to in this like data source.
And you can have as many of those as you want on only the service anyway.
Excellent.