Okay.
Now we'll use that notification channel.
We'll simulate some errors and get an alert for that.
All uses of the devices that I set up in the influx DB section.
So when one of these devices stops working or at least there's an MP daemon stops will get an alert.
So when creating alerts, there are many things to consider.
Now CAFAGNa here will be polling in flux.
DB for information and these telegraphs are pushing information about thesis and AMP days to the influx.
DB So what's actually breaking here in this scenario is one of these daemons.
So there are two things in between flux DB and Telegraph.
So in flux database are going to be broken and neither is telegraph.
So how you would test for that particular scenario?
And Gasana It's going to be different for every situation.
But anyway, this may be ways to solve a problem.
I'll show you one way of doing it.
Okay, so I'll set up now an alert, starting with my Moscow S&P Daemon so that we know if we have lost
connection to that, that could be because the process stopped.
Okay.
So in the alerting alert rules page, we have an option to create a new alert rule.
Hey, I'm going to call this s mp d down and for my is choose a type Bafana mage to load because we're
managing this alert in Gryphon so folder this version of Gryphon 8.3 says you must create a folder.
I haven't created a folder in all the previous spaces so far before I can say it, this is going to
want a folder so I'm just going to create new folder.
S an MP d I'm going to create that nice.
That folder is now credit and you'll find folders underneath dashboard.
So that's a way for us to organize our dashboards data into folders and we'll sit at the moment.
Okay, so that is compulsory.
It might not be in later versions of Gryphon.
It's hard to know.
Okay, we need to create a query.
In flux.
Debbie, now the alert manager will be running queries periodically looking for the results, whether
they exist within a range or whether they don't exist at all.
I know data, so we need to create a query.
I'm going to go into influx day being prepare my query.
So in the Explore tab, I'm going to look at S&P.
There is an MP.
Uptime is a good property to know about whether it's an MP is working or not.
I'm not going to select agent host, I'm going to stick source and I'm going to select just one of them
being the most Q of first.
Okay.
So script editor.
Okay.
That's my query.
Excellent.
Put that query into flux db and go out and we have an updated graph.
Now what we need to do for this particular case, what I want to know the SNP daemon is not working.
We're actually going to stop getting data.
So I need to think about what is a good time range for me to decide that I'm not getting data.
So for this influx.
DB I'm going to change that down to now minus one minute and just rerun that query.
Okay.
So now I'm looking at 1 minutes worth of data, so I'm going to decide that if I'm not getting any data
for one minute from my SNP daemon, then I'm going to consider that down.
Now one thing about reading data from as an MP daemons is that it uses the UDP protocol.
UDP can be unreliable by design, so there could actually be a chance that your SNP Damian is running.
But there's a problem on the network preventing telegraph from querying the data.
So I'm just bringing that to your attention just so that you understand that doing this query in Gafah
is not the most reliable way of testing whether ICMP Diamond is actually working, but it's a good example
to show anyway because quite a complex task to try and achieve in your phone.
So anyway, I have a query, I have 1 minutes worth of data and I'm checking whether I have results
in that one minute.
So the condition here under b expression conditions when the average or last that doesn't really matter
in this particular case of a has no value.
So what I'm checking for is if there are no values in that query, one minute to now, I can even change
that to now minus 30 seconds or even 10 seconds, but 1 minutes.
Okay.
So that query is done so we can just run that again and well, I can see it's still working.
I still have results in that time period.
Three Define the alert condition.
So it's looking at the query, be there.
So it needs to be looking at the expression.
We're going to evaluate this query every one minute.
Now, in this particular example, this for property has no effect in older versions of Gryphon up before
8.3.
This would cause the alert to go into a pending state for this particular problem I'm trying to solve
and then go into a following state.
And when it's in the following state, we then get the alerts, for example, through our email channel
in this version of Gryphon to 8.3.3.
This particular property has no effect for this kind of query where I'm looking for no data that may
change in future versions of FINA.
The intended purpose of this field is to once a condition is breached, for example, in one minute
is decide that there's no data.
For example, the alert will go into a pending state.
If it's pending for longer than that full value being 5 minutes, it will become a firing alert.
Now, the way I've set this up is that it's going to go into firing state as soon as this decided.
There is no data in the last minute.
And that's because I have one minute set up there.
Okay.
These options here for summary description and run book URL, these are up to you.
I'm not going to use any of those six things that you can make happen when it goes into firing state.
Okay.
So let's preview what the state of the problem or the alert would be now as saying normal.
Okay.
So looking at this graph, it appears normal because I'm doing a condition where that condition returns
false.
The average of I has no value while there are values for the average of I and that's the A query there.
So now you actually rename your charisma default if you want to take a problem to leave it as I.
Okay, so going down that looks pretty good to me.
Let's save and exit that.
Okay, that's done.
Now in the alerting panel, get the rules section under the final I before the S&P do and in that I
have one rule for S&P de down my school state normal and we can view that never go and that's what the
query is checking against for anything in the last one minute the query remember is asking is there
nothing in that last one minute?
Then it would consider that as true and it actually would be writing a number one down there.
So what now?
It's saying zero is not a problem.
The state is considered normal.
So when everything's normal like that, we're not getting anything being highlighted on that screen.
But what I'll do now is actually turn off that s an impact daemon on that mask crossover.
Okay so my my my you all server pseudo service, S&P DS stop it.
Stop it.
Okay.
Now, in California, we look at this broad view.
We'll start to see that.
These data points will just disappear out of the query range.
So if I just refresh that, it's right over here now.
So do I.
Going back to the alerting rule page.
Nothing's showing up there.
I'll fast forward the video until we get something showing up.
There's a problem.
Okay.
So one minute has passed.
It's now considered that it's firing as one error so I can filter by firing for normal or pending.
If I wanted to clear a filter, let's look at this firing rule here.
Firing SNP down MySchool.
Let's look at it.
More detail.
If we view it, we get this error up here file to evaluate queries and expressions, fail to execute
conditions, etc. That is because there's no data being returned.
Okay.
So while influx db still working and telegraph is still working, this mp d is not working.
So telegraph op dragging flux db carafano is only looking at that one particular thing too.
There's no data coming from the S and MPD case, so we're not seeing anything in here, but it's telling
us that it's firing and alerting anyway.
I'm gonna check my emails now.
Okay.
So if I check my inbox, I'm actually not getting any alert at the moment.
What we need to do is we need to set a notification policy, says all alerts go to the default contact
point.
Unless you set additional matches in the specific reading area, we can use the specific policy for
that or we can just change our default contact point some and edit that and select email as my default
contact point and then press save.
Okay, alert rules.
Okay.
I'm going to check my emails again and I've got S&P down my school.
Okay.
So that's worked.
Okay.
So turn the S&P demon back on.
That is using start like so.
And looking back at this Flirting Rules page.
Okay.
After some time, it should trigger that we are now getting data again.
And it says, okay.
Now, that took actually about a minute to actually realize that the system's back up.
In fact, the system was up actually much sooner than that.
That's just one of the things that you have to deal with when having your monitoring alert rules so
far back from the source of the problem you're trying to look for.
That's just the reality of monitoring in every system, actually.
So anyway, as you can see, it's quite a lot of steps for a simple check.
Now, another thing to consider, too, is that I've only done this in a day.
Check on my mind also.
Okay, let's just say I wanted to check all of my s and AP daemons at the same time.
Okay.
So I've got s an impact daemon on that server and also on my influx DB server.
One option you have is to go back into your role here, press edit and you could have multiple queries
so I could create another query thing identical to this one up here.
Down here.
But just name it in flux.
DB For example.
Okay, so it's filling data.
I'll change it to one minute like I did before to run the query.
The condition here.
Now, I can add a second condition with the average of B or C actually, because this next one is called
C has no value and I can keep adding those for every single server.
But the problem then would be the other condition.
B is one way checking would only be getting a true that that was failing if both of those service were
down.
So this is not really the ideal way of doing it.
Instead, cancel that.
Okay, let's just look at that.
We don't have an option to duplicate at the moment, but that might come in the future.
We need to create new alert rule s and MP d down in flux.
DB This time Farnham managed to choose day that folders now credit flux db.
My query.
Fox TV.
Now.
One minute.
A ritual asked in this particular case.
Doesn't matter.
A has no value.
That's from the query.
Very good.
Define alert condition.
Every one minute makes no difference.
Again.
Preview alerts set as normal.
I'm not fussed about this extra information save and exit incident.
I have two rules now that either of those could trigger at any time.
And I could do the same with the Senate by day running a microphone or server as well, which I'll do
really quickly.
Very good case that when one of those goes down, I'll get an email alert.
And also note that you also get an email alert when it comes back up resolved.
Excellent.