Once we have a model and a period, then we can evaluate the model on it, and
we'll need a metric to calculate their performance.
So let's start simply by calculating the errors,
which is the difference between the forecasted values from our model and
the actual values over the evaluation period.
The most common metric to evaluate the forecasting performance of a model
is the mean squared error or mse where we square the errors and
then calculate their mean.
Well, why would we square it?
Well, the reason for this is to get rid of negative values.
So, for example, if our error was two above the value, then it will be two, but
if it were two below the value, then it will be minus two.
These errors could then effectively cancel each other out,
which will be wrong because we have two errors and not none.
But if we square the error of value before analyzing, then both of these errors would
square to four, not canceling each other out and effectively being equal.
And if we want the mean of our errors' calculation to be of the same scale
as the original errors, then we just get its square root,
giving us a root means squared error or rmse.
Another common metric and one of my favorites is the mean absolute error or
mae, and it's also called the main absolute deviation or mad.
And in this case, instead of squaring to get rid of negatives,
it just uses their absolute value.
This does not penalize large errors as much as the mse does.
Depending on your task, you may prefer the mae or the mse.
For example, if large errors are potentially dangerous and
they cost you much more than smaller errors, then you may prefer the mse.
But if your gain or
your loss is just proportional to the size of the error, then the mae may be better.
Also, you can measure the mean absolute percentage error or mape,
this is the mean ratio between the absolute error and the absolute value,
this gives an idea of the size of the errors compared to the values.
If we look at our data, we can measure the MAE using code like this.
The keras metrics libraries include an MAE that can be called like this.
With the synthetic data we showed earlier, we're getting about 5.93,
let's consider that our baseline.