Hello guys! I want to talk a widely used metric, specially in time series modelling, which it is MAPE.

Why MAPE is used

MAPE is a largely used metric because has the following benefits:

It is a percentage metric; so, it can be used to compare time series with different units.
It is largely used. People specially from other fields use it without knowing that are using it because it is so straight forward and natural.

I will give an example for the last bit: Let’s suppose that a salesperson had to sell 100 pairs of boots, but he sold 80, so how far is he from the target? Well, (80-100)/100 = -0,20 = -20%. So the salesperson is 20% far from the target. The formula for MAPE can be described as:

\[\text{MAPE} = \frac{\sum_{i=1}^{n} \left| \frac{y_i - \hat{y}_i}{y_i} \right|}{n}\]

where:

\(y_i\) is the true value of observation \(i\) (y_true);
\(\hat{y}_i\) is the predicted value of observation \(i\) (y_pred);
\(n\) is the number of observations.

It is so intuitive. However, behind this popularity, MAPE has several pitfalls, specially when working with high number of time series at the same time:

BUT:

MAPE is a biased metric:

To give a simple example, let’s suppose that our y_pred is 100; if the y_true is 150, we have a MAPE equals to 0.33, in contrast to MAPE equals 1 if y_true is 50. So, MAPE will always push toward to biased underforecasting.

See Table 1, which illustrates this:

Table 1: Illustration of a case when MAPE is biased

y_true	y_pred	MAE	MAPE	MSE
100	150	50	0.50	2500
100	50	50	0.50	2500
150	100	50	0.33	2500
50	100	50	1.00	2500

Check in Table 1 that MAE and MSE behave as expected (they don’t bias for under/over forecasts).

MAPE is not defined when `y_true` is zero

The denominator of mape is y_true. So, MAPE might return NaN on your programming software or Inf. In order to “fix” it, what many people do is to set y_true to a small value near to zero, which is what sklearn.metrics.mean_absolute_percentage_error do (setting y_true to \(2.22*10^-16\) when y_true is zero). But it gives other problems:

Setting y_true to \(2.22*10^-16\) will lead to a even higher MAPE number;
You will never know a “true” value of the error for this series;
Your prediction will be compromised even if your predictions are really good (let’s say 0.00001);
If you need to aggregate the metric of this series with other series, your comparison will be compromised because the resulting MAPE will be towards a very high number.

In practical terms, see in Table 2 how MAPE will behave if for a y_true of 0, we set 2.22E-16 for the last two rows of the table:

Table 2: Impact in MAPE when y_true is zero or near zero

y_true	y_pred	MAE	MAPE	MSE
0	0.5	0.5	#DIV/0!	0.25
0	1.5	1.5	#DIV/0!	2.25
2.22E-16	0.5	0.5	2.25x10^15	0.25
2.22E-16	1.5	1.5	6.76x10^15	2.25

Check in Table 2 that MAE and MSE behave as expected (are not impacted when y_true is zero or 2.22E-16).

Others also don’t calculate the forecast for these points, but you will end up with same problem as number 2.

MAPE does not minimizes the error towards the mean value

This is not a pitfall, but it is a warning, because usually people want to be right on average, and this is not what MAPE does. In contrast, it will give a prediction towards a biased median, if I can so.

A very good resource to read about is shortcomings of MAPE, that will lead to many other resources.

Alternatives

In order to overcome the asymmetry problem of MAPE, sMAPE and other adjusted MAPEs were proposed, but no one of them have good properties: they are still undefined if y_true and y_pred are zero, check it out.

A really good alternative for this scenario is to use rRMSE, which is the relative Root mean squared error. All you need to do is to divide the RMSE by a value that will be used as a reference of your data. If we think on a time series problem, we can take the average of all values of a series, or the average of the last points of the series (as many points as the length of the forecast horizon), or of a window of any other length.

Of course, if all the values of your time series are 0, then you can not use rRMSE, because the division will be by 0. However, if all points are zero, perhaps, you do not need a forecast 😛.

With it, you get an unbiased metric that is defined when y_true is zero (except in the edge case where the reference value itself is zero, e.g. a series of only zeros), and that is easy to read and understand, because it is a percentage.

Happy forecasting!