A weather forecast accuracy study

Can you trust the weather forecast ?

It's a question that can trigger long passionate conversations, especially in this country (UK) where talking about the weather is deeply embedded in the culture. An answer to such a question can therefore be of interest to anybody living in the UK, but also for companies like us who work constantly with weather forecast data in order to produce accurate estimation of renewable energy production. This study will focus on the daily-averaged solar radiation over the Tokyo area from November 2013 to July 2014.

First, we plot in figure 1 the actual solar radiation in kWh over the Tokyo area.

Figure 1: Actual solar radiation over the Tokyo area from November 2013 to June 2014. It's getting hot! The first and third quartile range shows the geographic variability over the area. We observe that the solar radiation is homogeneous over Tokyo.

We can access weather forecasts for the day ahead, the following day or even a week or a month in advance. The number of days separating the date of the prediction and the date the prediction was done for is called the forecast horizon (abbreviated fh in the following). Our dataset consists of 31 different forecasts for every day, from a forecast horizon of 1 to 31. Short term forecasts (0 < fh ≤ 11) are created by weather models. These models take weather data from the past and use it to predict the future of the weather conditions (temperature, pressure, wind, humidity…). Long term forecasts (11 < fh ≤ 31) are determined from considering previous years’ averages. For all these forecasts, we can compute the error by comparing the forecast solar radiation value to the actual value. We represent in figure 2 the error distribution for both long term and short term forecasts.

Figure 2: Error distribution for long and short term forecasts. We notice that the long term distribution is biased towards the negative values (under-estimation) while the short term is slightly biased towards the positive values (over-estimation). The short term distribution shows more of an even distribution around zero, indicating that this forecast is more accurate than the long term one... but not by much!

This figure shows us that the short term forecast performs better than the long term forecast. Now we can question how this error varries with the forecast horizon. We therefore plot in figure 3 the MAPE (measure of the accuracy) and MPE (measure of the bias) as a function of the forecast horizon.

Figure 3: MAPE (measure of the accuracy) and MPE (measure of the bias) as a function of the forecast horizon. The error and the confidence interval decrease as the forecast horizon approaches 0.

We note that the forecast accuracy is improving (the error is decreasing) when the forecast horizon gets closer to 0, for which we have a vanishing error since it corresponds to actual weather values. Not only does the accuracy gets better, but so does the confidence interval (the shaded regions are narrower for short term forecasts). We also confirm that the solar radiation is under-estimated by the long term forecast and slightly over-estimated by the short term one (see right-hand-side plot). This could be due to the fact that the studied period is particularly sunny compared to previous years. However, we also notice that a 10-day weather model forecast is, on average, worse than a prediction based on previous years’ averages... We know since Lorenz that chaotic systems (weather models are good examples of chaotic equations) are very sensitive to initial conditions... Therefore, a middle term forecast based on a weather model has to be considered with caution.

So please, keep chatting about the weather but when it comes to planning a barbecue a week in advance, don't put all your trust in the forecast!