More than just accuracy: Measuring forecast "goodness"

Weather forecasting is not only about being accurate but also about efficiently transmitting the correct message to promote actionable responses.

A typical way to measure forecast skill is through statistical analysis by verifying a prediction against actuality. For example, take a forecast that predicts a maximum temperature of 16C for tomorrow, but the temperature only reaches 15C. Thus, the forecast was wrong by one degree.

Is this statistical approach the only gauge of forecast skill, or are there other qualitative metrics that measure forecast “goodness”? Meteorologists argue there are, and here are the reasons why you should be looking for them in a forecast.

Four qualities that help measure the ‘goodness’ of a forecast:

Take the above example about the afternoon high temperature. What was the certainty in the forecast that a maximum temperature would be 16C? Is there also a slight chance that it could be 15C tomorrow? Or perhaps a more important question; why are we splitting hairs? Well, the importance of uncertainty becomes more apparent with this example.

Imagine you own a farm, allotment, or even for your own back garden you must cover your crops and plants to avoid damage when the temperature drops sub-zero. The forecast says the temperature will drop to 2C. The next day you find out that a portion of your farm was damaged due to sub-freezing temperatures.

This is because there was no uncertainty expressed in that forecast. How about “overnight lows will drop to 2C, with isolated areas potentially seeing slightly lower temperatures”? That takes us to the second quality of a good forecast. Is the forecast actionable? In other words, does the forecast convey enough information and uncertainty to the user to make them take proper action.

Visualising impacts from the end user is another essential quality of a good forecast. In other words, can the user build an image or movie from the information provided in the forecast? One way a forecast achieves this quality is through visual comparison.

The last quality of a good forecast is its integrity. Forecasts should be easily transferable and resistant to losing the key message from one communication channel or media platform to the next. Predictions aren’t only available on the news station; you can also get a forecast on your phone or radio.

The weather affects us all day, every day, so it’s important to find the right forecast. Whilst there are plenty of forecasts on television, online, and on your phone, use the “goodness” qualities to sift through the best ones and find your perfect weather companion.