A strategy for verification of weather element forecasts from an ensemble prediction system

Using a Bayesian context, new measures of accuracy and skill are proposed t o verify weather element forecasts from ensemble prediction systems (EPSs) with respect to individual observations. The new scores are in the form of probabilities of occurrence of the observation given the EPS distribution a nd can be applied to individual point forecasts or summarized over a sample of forecasts. It is suggested that theoretical distributions be fit to the ensemble, assuming a shape similar to the shape of the climatological dist ribution of the forecast weather element. The suggested accuracy score is s imply the probability of occurrence of the observation given the fitted dis tribution, and the skill score follows the standard format for comparison o f the accuracy of the ensemble forecast with the accuracy of an unskilled f orecast such as climatology. These two scores are sensitive to the location and spread of the ensemble distribution with respect to the verifying obse rvation. The new scores are illustrated using the output of the European Centre for Medium-Range Weather Forecasts EPS. Tests were carried out on 108 ensemble forecasts of 2-m temperature, precipitation amount, and windspeed, interpol ated to 23 Canadian stations. Results indicate that the scores are especial ly sensitive to location of the ensemble distribution with respect to the o bservation; even relatively modest errors cause a score value significantly below the maximum possible score of 1.0. Nevertheless, forecasts were foun d that achieved the perfect score. The results of a single application of t he scoring system to verification of ensembles of 500-mb heights suggests c onsiderable potential of the score for assessment of the synoptic behavior of upper-air ensemble forecasts. The paper concludes with a discussion of the new scoring method in the more general context of verification of probability distributions.