Comparative forecast evaluation: Graphical Gaussian models and sufficiencyrelations

This paper deals with the comparative evaluation of categorical forecasts s upposing that forecasts and observations are continuous variables and have a jointly normal distribution. An information content approach based on the well-established covariance fitting technique of graphical Gaussian modeli ng is proposed to evaluate the possibly correlated random errors in competi ng forecasts. Suppose that two alternative forecasting systems deliver forecasts, say f(a ) and f(b), for a scalar variable theta. Two questions are relevant when us ing these forecasts: 1) Is one forecasting system definitely better than th e other? 2) Knowing the forecasts of the better system, can additional info rmation be obtained from also consulting the second system? The main part o f this paper addresses the second question. If, for instance, the forecasts f(b) are redundant given the value of f(a), the forecasts f(a) are suffici ent for the pair of forecasts (f(a), f(b)). The appropriate statistical con cept to describe this situation is conditional independence of f(b) and the ta given f(a). Pairwise conditional independences in a dataset can conveniently be display ed in a graph by a lack of direct connection between nodes representing the corresponding variables. For multivariate normal data missing links in the graph are characterized by zero elements of the inverse variance-covarianc e matrix. This study applies a known maximum likelihood technique of fittin g graphical models to data in order to specify the amount of incremental in formation in f(b). A prototypical example is elaborated that indicates a po tential of graphical modeling for evaluating the dependence structure in a set of multisite forecasts. Several studies have examined a different sufficiency concept to identify w hich of two given forecasting systems is unambiguously more useful to any u ser. The forecasts f(a) are termed sufficient for the forecasts f(b) if the statistical properties of f(b) can be simulated by additionally randomizin g the forecasts f(a). Assuming joint normal distributions of forecasts and corresponding observations, this randomization translates into a simple red uction of explained variance. In this study the relation between f(a) being sufficient For f(b) and it being sufficient for the pair (f(a), f(b)) is e lucidated.