A DEGENERACY IN CROSS-VALIDATED SKILL IN REGRESSION-BASED FORECASTS

Highly negative skill scores may occur in regression-based experimenta l forecast trials in which the data being forecast are withheld in tum from a fixed sample, and the remaining data are used to develop regre ssion relationships-that is, exhaustive cross-validation methods. A sm all negative bias in skill is amplified when forecasts are verified us ing the correlation between forecasts and actual data. The same outcom e occurs when forecasts are amplitude-inflated in conversion to a cate gorical system and scored in a ''number of hits'' framework. The effec t becomes severe when predictor-predictand relationships are weak, as is often the case in climate prediction. Some basic characteristics of this degeneracy are explored for regression-based cross-validation. S imulations using both randomized and designed datasets indicate that t he correlation skill score degeneracy becomes important when nearly al l of the available sample is used to develop forecast equations for th e remaining (very few) points, and when the predictability in the full dependent sample falls short of the conventional requirement for stat istical significance for the sample size. The undesirable effects can be reduced with one of the following methodological adjustments: 1) ex cluding more than a very small portion of the sample from the developm ent group for each cross-validation forecast trial or 2) redefining th e ''total available sample'' within one cross-validation exercise. A m ore complete elimination of the effects is achieved by 1) downward adj usting the magnitude of negative correlation skills in proportion to f orecast amplitude, 2) regarding negative correlation skills as zero, o r 3) using a forecast verification measure other than correlation such as root-mean-square error. When the correlation skill score degenerac y is acknowledged and treated appropriately, cross-validation remains an effective and valid technique for estimating predictive skill for i ndependent data.