Beyond forecasting competitions

The M3-competition is to be welcomed as an extension and replication of the earlier competitions, in particular the original M-Competition. It has the same characteristics as that earlier evaluation, see for example Fildes an d Makridakis (1995) and Fildes and Ord (2001). It suffers from some of the same problems, in particular the problem posed by the lack of definition as to the population of time series under study, although as these papers poi nt out, this criticism is not damning(1). The other major criticisms levell ed at competitions (summarised by Fildes & Makridakis, 1995) such as the ch oice of loss function, aggregation over lead time, aggregation over series, etc. are all to a greater extent answered within the methodology Hibon and Makridakis (2000) have adopted here. The failure to consider multiple time origins (Fildes et al., 1998) is inevitable given the different lengths of the data series and is not particularly likely to be problematic because o f the disparate nature of the series. I would be interested to see further details summarising the data sources however to resolve this issue conclusi vely. With the usual objections to competitions dismissed as unimportant, i s the considerable effort put in by Makridakis and his colleagues justified by the originality of the results? The paper offers surprises such as the consistently strong performance of Theta, confirmation of prejudices, e.g. the unexciting performance of neural networks, and reassurance, the continu ing support for the major conclusions of the M-Competition. Thus, these are sufficient riches to justify this research experiment. But let us go on now to consider how to extend the methodology of forecasti ng competitions. Multivariate data presents considerable challenges althoug h the same questions remain meaningful, supplemented by many other importan t issues (Fildes & Ord, 2001). In a univariate context the unanswered quest ions are primarily concerned with model selection. If the choice of model i s potentially important, is it possible for a forecaster to go beyond the n aive strategy of selecting the method performing best within the classifica tion closest to his/her interests, e.g. micro, monthly, seasonal data?