Discovering interesting patterns for investment decision making with GLOWER - A genetic learner overlaid with entropy reduction

Prediction in financial domains is notoriously difficult for a number of re asons. First, theories tend to be weak or non-existent, which makes problem formulation open ended by forcing us to consider a large number of indepen dent variables and thereby increasing the dimensionality of the search spac e. Second, the weak relationships among variables tend to be nonlinear, and may hold only in limited areas of the search space. Third, in financial pr actice, where analysts conduct extensive manual analysis of historically we ll performing indicators, a key is to find the hidden interactions among va riables that perform well in combination. Unfortunately, these are exactly the patterns that the greedy search biases incorporated by many standard ru le learning algorithms will miss. In this paper, we describe and evaluate s everal variations of a new genetic learning algorithm (GLOWER) on a variety of data sets. The design of GLOWER has been motivated by financial predict ion problems, but incorporates successful ideas from tree induction and rul e learning. We examine the performance of several GLOWER variants on two UC I data sets as well as on a standard financial prediction problem (S&P500 s tock returns), using the results to identify one of the better variants for further comparisons. We introduce a new (to KDD) financial prediction prob lem (predicting positive and negative earnings surprises), and experiment w ith GLOWER, contrasting it with tree- and rule-induction approaches. Our re sults are encouraging, showing that GLOWER has the ability to uncover effec tive patterns for difficult problems that have weak structure and significa nt nonlinearities.