Experimentation helps determine the effectiveness of proposed theories
and methods. But computer science has not developed a concise taxonom
y of methods that could be used to demonstrate the validity of new sof
tware development techniques. Partly as a result of this, the state of
experimentation in software engineering is poor. In this article, the
authors describe the: four general approaches to experimentation and
the important aspects of data collection. They propose a taxonomy for
software engineering experimentation that describes 12 different exper
imental approaches. To underscore the need for experimentation, the au
thors also analyze 612 papers in software engineering published in 198
5, 1990, and 1995. They classified each paper according to the data co
llection method used to validate the claims in the paper. They found t
hat about a third of the papers had no experimental validation, and ma
ny authors used sloppy terminology to describe their validation method
. The software engineering community, the authors conclude, can do a b
etter job reporting its results. This, in turn, will help industry sel
ect the right new technology to implement.