Bayesian analysis of empirical software engineering cost models

To date many software engineering cost models have been developed to predic t the cost, schedule, and quality of the software under development. But, t he rapidly changing nature of software development has made ii extremely di fficult to develop empirical models that continue to yield high prediction accuracies. Software development costs continue to increase and practitione rs continually express their concerns over their inability to accurately pr edict the costs involved. Thus, one of the most important objectives of the software engineering community has been to develop useful models that cons tructively explain the software development life-cycle and accurately predi ct the cost of developing a software product. To that end, many parametric software estimation models have evolved in the last two decades [25], [17], [26], [15], [28], [1], [2], [33], [7], [10], [22], [23], Almost all of the above mentioned parametric models have been empirically c alibrated to actual data from completed software projects. The most commonl y used technique for empirical calibration has been the popular classical m ultiple regression approach. As discussed in this paper, the multiple regre ssion approach imposes a few assumptions frequently violated by software en gineering datasets, The source data is also generally imprecise in reportin g size, effort, and cost-driver ratings, particularly across different orga nizations. This results in the development of inaccurate empirical models t hat don't perform very well when used for prediction. This paper illustrate s the problems faced by the multiple regression approach during the calibra tion of one of the popular software engineering cost models, COCOMO II. It describes the use of a pragmatic 10 percent weighted average approach that was used for the first publicly available calibrated version [6]. It then m oves on to show how a more sophisticated Bayesian approach can be used to a lleviate some of the problems faced by multiple regression. It compares and contrasts the two empirical approaches, and concludes that the Bayesian ap proach was better and more robust than the multiple regression approach. Bayesian analysis is a well-defined and rigorous process of inductive reaso ning that has been used in many scientific disciplines (the reader can refe r to [11], [35], [3] for a broader understanding of the Bayesian Analysis a pproach). A distinctive feature of the Bayesian approach is that it permits the investigator to use both sample (data) and prior (expert-judgment) inf ormation in a logically consistent manner in making inferences. This is don e by using Bayes' theorem to produce a 'postdata' or posterior distribution for the model parameters. Using Bayes' theorem, prior (or initial) values are transformed to postdata views. This transformation can be viewed as a l earning process. The posterior distribution is determined by the variances of the prior and sample information. If the variance of the prior informati on is smaller than the variance of the sampling information, then a higher weight is assigned to the prior information. On the other hand, if the vari ance of the sample information is smaller than the variance of the prior in formation, then a higher weight is assigned to the sample information causi ng the posterior estimate to be closer to the sample information. The Bayesian approach discussed in this paper enables stronger solutions to one of the biggest problems faced by the software engineering community: t he challenge of making good decisions using data that is usually scarce and incomplete. We note that the predictive performance of the Bayesian approa ch (i.e., within 30 percent of the actuals 75 percent of the time) is signi ficantly better than that of the previous multiple regression approach (i.e ., within 30 percent of the actuals only 52 percent of the time) on our lat est sample of 161 project datapoints.