Validation of an approach for improving existing measurement frameworks

Software organizations are in need of methods to understand, structure, and improve the data they are collecting. We have developed an approach for us e when a large number of diverse metrics are already being collected by a s oftware organization [1], [2]. The approach combines two methods. One looks at an organization's measurement framework in a top-down goal-oriented fas hion and the other looks at it in a bottom-up data-driven fashion. The top- down method is based on a measurement paradigm called Goal-Question-Metric (GQM). The bottom-up method is based on a data mining technique called Attr ibute Focusing (AF). A case study was executed to validate this approach an d to assess its usefulness in an industrial environment. The top-down and b ottom-up methods were applied in the customer satisfaction measurement fram ework at the IBM Toronto Laboratory. The top-down method was applied to imp rove the customer satisfaction (CUSTSAT) measurement from the point of view of three data user groups. It identified several new metrics for the inter viewed groups, and also contributed to better understanding the data user n eeds. The bottom-up method was used to gain new insights into the existing CUSTSAT data. Unexpected associations between key variables prompted new bu siness insights, and revealed problems with the process used to collect and analyze the CUSTSAT data. This paper uses the case study and its results t o qualitatively compare our approach against current ad hoc practices used to improve existing measurement frameworks.