Assessing quality in decision analytic cost-effectiveness models - A suggested framework and example of application

Despite the growing use of decision analytic modelling in cost-effectivenes s analysis, there is a relatively small literature on what constitutes good practice in decision analysis. The aim of this paper is to consider the co ncept of 'validity' and 'quality' in this area of evaluation, and to sugges t a framework by which quality can be demonstrated on the part of the analy st and assessed by the reviewer and user. The paper begins by considering the purpose of cost-effectiveness models an d argues that the their role is to identify optimum treatment decisions in the context of uncertainty about future states of the world. The issue of w hether such models can be defined as 'scientific' is considered. The notion that decision analysis undertaken at time t can only be considered scienti fic if its outputs closely predict the results of a trial undertaken at rim e t+1 is rejected as this ignores the need to make decisions on the basis o f currently available evidence. Rather, the scientific characteristic of de cision models is based on the fact that, in principle at least, such analys es can be falsified by comparison of two states of the world, one where res ource allocation decisions are based on formal decision analysis and the ot her where such decisions are not. This section of the paper also rejects th e idea of exact codification of scientific method in general, and of decisi on analysis in particular, as this risks rejecting potentially valuable mod els, may discourage the development of novel methods and can distort resear ch priorities. However, the paper argues that it is both possible and neces sary; to develop a framework for assessing quality in decision models. Building on earlier work, various dimensions of quality in decision modelli ng are considered: model structure (disease states. options, time horizon a nd cycle length), data (identification, incorporation, handling uncertainty ); and consistency (internal and external). Within this taxonomy a (nonexha ustive) list of questions about quality is suggested which are illustrated by their application to a specific published model. The paper argues that s uch a framework can never be prescriptive about every aspect of decision mo delling. Rather, it should encourage the analyst to provide an explicit and comprehensive justification of their methods, and allow the user of the mo del to make an informed judgment about the relevance. coherence and usefuln ess of the analysis.