THE COST OF ADDING PARAMETERS TO A MODEL

For a general regression model with n independent observations we cons ider the variance of the estimate of a quantity of interest under two scenarios. One scenario is where all the parameters are estimated from the data; the other scenario is where a subset of the parameters is a ssumed known at their true values and the remaining parameters are est imated. We focus on quantities of interest which are defined on the sc ale of the response variable. We show that, under certain conditions, the ratio of a weighted sum across the design points of the variance o f the quantity of interest is given by q/p, where q and p are the numb er of free parameters in the two scenarios. Thus, in this average sens e, the inflation in variance associated with adding parameters, also i nterpreted as the cost of adding parameters to a model, is directly pr oportional to the number of parameters. We study models involving powe r transformations, non-linear models and exponential family models.