TOWARD A CURSE OF DIMENSIONALITY APPROPRIATE (CODA) ASYMPTOTIC THEORYFOR SEMIPARAMETRIC MODELS

We argue, that due to the curse of dimensionality, there are major dif ficulties with any pure or smoothed likelihood-based method of inferen ce in designed studies with randomly missing data when missingness dep ends on a high-dimensional vector of variables. We study in detail a s emi-parametric superpopulation version of continuously stratified rand om sampling. We show that all estimators of the population mean that a re uniformly consistent or that achieve an algebraic rate of convergen ce, no matter how slow, require the use of the selection (randomizatio n) probabilities. We argue that, in contrast to likelihood methods whi ch ignore these probabilities, inverse selection probability weighted estimators continue to perform well achieving uniform n(1/2)-rates of convergence. We propose a curse of dimensionality appropriate (CODA) a symptotic theory for inference in non- and semi-parametric models in a n attempt to formalize our arguments. We discuss whether our results c onstitute a fatal blow to the likelihood principle and study the attit ude toward these that a committed subjective Bayesian would adopt. Fin ally, we apply our CODA theory to analyse the effect of the 'curse of dimensionality' in several interesting semi-parametric models, includi ng a model for a two-armed randomized trial with randomization probabi lities depending on a vector of continuous pretreatment covariates X. We provide substantive settings under which a subjective Bayesian woul d ignore the randomization probabilities in analysing the trial data. We then show that any statistician who ignores the randomization proba bilities is unable to construct nominal 95 per cent confidence interva ls for the true treatment effect that have both: (i) an expected lengt h which goes to zero with increasing sample size; and (ii) a guarantee d expected actual coverage rate of at least 95 per cent over the ensem ble of trials analysed by the statistician during his or her lifetime. However, we derive a new interval estimator, depending on the Randomi zation probabilities, that satisfies (i) and (ii).