Estimation and selection of complex covariate effects in pooled nested case.control studies with heterogeneity

A major challenge in cancer epidemiologic studies, especially those of rare cancers, is observing enough cases.To address this, researchers often join forces by bringing multiple studies together to achieve large sample sizes, allowing for increased power in hypothesis testing, and improved efficiency in effect estimation.Combining studies, however, renders the analysis difficult owing to the presence of heterogeneity in the pooled data.In this article, motivated by a collaborative nested case.control (NCC) study of ovarian cancer in three cohorts from United States, Sweden, and Italy, we investigate the use of penalty regularized partial likelihood estimation in the context of pooled NCC studies to achieve two goals.First, we propose an adaptive group lasso (gLASSO) penalized approach to simultaneously identify important variables and estimate their effects.Second, we propose a composite agLASSO penalized approach to identify variables with heterogeneous effects.Both methods are readily implemented with the group coordinate gradient decent algorithm and shown to enjoy the oracle property.We conduct simulation studies to evaluate the performance of our proposed approaches in finite samples under various heterogeneity settings, and apply them to the pooled ovarian cancer study.