MIXED EFFECTS SMOOTHING SPLINE ANALYSIS OF VARIANCE

We propose a general family of nonparametric mixed effects models. Smo othing splines are used to model the fixed effects and are estimated b y maximizing the penalized likelihood function. The random effects are generic and are modelled parametrically by assuming that the covarian ce function depends on a parsimonious set of parameters. These paramet ers and the smoothing parameter are estimated simultaneously by the ge neralized maximum likelihood method. We derive a connection between a nonparametric mixed effects model and a linear mixed effects model. Th is connection suggests a way of fitting a nonparametric mixed effects model by using existing programs. The classical two-way mixed models a nd growth curve models are used as examples to demonstrate how to use smoothing spline analysis-of-variance decompositions to build nonparam etric mixed effects models. Similarly to the classical analysis of var iance, components of these nonparametric mixed effects models can be i nterpreted as main effects and interactions. The penalized likelihood estimates of the fixed effects in a two-way mixed model are extensions of James-Stein shrinkage estimates to correlated observations. In an example three nested nonparametric mixed effects models are fitted to a longitudinal data set.