A Monte Carlo EM algorithm for generalized linear mixed models with flexible random effects distribution

A popular way to represent clustered binary, count, or other data is via the generalized linear mixed model framework, which accommodates correlation through incorporation of random effects.A standard assumption is that the random effects follow a parametric family such as the normal distribution; however, this may be unrealistic or too restrictive to represent the data.We relax this assumption and require only that the distribution of random effects belong to a class of .smooth. densities and approximate the density by the seminonparametric (SNP) approach of Gallant and Nychka (1987).This representation allows the density to be skewed, multi.modal, fat. or thin.tailed relative to the normal and includes the normal as a special case.Because an efficient algorithm to sample from an SNP density is available, we propose a Monte Carlo EM algorithm using a rejection sampling scheme to estimate the fixed parameters of the linear predictor, variance components and the SNP density.The approach is illustrated by application to a data set and via simulation.