CLUSTERED BINARY LOGISTIC-REGRESSION IN TERATOLOGY DATA USING A FINITE MIXTURE DISTRIBUTION

The beta-binomial distribution introduced by Skellam has been applied in many teratology problems for modelling the litter effect. Recently, Morel and Nagaraj proposed a new distribution for modelling cluster m ultinomial data when the clustering is believed to be caused by clumpe d sampling. It turns out that the distribution is a mixture of two bin omial distributions and accommodates the estimation of an additional p arameter to account for intra-litter effect. The new distribution aris es from a cluster mechanism in which some individuals within a cluster exhibit the same behaviour while the remaining individuals from the c luster react independently of each other. Such a mechanism is a natura l model in teratology problems, where typically a genetic trait is pas sed with a certain probability to the foetuses of the same litter. In this article, we use the new distribution to model binary responses wi th logistic regression. We analyse data from a teratology experiment t o demonstrate that the new model provides a useful addition to current methodology. The experiment investigates the synergistic effect of th e anticonvulsant phenytoin and trichloropopene oxide on the prenatal d evelopment of inbred mice. In a simulation study we investigate the ty pe I error rate and the power of the maximum likelihood ratio test whe n the data follow a finite mixture distribution. (C) 1997 by John Wile y & Sons, Ltd.