Maximum likelihood methodology was applied to determine the mode of in
heritance of rare binary traits with data structures typical for swine
populations. The genetic models considered included a monogenic, a di
genic, a polygenic, and three mixed polygenic and major gene models. T
he main emphasis was on the detection of major genes acting on a polyg
enic background. Deterministic algorithms were employed to integrate a
nd maximize likelihoods. A simulation study was conducted to evaluate
model selection and parameter estimation. Three designs were simulated
that differed in the number of sires/number of dams within sires (10/
10, 30/30, 100/30). Major gene effects of at least one SD of the liabi
lity were detected with satisfactory power under the mixed model of in
heritance, except for the smallest design. Parameter estimates were em
pirically unbiased with acceptable standard errors, except for the sma
llest design, and allowed to distinguish clearly between the genetic m
odels. Distributions of the likelihood ratio statistic were evaluated
empirically, because asymptotic theory did not hold. For each simulati
on model, the Average Information Criterion was computed for all model
s of analysis. The model with the smallest value was chosen as the bes
t model and was equal to the true model in almost every case studied.