PROBABILITY OF A SEGREGATING PATTERN IN A SAMPLE OF DNA-SEQUENCES

Mutations that result in segregating sites (polymorphic sites) in a sa mple of DNA sequences can be classified into different types. A patter n of segregating sites is an array of the numbers of various types of mutations. Using an urn model, the probability of a pattern of segrega ting sites can be expressed as a recurrence equation and its value can be computed sequentially. Among those that can be computed by this me thod are the probability of obtaining k external mutations (mutations that occur in external branches of the genealogy of a sample), the pro bability of obtaining k internal mutations (mutations that occur in in ternal branches), the probability of obtaining k singletons (segregati ng sites at which one of the two segregating nucleotides is present in on ly one sequence), and the probability of obtaining k non-singleton s. Two applications of the method are discussed. One is a maximum like lihood estimation of theta and another is a Bayesian statistical test of the hypothesis of neutral mutations. (C) 1998 Academic Press.