Estimation of past demographic parameters from the distribution of pairwise differences when the mutation rates very among sites: Application to human mitochondrial DNA

Distributions of pairwise differences often called "mismatch distributions" have been extensively used to estimate the demographic parameters of past population expansions. However, these estimations relied on the assumption that all mutations occurring in the ancestry of a pair of genes lead to obs ervable differences (the infinite-sites model). This mutation model may not be very realistic, especially in the case of the control region of mitocho ndrial DNA, where this methodology has been mostly applied. In this article , we show how to infer past demographic parameters by explicitly taking int o account a finite-sites model with heterogeneity of mutation rates. We als o propose an alternative way to derive confidence intervals around the esti mated parameters, based on a bootstrap approach. By checking the validity o f these confidence intervals by simulations, we find that only those associ ated with the timing of the expansion are approximately correctly estimated , while those around the population sizes are overly large. We also propose a rest of the validity of the estimated demographic expansion scenario, wh ose proper behavior is verified by simulation. We illustrate our method wit h human mitochondrial DNA, where estimates of expansion times are found to be 10-20% larger when taking into account heterogeneity of mutation rates t han under the infinite-sites model.