1 Coalescent Models
7
generations back to the common ancestor of the sample is geometrically distributed:
P (g) = c N (1 − c N )
g−1
.
(1.1)
Then on average, looking backward in time, it will take 1/c N generations for
the pair of genetic lineages ancestral to the sample to coalesce. Under the diploid,
monoecious Wright–Fisher model, this would be 2N generations. This is what
sets the timescale of the coalescent process. Time is usually measured in units
proportional to N generations, e.g., 2N in this case of the diploid, monoecious
Wright–Fisher model.
The derivation of Eq. (1.1) is incorrect for two reasons. First, it is exact only
in the diploid, monoecious model if “random mating” includes the possibility of
reproduction by selfing. When there are two sexes or if selfing is not possible,
it is wrong to apply the same probability, c N , in every generation because when
the two lineages ancestral to the sample are in the same individual and they are
distinct, they necessarily descend separately from the two parents. Consequently,
the probability of coalescence in the immediately previous generation is equal to
zero. In spite of this, the geometric distribution is still approximately correct as long
as the population size is large (Möhle 1998b, c), because the two ancestral lineages
will only be in the same individual a small number of times, while the number of
generations back to their coalescence would be large, proportional to N generations
on average.
The second reason has to do with how the population pedigree is treated in the
derivation. Here it is important to recognize that coalescent theory is typically used
to describe patterns of genetic variation across the genome, for example, as in Li and
Durbin (2011) mentioned above. Because this chapter does not treat recombination,
we may imagine a genomic dataset made up of sequences at a number of genetic
loci within which there is no recombination but between which there is either
completely independent assortment, as with different chromosomes, or effectively
independent assortment, as with loci that far enough apart on the same chromosome.
It is conceptually wrong to use results such as Eq. (1.1) because, in averaging over
the process of reproduction, they do not capture the actual patterns of relatedness
among the sampled individuals which are encoded in the population pedigree.
Figure 1.2, which is adapted from Fig. 3 in Wakeley et al. (2016), shows part of
a larger pedigree for the Spanish Habsburg royal family reported in Alvarez et al.
(2009). At a single genetic locus, two sequences sampled from Mary of Portugal
and King Philip II would have zero chance of being descended from a common
ancestral sequence in the immediately previous generation. In the grand-parental
generation, however, the probability of coalescence is a substantial 1/8 due to the
special relatedness of Mary and Philip as double first cousins. Since Mary and Philip
also share one pair of great grandparents, there is a 1/32 chance of coalescence in
the third generation in the past.
These probabilities are calculated by tracing each ancestral lineage back to the
mother or the father of each individual with a 50:50 chance and letting two lineages
coalesce with probability 1/2 whenever they trace back to the same individual. If
7
generations back to the common ancestor of the sample is geometrically distributed:
P (g) = c N (1 − c N )
g−1
.
(1.1)
Then on average, looking backward in time, it will take 1/c N generations for
the pair of genetic lineages ancestral to the sample to coalesce. Under the diploid,
monoecious Wright–Fisher model, this would be 2N generations. This is what
sets the timescale of the coalescent process. Time is usually measured in units
proportional to N generations, e.g., 2N in this case of the diploid, monoecious
Wright–Fisher model.
The derivation of Eq. (1.1) is incorrect for two reasons. First, it is exact only
in the diploid, monoecious model if “random mating” includes the possibility of
reproduction by selfing. When there are two sexes or if selfing is not possible,
it is wrong to apply the same probability, c N , in every generation because when
the two lineages ancestral to the sample are in the same individual and they are
distinct, they necessarily descend separately from the two parents. Consequently,
the probability of coalescence in the immediately previous generation is equal to
zero. In spite of this, the geometric distribution is still approximately correct as long
as the population size is large (Möhle 1998b, c), because the two ancestral lineages
will only be in the same individual a small number of times, while the number of
generations back to their coalescence would be large, proportional to N generations
on average.
The second reason has to do with how the population pedigree is treated in the
derivation. Here it is important to recognize that coalescent theory is typically used
to describe patterns of genetic variation across the genome, for example, as in Li and
Durbin (2011) mentioned above. Because this chapter does not treat recombination,
we may imagine a genomic dataset made up of sequences at a number of genetic
loci within which there is no recombination but between which there is either
completely independent assortment, as with different chromosomes, or effectively
independent assortment, as with loci that far enough apart on the same chromosome.
It is conceptually wrong to use results such as Eq. (1.1) because, in averaging over
the process of reproduction, they do not capture the actual patterns of relatedness
among the sampled individuals which are encoded in the population pedigree.
Figure 1.2, which is adapted from Fig. 3 in Wakeley et al. (2016), shows part of
a larger pedigree for the Spanish Habsburg royal family reported in Alvarez et al.
(2009). At a single genetic locus, two sequences sampled from Mary of Portugal
and King Philip II would have zero chance of being descended from a common
ancestral sequence in the immediately previous generation. In the grand-parental
generation, however, the probability of coalescence is a substantial 1/8 due to the
special relatedness of Mary and Philip as double first cousins. Since Mary and Philip
also share one pair of great grandparents, there is a 1/32 chance of coalescence in
the third generation in the past.
These probabilities are calculated by tracing each ancestral lineage back to the
mother or the father of each individual with a 50:50 chance and letting two lineages
coalesce with probability 1/2 whenever they trace back to the same individual. If