Evolutionary branching in a finite population: Deterministic branching vs. stochastic branching

Adaptive dynamics formalism demonstrates that, in a constant environment, a continuous trait may first converge to a singular point followed by spontaneous transition from a unimodal trait distribution into a bimodal one, which is called "evolutionary branching." Most previous analyses of evolutionary branching have been conducted in an infinitely large population. Here, we study the effect of stochasticity caused by the finiteness of the population size on evolutionary branching. By analyzing the dynamics of trait variance, we obtain the condition for evolutionary branching as the one under which trait variance explodes. Genetic drift reduces the trait variance and causes stochastic fluctuation. In a very small population, evolutionary branching does not occur. In larger populations, evolutionary branching may occur, but it occurs in two different manners: in deterministic branching, branching occurs quickly when the population reaches the singular point, while in stochastic branching, the population stays at singularity for a period before branching out. The conditions for these cases and the mean branching-out times are calculated in terms of population size, mutational effects, and selection intensity and are confirmed by direct computer simulations of the individual-based model.