I was thinking about cost in terms of lowered fitness, not in terms of
time, which partly accounts for my messages not entirely making sense.
A small population isolated on the fringe of a larger population is
unlikely to be a representative, random sample. In particular, various
fitness-neutral variations are likely to be present in the larger
population, and the isolated population will probably have some
overrepresented and others underrepresented or absent (e.g., the blue
people of eastern Kentucky). If the subpopulation stays isolated, it could
become a new species (They didn't.). During this period of isolation, it
is quite improbable that all of the exact same mutations will prevail in
both populations. As a result, many genetic differences will result from
the fixing of genetic variation inherent in the ancestral population.
Additionally, any new mutation will not be introduced into both
populations. The total number of generations present in which to get the
current level of difference is thus as large or greater than the number of
generations since reproductive isolation.
In at least a few species (such as cheetahs and humans), genetic
analyses suggest an extremely low population size (a few individuals) at
some point in the past. Such small populations don't take long to reach 4N
generations.