Evolutionary dynamics of mutator phenotypes in cancer: implications for chemotherapy.

Genetic instability is a central characteristic of cancers. However, the selective forces responsible for the emergence of genetic instability are not clear. We use mathematical models to determine the conditions under which selection favors instability, and when stable cells are advantageous. We take into account the processes of DNA damage, repair, cell cycle arrest, mutation, and death. We find that the rate of DNA damage can play a major role in this context. In particular, an increase in the rate of DNA damage can reverse the relative fitness of stable and unstable cells. In terms of cancer progression, we find the following results. If cells have intact apoptotic responses, stable cells prevail if the DNA hit rate is low. A high DNA hit rate can result in the selection of genetically unstable cells. This has implications for the induction of tumors by carcinogens. On the other hand, if cells are characterized by impaired apoptosis, we observe the opposite. Genetic instability is selected for if the DNA hit rate is low. A high DNA hit rate can select against instability and result in the persistence of stable cells. We propose that chemotherapy can be used to reverse the relative fitness of stable and unstable cells, such that unstable cells are the inferior competitors. This could result in the competitive exclusion of progressing cancer cells.