The future challenges in oncology imaging are to assess the response to treatment even earlier. As an addition to functional imaging, mathematical modeling based on the imaging is an alternative, cross-disciplinary area of development. Modeling was developed in oncology not only in order to understand and predict tumor growth, but also to anticipate the effects of targeted and untargeted therapies. A very wide range of these models exist, involving many stages in the progression of tumors. Few models, however, have been proposed to reproduce in vivo tumor growth because of the complexity of the mechanisms involved. Morphological imaging combined with “spatial” models appears to perform well although functioning imaging could still provide further information on metabolism and the micro-architecture. The combination of imaging and modeling can resolve complex problems and describe many facets of tumor growth or response to treatment. It is now possible to consider its clinical use in the medium term. This review describes the basic principles of mathematical modeling and describes the advantages, limitations and future prospects for this in vivo approach based on imaging data.