In this paper, we report on the construction of a pancreatic tumor model that represents the relationship between the tumor growth and the micro anatomical structures. The former, the tumor growth, is described by referring to the temporal series of MRI images of the whole body and the latter, the micro structures of the tumor, is described by a spatial series of microscopic images of thin-sections sliced from the extracted pancreatic tumor. For the model construction, we developed new non-rigid registration methods for (1) accurate description of tumor growth, (2) reconstruction of 3D microscopic images, and (3) registration between an MRI image and corresponding microscopic images. In addition, we constructed a neural network that can generate a set of fake microscopic image patches of a pancreatic tumor that corresponds to each voxel inside the tumor region in an MRI image. The outlines of the methods are introduced and some examples of experimental results are demonstrated.