The modal decomposition based on the spectra of the Koopman operator has gained much attention in various areas such as data science and optimal control, and dynamic mode decomposition (DMD) has been known as a data-driven method for this purpose. However, there is a fundamental limitation in DMD and most of its variants; these methods are based on the premise that the target system is time-invariant at least within the data at hand. In this work, we aim to compute DMD on time-varying dynamical systems. To this end, we propose a probabilistic model that has factorially switching dynamic modes. In the proposed model, which is based on probabilistic DMD, observation at each time is expressed using a subset of dynamic modes, and the activation of the dynamic modes varies over time. We present an approximate inference method using expectation propagation and demonstrate the modeling capability of the proposed method with numerical examples of temporally-local events and transient phenomena.