This paper proposes a novel method for generating arbitrary stereoscopic view from multiple omnidirectional image sequences. Although conventional methods for arbitrary view generation with an image-based rendering approach can create binocular views, positions and directions of viewpoints for stereoscopic vision are limited to a small range. In this research, we attempt to generate arbitrary stereoscopic views from omnidirectional image sequences that are captured in various multiple paths. To generate a high-quality stereoscopic view from a number of images captured at various viewpoints, appropriate ray information needs to be selected. In this paper, appropriate ray information is selected from a number of omnidirectional images using a penalty function expressed as ray similarity. In experiments, we show the validity of this penalty function by generating stereoscopic view from multiple real image sequences.