This paper proposes a new algorithm for Simultaneous Localization and Mapping (SLAM) with omnidirectional stereo vision. In our approach, stereo matching is solved efficiently by using the estimated spatial information of the environment and robot motion. The use of two EKF (Extended Kalman Filter) estimators brings about a more reliable robot trajectory and a better map of the environment with more landmarks. The first estimator (a binocular estimator) mostly focuses on the robot trajectory; the second estimator (a monocular estimator) is devoted to the map building. Reliably matched landmarks are entered into the first estimator to establish the reliable robot position and some of the landmark positions. Other landmarks, which have more uncertainty of stereo combination or are observed by only one camera, are passed to the second estimator to establish their positions. This structure results in more precise estimation of robot trajectory and a map of the environment with more landmarks.