Object detection is an important task for computer vision applications. Many researchers have proposed a lot of methods to detect the objects through the background modeling. Most of previous approaches model the background independently for each pixel and detect foreground objects based on it. Then, it is difficult for the background model to deal with illumination changes, which cause significant intensity changes as in the case that a foreground object appears. To solve this problem, in this paper, we propose a new background model considering the similarity in the intensity changes among pixels. In particular, we classify all the pixels into several clusters based on the similarity of their intensity changes. Then, focusing on each cluster, we can easily identify whether the significant intensity changes are caused by foreground objects or illumination changes. This is because, if the illumination changes, most of the pixels belonging to the same cluster exhibit the similar intensity changes.