The growth of photo-sharing website such as Flickr and Picasa enables us to access the billions of images easily. Recent years, many researchers leverage such photo-sharing site to tackle the image annotation problem. The aim of the image annotation is to give a proper label to an unknown image. Generally, image features and label features are used to acquire the relationship between them. Meanwhile, we use not only such image and label features but also geolocation which indicate the information where the image was taken. We formulate the image annotation problem as two important issues; image-based labeling and label-based localization. The former issue is to estimate a proper label from a given image. The latter is the issue to estimate the location from the label. Our approach combine these two estimation strategies. We conducted some experiments and found that our approach outperformed the traditional approach.