Today, it has become difficult for people to find books that fulfill their preference because the number of books in the world has become very large. In order to select books to read, the reviews on candidate books are helpful. Therefore, the role of online review sites which are web sites storing reviews of books is important. One of the functions in the online review sites is the collaborative filtering (CF) with reviewers. In general, the user-based CF is based on the assumption that the preferences of users who select the same items will be similar. However, in many cases, each user would have different viewpoints for their evaluation of books. In this paper, we focused on the difference of viewpoints when each user evaluate a book. We propose a CF method based on the empathy with other reviewers. Our method utilizes the evaluation for a review as feedback. Experimental results show the effectiveness of the proposed method.