This paper presents an algorithm that seeks every possible exception rule which violates a common sense rule and satisfies several assumptions of simplicity. Exception rules, which represent systematic deviation from common sense rules, are often found interesting. Discovery of pairs that consist of a common sense rule and an exception rule, resulting from undirected search for unexpected exception rules, was successful in various domains. In the past, however, an exception rule represented a change of conclusion caused by adding an extra condition to the premise of a common sense rule. That approach formalized only one type of exceptions, and failed to represent other types. In order to provide a systematic treatment of exceptions, we categorize exception rules into eleven categories, and we propose a unified algorithm for discovering all of them. Preliminary results on fifteen real-world data sets provide an empirical proof of effectiveness of our algorithm in discovering interesting knowledge. The empirical results also match our theoretical analysis of exceptions, showing that the eleven types can be partitioned in three classes according to the frequency with which they occur in data.