Toward drawing an atlas of hypothesis classes: Approximating a hypothesis via another hypothesis model

Computational knowledge discovery can be considered to be a complicated human activity concerned with searching for something new from data with computer systems. The optimization of the entire process of computational knowledge discovery is a big challenge in computer science. If we had an atlas of hypothesis classes which describes prior and basic knowledge on relative relationship between the hypothesis classes, it would be helpful in selecting hypothesis classes to be searched in discovery processes. In this paper, to give a foundation for an atlas of various classes of hypotheses, we have defined a measure of approximation of a hypothesis class C₁ to another class C₂. The hypotheses we consider here are restricted to m-ary Boolean functions. For 0 ≤ ε ≤ 1, we say that C₁ is (1−ε)-approximated to C₂ if, for every distribution D over {0, 1}^m, and for each hypothesis h₁ ∈ C₁, there exists a hypothesis h₂ ∈ C₂ such that, with the probability at most ε, we have h₁(x) ≠ h₂(x) where x ∈ {0, 1}^m is drawn randomly and independently according to D. Thus, we can use the approximation ratio of C₁ to C₂ as an index of how similar C₁ is to C₂. We discuss lower bounds of the approximation ratios among representative classes of hypotheses like decision lists, decision trees, linear discriminant functions and so on. This prior knowledge would come in useful when selecting hypothesis classes in the initial stage and the sequential stages involved in the entire discovery process.