Top-down decision tree learning as information based boosting

Eiji Takimoto, Akira Maruoka

Research output: Contribution to journalArticle

13 Citations (Scopus)

Abstract

We consider a boosting technique that can be directly applied to multiclass classification problems. Although many boosting algorithms have been proposed so far, most of them are developed essentially for binary classification problems, and in order to handle multiclass classification problems, they need to be reduced somehow to binary ones. In order to avoid such reductions, we introduce a notion of the pseudo-entropy function G that gives an information-theoretic criterion, called the conditional G-entropy, for measuring the loss of hypotheses. The conditional G-entropy turns out to be useful for defining the weakness of hypotheses that approximate, in some way, a multiclass function in general, so that we can consider the boosting problem without reduction. We show that the top-down decision tree learning algorithm using the conditional G-entropy as its splitting criterion is an efficient boosting algorithm. Namely, the algorithm intends to minimize the conditional G-entropy, rather than the classification error. In the binary case, our algorithm turns out to be identical to the error-based boosting algorithm proposed by Kearns and Mansour, and our analysis gives a simpler proof of their results.

Original languageEnglish
Pages (from-to)447-464
Number of pages18
JournalTheoretical Computer Science
Volume292
Issue number2
DOIs
Publication statusPublished - Jan 27 2003
Externally publishedYes

Fingerprint

Boosting
Decision trees
Decision tree
Entropy
Classification Problems
Multi-class Classification
Binary
Entropy Function
Binary Classification
Tree Algorithms
Multi-class
Learning algorithms
Learning Algorithm
Learning
Minimise

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Top-down decision tree learning as information based boosting. / Takimoto, Eiji; Maruoka, Akira.

In: Theoretical Computer Science, Vol. 292, No. 2, 27.01.2003, p. 447-464.

Research output: Contribution to journalArticle

@article{7f1e255d6146438a82f504fc7e2ac2c4,
title = "Top-down decision tree learning as information based boosting",
abstract = "We consider a boosting technique that can be directly applied to multiclass classification problems. Although many boosting algorithms have been proposed so far, most of them are developed essentially for binary classification problems, and in order to handle multiclass classification problems, they need to be reduced somehow to binary ones. In order to avoid such reductions, we introduce a notion of the pseudo-entropy function G that gives an information-theoretic criterion, called the conditional G-entropy, for measuring the loss of hypotheses. The conditional G-entropy turns out to be useful for defining the weakness of hypotheses that approximate, in some way, a multiclass function in general, so that we can consider the boosting problem without reduction. We show that the top-down decision tree learning algorithm using the conditional G-entropy as its splitting criterion is an efficient boosting algorithm. Namely, the algorithm intends to minimize the conditional G-entropy, rather than the classification error. In the binary case, our algorithm turns out to be identical to the error-based boosting algorithm proposed by Kearns and Mansour, and our analysis gives a simpler proof of their results.",
author = "Eiji Takimoto and Akira Maruoka",
year = "2003",
month = "1",
day = "27",
doi = "10.1016/S0304-3975(02)00181-0",
language = "English",
volume = "292",
pages = "447--464",
journal = "Theoretical Computer Science",
issn = "0304-3975",
publisher = "Elsevier",
number = "2",

}

TY - JOUR

T1 - Top-down decision tree learning as information based boosting

AU - Takimoto, Eiji

AU - Maruoka, Akira

PY - 2003/1/27

Y1 - 2003/1/27

N2 - We consider a boosting technique that can be directly applied to multiclass classification problems. Although many boosting algorithms have been proposed so far, most of them are developed essentially for binary classification problems, and in order to handle multiclass classification problems, they need to be reduced somehow to binary ones. In order to avoid such reductions, we introduce a notion of the pseudo-entropy function G that gives an information-theoretic criterion, called the conditional G-entropy, for measuring the loss of hypotheses. The conditional G-entropy turns out to be useful for defining the weakness of hypotheses that approximate, in some way, a multiclass function in general, so that we can consider the boosting problem without reduction. We show that the top-down decision tree learning algorithm using the conditional G-entropy as its splitting criterion is an efficient boosting algorithm. Namely, the algorithm intends to minimize the conditional G-entropy, rather than the classification error. In the binary case, our algorithm turns out to be identical to the error-based boosting algorithm proposed by Kearns and Mansour, and our analysis gives a simpler proof of their results.

AB - We consider a boosting technique that can be directly applied to multiclass classification problems. Although many boosting algorithms have been proposed so far, most of them are developed essentially for binary classification problems, and in order to handle multiclass classification problems, they need to be reduced somehow to binary ones. In order to avoid such reductions, we introduce a notion of the pseudo-entropy function G that gives an information-theoretic criterion, called the conditional G-entropy, for measuring the loss of hypotheses. The conditional G-entropy turns out to be useful for defining the weakness of hypotheses that approximate, in some way, a multiclass function in general, so that we can consider the boosting problem without reduction. We show that the top-down decision tree learning algorithm using the conditional G-entropy as its splitting criterion is an efficient boosting algorithm. Namely, the algorithm intends to minimize the conditional G-entropy, rather than the classification error. In the binary case, our algorithm turns out to be identical to the error-based boosting algorithm proposed by Kearns and Mansour, and our analysis gives a simpler proof of their results.

UR - http://www.scopus.com/inward/record.url?scp=0037467653&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0037467653&partnerID=8YFLogxK

U2 - 10.1016/S0304-3975(02)00181-0

DO - 10.1016/S0304-3975(02)00181-0

M3 - Article

VL - 292

SP - 447

EP - 464

JO - Theoretical Computer Science

JF - Theoretical Computer Science

SN - 0304-3975

IS - 2

ER -