Top-down decision tree boosting and its applications

Eiji Takimoto, Akira Maruoka

Research output: Chapter in Book/Report/Conference proceedingChapter

Abstract

Top-down algorithms such as C4.5 and CART for constructing decision trees are known to perform boosting, with the procedure of choosing classification rules at internal nodes regarded as the base learner. In this work, by introducing a notion of pseudo-entropy functions for measuring the loss of hypotheses, we give a new insight into this boosting scheme from an information-theoretic viewpoint: Whenever the base learner produces hypotheses with non-zero mutual information, the top-down algorithm reduces the conditional entropy (uncertainty) about the target function as the tree grows. Although its theoretical guarantee on its performance is worse than other popular boosting algorithms such as AdaBoost, the top-down algorithms can naturally treat multiclass classification problems. Furthermore we propose a base learner LIN that produces linear classification functions and carry out some experiments to examine the performance of the top-down algorithm with LIN as the base learner. The results show that the algorithm can sometimes perform as well as or better than AdaBoost.

Original languageEnglish
Title of host publicationProgress in Discovery Science
EditorsSetsuo Arikawa, Ayumi Shinohara
Pages327-337
Number of pages11
Publication statusPublished - Dec 1 2002

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume2281
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Fingerprint

Boosting
Decision trees
Decision tree
Adaptive boosting
AdaBoost
Entropy
Conditional Entropy
Multi-class Classification
Entropy Function
Classification Rules
Mutual Information
Classification Problems
Internal
Uncertainty
Target
Vertex of a graph
Experiment
Experiments

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Takimoto, E., & Maruoka, A. (2002). Top-down decision tree boosting and its applications. In S. Arikawa, & A. Shinohara (Eds.), Progress in Discovery Science (pp. 327-337). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 2281).

Top-down decision tree boosting and its applications. / Takimoto, Eiji; Maruoka, Akira.

Progress in Discovery Science. ed. / Setsuo Arikawa; Ayumi Shinohara. 2002. p. 327-337 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 2281).

Research output: Chapter in Book/Report/Conference proceedingChapter

Takimoto, E & Maruoka, A 2002, Top-down decision tree boosting and its applications. in S Arikawa & A Shinohara (eds), Progress in Discovery Science. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 2281, pp. 327-337.
Takimoto E, Maruoka A. Top-down decision tree boosting and its applications. In Arikawa S, Shinohara A, editors, Progress in Discovery Science. 2002. p. 327-337. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
Takimoto, Eiji ; Maruoka, Akira. / Top-down decision tree boosting and its applications. Progress in Discovery Science. editor / Setsuo Arikawa ; Ayumi Shinohara. 2002. pp. 327-337 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inbook{2c9ec7027d4f4467b013e7ae9bc62691,
title = "Top-down decision tree boosting and its applications",
abstract = "Top-down algorithms such as C4.5 and CART for constructing decision trees are known to perform boosting, with the procedure of choosing classification rules at internal nodes regarded as the base learner. In this work, by introducing a notion of pseudo-entropy functions for measuring the loss of hypotheses, we give a new insight into this boosting scheme from an information-theoretic viewpoint: Whenever the base learner produces hypotheses with non-zero mutual information, the top-down algorithm reduces the conditional entropy (uncertainty) about the target function as the tree grows. Although its theoretical guarantee on its performance is worse than other popular boosting algorithms such as AdaBoost, the top-down algorithms can naturally treat multiclass classification problems. Furthermore we propose a base learner LIN that produces linear classification functions and carry out some experiments to examine the performance of the top-down algorithm with LIN as the base learner. The results show that the algorithm can sometimes perform as well as or better than AdaBoost.",
author = "Eiji Takimoto and Akira Maruoka",
year = "2002",
month = "12",
day = "1",
language = "English",
isbn = "3540433384",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
pages = "327--337",
editor = "Setsuo Arikawa and Ayumi Shinohara",
booktitle = "Progress in Discovery Science",

}

TY - CHAP

T1 - Top-down decision tree boosting and its applications

AU - Takimoto, Eiji

AU - Maruoka, Akira

PY - 2002/12/1

Y1 - 2002/12/1

N2 - Top-down algorithms such as C4.5 and CART for constructing decision trees are known to perform boosting, with the procedure of choosing classification rules at internal nodes regarded as the base learner. In this work, by introducing a notion of pseudo-entropy functions for measuring the loss of hypotheses, we give a new insight into this boosting scheme from an information-theoretic viewpoint: Whenever the base learner produces hypotheses with non-zero mutual information, the top-down algorithm reduces the conditional entropy (uncertainty) about the target function as the tree grows. Although its theoretical guarantee on its performance is worse than other popular boosting algorithms such as AdaBoost, the top-down algorithms can naturally treat multiclass classification problems. Furthermore we propose a base learner LIN that produces linear classification functions and carry out some experiments to examine the performance of the top-down algorithm with LIN as the base learner. The results show that the algorithm can sometimes perform as well as or better than AdaBoost.

AB - Top-down algorithms such as C4.5 and CART for constructing decision trees are known to perform boosting, with the procedure of choosing classification rules at internal nodes regarded as the base learner. In this work, by introducing a notion of pseudo-entropy functions for measuring the loss of hypotheses, we give a new insight into this boosting scheme from an information-theoretic viewpoint: Whenever the base learner produces hypotheses with non-zero mutual information, the top-down algorithm reduces the conditional entropy (uncertainty) about the target function as the tree grows. Although its theoretical guarantee on its performance is worse than other popular boosting algorithms such as AdaBoost, the top-down algorithms can naturally treat multiclass classification problems. Furthermore we propose a base learner LIN that produces linear classification functions and carry out some experiments to examine the performance of the top-down algorithm with LIN as the base learner. The results show that the algorithm can sometimes perform as well as or better than AdaBoost.

UR - http://www.scopus.com/inward/record.url?scp=84867637406&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84867637406&partnerID=8YFLogxK

M3 - Chapter

AN - SCOPUS:84867637406

SN - 3540433384

SN - 9783540433385

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 327

EP - 337

BT - Progress in Discovery Science

A2 - Arikawa, Setsuo

A2 - Shinohara, Ayumi

ER -