抄録
We propose a preprocessing method for Web mining which, given semi-structured documents with the same structure and style, distinguishes useless parts and non-useless parts in each document without any knowledge on the documents. It is based on a simple idea that any n-gram is useless if it appears frequently. To decide an appropriate pair of length n and frequency a, we introduce a new statistic measure alternation count. It is the number of alternations between useless parts and non-useless parts. Given news articles written in English or Japanese with some non-articles, the algorithm eliminates frequent n-grams used for the structure and style of articles and extracts the news contents and headlines with more than 97% accuracy if articles are collected from the same site. Even if input articles are collected from different sites, the algorithm extracts contents of articles from these sites with at least 95% accuracy. Thus, the algorithm does not depend on the language, is robust for noises, and is applicable to multiple formats.
| 元の言語 | 英語 |
|---|---|
| ホスト出版物のタイトル | Discovery Science - 4th International Conference, DS 2001, Proceedings |
| 出版者 | Springer Verlag |
| ページ | 113-127 |
| ページ数 | 15 |
| 巻 | 2226 |
| ISBN（印刷物） | 9783540429562 |
| 出版物ステータス | 出版済み - 2001 |
| イベント | 4th International Conference on Discovery Science, DS 2001 - Washington, 米国 継続期間: 11 25 2001 → 11 28 2001 |
出版物シリーズ
| 名前 | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
|---|---|
| 巻 | 2226 |
| ISSN（印刷物） | 03029743 |
| ISSN（電子版） | 16113349 |
その他
| その他 | 4th International Conference on Discovery Science, DS 2001 |
|---|---|
| 国 | 米国 |
| 市 | Washington |
| 期間 | 11/25/01 → 11/28/01 |
Fingerprint
All Science Journal Classification (ASJC) codes
- Computer Science(all)
- Theoretical Computer Science
これを引用
Eliminating useless parts in semi-structured documents using alternation counts. / Ikeda, Daisuke; Yamada, Yasuhiro; Hirokawa, Sachio.
Discovery Science - 4th International Conference, DS 2001, Proceedings. 巻 2226 Springer Verlag, 2001. p. 113-127 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); 巻 2226).研究成果: 著書/レポートタイプへの貢献 › 会議での発言
}
TY - GEN
T1 - Eliminating useless parts in semi-structured documents using alternation counts
AU - Ikeda, Daisuke
AU - Yamada, Yasuhiro
AU - Hirokawa, Sachio
PY - 2001
Y1 - 2001
N2 - We propose a preprocessing method for Web mining which, given semi-structured documents with the same structure and style, distinguishes useless parts and non-useless parts in each document without any knowledge on the documents. It is based on a simple idea that any n-gram is useless if it appears frequently. To decide an appropriate pair of length n and frequency a, we introduce a new statistic measure alternation count. It is the number of alternations between useless parts and non-useless parts. Given news articles written in English or Japanese with some non-articles, the algorithm eliminates frequent n-grams used for the structure and style of articles and extracts the news contents and headlines with more than 97% accuracy if articles are collected from the same site. Even if input articles are collected from different sites, the algorithm extracts contents of articles from these sites with at least 95% accuracy. Thus, the algorithm does not depend on the language, is robust for noises, and is applicable to multiple formats.
AB - We propose a preprocessing method for Web mining which, given semi-structured documents with the same structure and style, distinguishes useless parts and non-useless parts in each document without any knowledge on the documents. It is based on a simple idea that any n-gram is useless if it appears frequently. To decide an appropriate pair of length n and frequency a, we introduce a new statistic measure alternation count. It is the number of alternations between useless parts and non-useless parts. Given news articles written in English or Japanese with some non-articles, the algorithm eliminates frequent n-grams used for the structure and style of articles and extracts the news contents and headlines with more than 97% accuracy if articles are collected from the same site. Even if input articles are collected from different sites, the algorithm extracts contents of articles from these sites with at least 95% accuracy. Thus, the algorithm does not depend on the language, is robust for noises, and is applicable to multiple formats.
UR - http://www.scopus.com/inward/record.url?scp=33745304536&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=33745304536&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:33745304536
SN - 9783540429562
VL - 2226
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 113
EP - 127
BT - Discovery Science - 4th International Conference, DS 2001, Proceedings
PB - Springer Verlag
ER -