Method for selecting appropriate sentence from documents on the WWW for the open-ended conversation dialog system

Masahiro Shibata, Yoichi Tomiura, Tomomi Nishiguchi

Research output: Contribution to journalArticle

8 Citations (Scopus)

Abstract

We propose an open-ended dialog system that generates a proper sentence to a user's utterance using abundant documents on the World Wide Web as sources. Existing knowledge-based dialog systems give meaningful information to a user, but they are unsuitable for open-ended input. The system Eliza can handle open-ended input, but it gives no meaningful information. Our system lies between the above two dialog systems; it converses on various topics and gives meaningful information related to the user's utterances. The system selects an appropriate sentence as a response from documents gathered through the Web, on the basis of surface cohesion and shallow semantic coherence. The surface cohesion follows centering theory and the semantic coherence is calculated on the basis of the conditional distribution and inverse document frequency of content words (nouns, verbs, and adjectives.) We developed a trial system to converse about movies and experimentally found that the proposed method generated 66 % appropriate responses.

Original languageEnglish
Pages (from-to)507-519
Number of pages13
JournalTransactions of the Japanese Society for Artificial Intelligence
Volume24
Issue number6
DOIs
Publication statusPublished - Jan 1 2009

Fingerprint

World Wide Web
Semantics
Knowledge based systems

All Science Journal Classification (ASJC) codes

  • Software
  • Artificial Intelligence

Cite this

Method for selecting appropriate sentence from documents on the WWW for the open-ended conversation dialog system. / Shibata, Masahiro; Tomiura, Yoichi; Nishiguchi, Tomomi.

In: Transactions of the Japanese Society for Artificial Intelligence, Vol. 24, No. 6, 01.01.2009, p. 507-519.

Research output: Contribution to journalArticle

@article{3888e44260a44ea6a049380e3948e37f,
title = "Method for selecting appropriate sentence from documents on the WWW for the open-ended conversation dialog system",
abstract = "We propose an open-ended dialog system that generates a proper sentence to a user's utterance using abundant documents on the World Wide Web as sources. Existing knowledge-based dialog systems give meaningful information to a user, but they are unsuitable for open-ended input. The system Eliza can handle open-ended input, but it gives no meaningful information. Our system lies between the above two dialog systems; it converses on various topics and gives meaningful information related to the user's utterances. The system selects an appropriate sentence as a response from documents gathered through the Web, on the basis of surface cohesion and shallow semantic coherence. The surface cohesion follows centering theory and the semantic coherence is calculated on the basis of the conditional distribution and inverse document frequency of content words (nouns, verbs, and adjectives.) We developed a trial system to converse about movies and experimentally found that the proposed method generated 66 {\%} appropriate responses.",
author = "Masahiro Shibata and Yoichi Tomiura and Tomomi Nishiguchi",
year = "2009",
month = "1",
day = "1",
doi = "10.1527/tjsai.24.507",
language = "English",
volume = "24",
pages = "507--519",
journal = "Transactions of the Japanese Society for Artificial Intelligence",
issn = "1346-0714",
publisher = "Japanese Society for Artificial Intelligence",
number = "6",

}

TY - JOUR

T1 - Method for selecting appropriate sentence from documents on the WWW for the open-ended conversation dialog system

AU - Shibata, Masahiro

AU - Tomiura, Yoichi

AU - Nishiguchi, Tomomi

PY - 2009/1/1

Y1 - 2009/1/1

N2 - We propose an open-ended dialog system that generates a proper sentence to a user's utterance using abundant documents on the World Wide Web as sources. Existing knowledge-based dialog systems give meaningful information to a user, but they are unsuitable for open-ended input. The system Eliza can handle open-ended input, but it gives no meaningful information. Our system lies between the above two dialog systems; it converses on various topics and gives meaningful information related to the user's utterances. The system selects an appropriate sentence as a response from documents gathered through the Web, on the basis of surface cohesion and shallow semantic coherence. The surface cohesion follows centering theory and the semantic coherence is calculated on the basis of the conditional distribution and inverse document frequency of content words (nouns, verbs, and adjectives.) We developed a trial system to converse about movies and experimentally found that the proposed method generated 66 % appropriate responses.

AB - We propose an open-ended dialog system that generates a proper sentence to a user's utterance using abundant documents on the World Wide Web as sources. Existing knowledge-based dialog systems give meaningful information to a user, but they are unsuitable for open-ended input. The system Eliza can handle open-ended input, but it gives no meaningful information. Our system lies between the above two dialog systems; it converses on various topics and gives meaningful information related to the user's utterances. The system selects an appropriate sentence as a response from documents gathered through the Web, on the basis of surface cohesion and shallow semantic coherence. The surface cohesion follows centering theory and the semantic coherence is calculated on the basis of the conditional distribution and inverse document frequency of content words (nouns, verbs, and adjectives.) We developed a trial system to converse about movies and experimentally found that the proposed method generated 66 % appropriate responses.

UR - http://www.scopus.com/inward/record.url?scp=70350128796&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=70350128796&partnerID=8YFLogxK

U2 - 10.1527/tjsai.24.507

DO - 10.1527/tjsai.24.507

M3 - Article

AN - SCOPUS:70350128796

VL - 24

SP - 507

EP - 519

JO - Transactions of the Japanese Society for Artificial Intelligence

JF - Transactions of the Japanese Society for Artificial Intelligence

SN - 1346-0714

IS - 6

ER -