NTCIR-3 WEB: An evaluation workshop for web retrieval

The authors conducted the Web Retrieval Task ('NTCIR-3 WEB') from 2001 to 2002 at the Third NTCIR Workshop. In the NTCIR-3 WEB, they attempted to assess the retrieval effectiveness of Web search engine systems using a common data set, and to build re-usable test collections that are suitable for evaluating Web information retrieval systems. With these objectives, they evaluated on searches using various types of user input, user models and document models. As the document data sets, they constructed 100-gigabyte and 10-gigabyte document collections that were gathered from the '.jp' domain. The user input was given as query term(s), sentence, and document(s). They assumed two user models where comprehensive relevant documents are required, and where precision of the top-ranked results is emphasized. They also assumed several document models, such as a document as an individual page, and a document as a page set connected by hyperlinks. This paper describes an overview of the test collections constructed in the NTCIR-3 WEB, the proposed evaluation methods, and the evaluation results. The evaluation results suggested that the link-based techniques can perform more effectively when short queries are input.