University of Illinois Urbana-Champaign

An Empirical Study of Tokenization Strategies for Biomedical Information Retrieval

Jiang, Jing; Zhai, ChengXiang

Content Files
An Empirical Study of Tokenization Strategies for Biomedical Information Retrieval.pdf

Permalink

https://hdl.handle.net/2142/11207

Description

Title
An Empirical Study of Tokenization Strategies for Biomedical Information Retrieval
Author(s)
  • Jiang, Jing
  • Zhai, ChengXiang
Issue Date
2006-05
Keyword(s)
  • bioinformatics
  • tokenization
  • information retrieval
Date of Ingest
2009-04-21T14:57:13Z
Abstract
Due to the great variation of biological names in biomedical text, appropriate tokenization is an important preprocessing step for biomedical IR. Despite its importance, there has been little study on the evaluation of various tokenization strategies for biomedical text. In this work, we conducted a careful, systematic evaluation of a set of tokenization heuristics on all the available TREC biomedical IR test collections with two representative retrieval methods. We also studied the effect of stemming and stop word removal on the retrieval performance. As expected, our experimental results show that tokenization can significantly affect the retrieval accuracy; appropriate tokenization can improve the performance by up to 80\%. In particular, it is shown that different query types require different tokenization heuristics, stemming is effective only for certain queries, and stop word removal in general does not improve the retrieval performance in biomedical text.
Type of Resource
text
Genre of Resource
Technical Report
Permalink
http://hdl.handle.net/2142/11207
Copyright and License Information
You are granted permission for the non-commercial reproduction, distribution, display, and performance of this technical report in any format, BUT this permission is only for a period of 45 (forty-five) days from the most recent time that you verified that this technical report is still available from the University of Illinois at Urbana-Champaign Computer Science Department under terms that include this permission. All other rights are reserved by the author(s).

Owning Collections

Research and Tech Reports - Computer Science PRIMARY