关键词: 文本片段化机制, 文档检索, 相关性评分, 相关片段比例, 片段整合计算

Abstract:

Document retrieval is a research hotspot of natural language processing. Compared with short text document which has the characteristics of information diversity and length, in long text retrieval, a query statement is often not related to all sentences in a long text, and strong interference of some highly similar segments will occur. Therefore, the correlation score between a query statement and a document can not be simply calculated based on the similarity between words or strings. Text snippet mechanism (TSM) is proposed for document retrieval. TSM first divides each candidate document into snippets, and then calculates the correlation between query statements and document snippets. The correlation matching scheme used takes into account the semantic and word frequency factors. TSM selects key text snippets and obtains the relevant snippet ratio, and then calculates the correlation score between query and target document based on these information, so as to obtain the Top-K document set. Experimental results show that TSM can improve the performance of information retrieval on IR test collection of Glasgow.

Key words: text snippet mechanism, document retrieval, correlation calculation, relevant snippet ratio, snippet integration score