Journal of Frontiers of Computer Science and Technology ›› 2011, Vol. 5 ›› Issue (1): 68-74.DOI: 10.3778/j.issn.1673-9418.2011.01.007

• 学术研究 • Previous Articles     Next Articles

Research and Optimization of Nutch Distributed Crawler

ZHAN Hengfei1+, YANG Yuexiang2, FANG Hong2   

  1. 1. School of Computer Science, National University of Defense Technology, Changsha 410073, China
    2. Information Center, National University of Defense Technology, Changsha 410073, China
  • Received:1900-01-01 Revised:1900-01-01 Online:2011-01-01 Published:2011-01-01
  • Contact: ZHAN Hengfei

Nutch 分布式网络爬虫研究与优化

詹恒飞1+, 杨岳湘2, 方 宏2   

  1. 1. 国防科学技术大学 计算机学院, 长沙 410073
    2. 国防科学技术大学 信息中心, 长沙 410073
  • 通讯作者: 詹恒飞

Abstract: As a good open-source search engine, Nutch kernel code uses a lot of MapReduce programming models,being used by more and more businesses and organizations to customize their needs in line with the distributed search engine product. As a good search engine, one of the important prerequisites is how to grab network data as much as possible to build indexes. This paper introduces Nutch’s working mechanism based on Hadoop distributed Web crawler, points out its shortcomings and proposes an improved program, which can make Web crawler using network resources more efficiently to capture network data. Experimental results show that it is indeed more efficient than the original programs.

Key words: Nutch, Web crawler, flexible crawling

摘要: Nutch 作为一个优秀的开源搜索引擎, 其内核代码大量采用了MapReduce 的编程模式, 被越来越多的企业和团体用来定制符合自身需求的分布式搜索引擎产品。作为优秀的搜索引擎, 其重要的前提是如何尽可能多地抓取到网页数据来建立索引。介绍了Nutch 基于Hadoop 下的分布式网络爬虫工作机制, 指出其不足之处, 并提出了改进方案, 从而使网络爬虫能够更加高效地利用网络资源来抓取网络数据。经过实验测试, 证明了此方案比原方案更加高效。

关键词: Nutch 搜索引擎, 网络爬虫, 弹性抓取机制

CLC Number: