关键词: Nutch 搜索引擎, 网络爬虫, 弹性抓取机制

Abstract: As a good open-source search engine, Nutch kernel code uses a lot of MapReduce programming models,being used by more and more businesses and organizations to customize their needs in line with the distributed search engine product. As a good search engine, one of the important prerequisites is how to grab network data as much as possible to build indexes. This paper introduces Nutch’s working mechanism based on Hadoop distributed Web crawler, points out its shortcomings and proposes an improved program, which can make Web crawler using network resources more efficiently to capture network data. Experimental results show that it is indeed more efficient than the original programs.

Key words: Nutch, Web crawler, flexible crawling