nutch subcommand reference

The nutch command sets up the java class environment for nutch to run. Subcommand aliases provide a conveninet way to run specific nutch classes.

The nutch subcommand is given below along with the aliased class name. The documentation comes from the version 0.6 code comments, usage descriptions, and the tutorial.

There is a canned crawl command for intranets:
$ bin/nutch crawl urls -dir crawl.test -depth 3 >& crawl.log

Details are forund in the Nutch tutorial file. A summary of the normal workflow for an internet crawl follows:

1. Bootstrap the nutch database:
   $ mkdir db # for the nutch page and link database
$ nutchpath=/path/to/nutch
$ datapath=/path/to/data
$ mkdir segments # subdirectories hold pages fetched and indexed as a unit
$ ${nutchpath}/bin/nutch admin ${datapath}/db -create
$ ${nutchpath}/bin/nutch inject ${datapath}/db -urlfile starting_urls.txt

2. Iteratively build a set of data segments:
   1. Generate a segment subdirectory and a fetchlist from the database:
      $ ${nutchpath}/bin/nutch generate ${datapath}/db ${datapath}/segments
   2. Get the segment subdirectory name that was just created:
      (N is the iteration number 1,2,3,...)
      $ sN=`ls -d ${datapath}/segments/2* | tail -1`
$ echo $sN
   3. Fetch the content of the segment from its fetchlist:
      $ ${nutchpath}/bin/nutch fetch $sN
   4. Update the nutch database with the fetched results:
      $ ${nutchpath}/bin/nutch updatedb ${datapath}/db $sN
   5. Run several iterations of link analysis to prioritize popular pages:
      $ ${nutchpath}/bin/nutch analyze ${datapath}/db 2 #use 5 for inital analysis
3. Index each of the segments that you fetched:
   $ ${nutchpath}/bin/nutch index $sN
4. Delete duplicate pages
   $ ${nutchpath}/bin/nutch dedup ${datapath}/segments dedup.tmp

admin = net.nutch.tools.WebDBAdminTool

The WebDBAdminTool is for Nutch administrators who need special access to the webdb. It allows for finer editing of the stored values.

analyze = net.nutch.tools.LinkAnalysisTool

LinkAnalysisTool performs link-analysis by using the DistributedAnalysisTool. This single-process all-in-one tool is a wrapper around the more complicated distributed one.

crawl = net.nutch.tools.CrawlTool

Perform complete crawling and indexing given a set of root urls.

datanode = 'net.nutch.ndfs.NDFS'

The NDFS class holds the NDFS client and server.

DataNode controls just one critical table: block-> BLOCK_SIZE stream of bytes

This info is stored on disk (the NameNode is responsible for asking other machines to replicate the data). The DataNode reports the table's contents to the NameNode upon startup and every so often afterwards.

dedup = net.nutch.indexer.DeleteDuplicates

Deletes duplicate documents in a set of Lucene indexes. Duplicates have either the same contents (via MD5 hash) or the same URL.

fetch = net.nutch.fetcher.Fetcher

The fetcher. Most of the work is done by plugins.

fetchlist = net.nutch.pagedb.FetchListEntry

generate = net.nutch.tools.FetchListTool

This class takes an IWebDBReader, computes a relevant subset, and then emits the subset.

index = net.nutch.indexer.IndexSegment

Creates an index for the output corresponding to a single fetcher run.

inject = net.nutch.db.WebDBInjector

This class takes a flat file of URLs and adds them as entries into a pagedb. Useful for bootstrapping the system.

merge = net.nutch.indexer.IndexMerger

IndexMerger creates an index for the output corresponding to a single fetcher run.

mergesegs = net.nutch.tools.SegmentMergeTool

This class cleans up accumulated segments data, and merges them into a single (or optionally multiple) segment(s), with no duplicates in it.

There are no prerequisites for its correct operation except for a set of already fetched segments (they don't have to contain parsed content, only fetcher output is required). This tool does not use DeleteDuplicates, but creates its own "master" index of all pages in all segments. Then it walks sequentially through this index and picks up only most recent versions of pages for every unique value of url or hash.

If some of the input segments are corrupted, this tool will attempt to repair them, using net.nutch.segment.SegmentReader.fixSegment(NutchFileSystem, File, boolean, boolean, boolean, boolean) method.

Output segment can be optionally split on the fly into several segments of fixed length.

The newly created segment(s) can be then optionally indexed, so that it can be either merged with more new segments, or used for searching as it is.

Old segments may be optionally removed, because all needed data has already been copied to the new merged segment. NOTE: this tool will remove also all corrupted input segments, which are not useable anyway - however, this option may be dangerous if you inadvertently included non-segment directories as input...

You may want to run SegmentMergeTool instead of following the manual procedures, with all options turned on, i.e. to merge segments into the output segment(s), index it, and then delete the original segments data.

namenode = 'net.nutch.ndfs.NDFS'

The NDFS class holds the NDFS client and server.

DataNode controls just one critical table: block-> BLOCK_SIZE stream of bytes

This info is stored on disk (the NameNode is responsible for asking other machines to replicate the data). The DataNode reports the table's contents to the NameNode upon startup and every so often afterwards.

ndfs = net.nutch.fs.TestClient

This class provides some NDFS administrative access.

parse = net.nutch.tools.ParseSegment

Parse contents in one segment.

It assumes, under given segment, existence of ./fetcher_output/, which is typically generated after a non-parsing fetcher run (i.e., fetcher is started with option -noParsing).

Contents in one segemnt are parsed and saved in these steps:

1. ./fetcher_output/ and ./content/ are looped together (possibly by multiple ParserThreads), and content is parsed for each entry. The entry number and resultant ParserOutput are saved in ./parser.unsorted.
2. ./parser.unsorted is sorted by entry number, result saved as ./parser.sorted.
3. ./parser.sorted and ./fetcher_output/ are looped together. At each entry, ParserOutput is split into ParseDate and ParseText, which are saved in ./parse_data/ and ./parse_text/ respectively. Also updated is FetcherOutput with parsing status, which is saved in ./fetcher/.

In the end, ./fetcher/ should be identical to one resulted from fetcher run WITHOUT option -noParsing.

By default, intermediates ./parser.unsorted and ./parser.sorted are removed at the end, unless option -noClean is used. However ./fetcher_output/ is kept intact.

Check Fetcher.java and FetcherOutput.java for further discussion.

prune = net.nutch.tools.PruneIndexTool

This tool prunes existing Nutch indexes of unwanted content. The main method accepts a list of segment directories (containing indexes). These indexes will be pruned of any content that matches one or more query from a list of Lucene queries read from a file (defined in standard config file, or explicitly overridden from command-line). Segments should already be indexed, if some of them are missing indexes then these segments will be skipped.

NOTE 1: Queries are expressed in Lucene's QueryParser syntax, so a knowledge of available Lucene document fields is required. This can be obtained by reading sources of index-basic and index-more plugins, or using tools like Luke. During query parsing a WhitespaceAnalyzer is used - this choice has been made to minimize side effects of Analyzer on the final set of query terms. You can use link net.nutch.searcher.Query.main(String[]) method to translate queries in Nutch syntax to queries in Lucene syntax.
If additional level of control is required, an instance of {@link PruneChecker} can be provided to check each document before it's deleted. The results of all checkers are logically AND-ed, which means that any checker in the chain can veto the deletion of the current document. Two example checker implementations are provided - PrintFieldsChecker prints the values of selected index fields, StoreUrlsChecker stores the URLs of deleted documents to a file. Any of them can be activated by providing respective command-line options.

The typical command-line usage is as follows:

PruneIndexTool index_dir -dryrun -queries queries.txt -showfields url,title
This command will just print out fields of matching documents.
PruneIndexTool index_dir -queries queries.txt
This command will actually remove all matching entries, according to the queries read from queries.txt file.

NOTE 2: This tool removes matching documents ONLY from segment indexes (or from a merged index). In particular it does NOT remove the pages and links from WebDB. This means that unwanted URLs may pop up again when new segments are created. To prevent this, use your own link net.nutch.net.URLFilter, or PruneDBTool (under construction...).

NOTE 3: This tool uses a low-level Lucene interface to collect all matching documents. For large indexes and broad queries this may result in high memory consumption. If you encounter OutOfMemory exceptions, try to narrow down your queries, or increase the heap size.

readdb = net.nutch.db.WebDBReader

The WebDBReader implements all the read-only parts of accessing our web database. All the writing ones can be found in WebDBWriter.

segread = net.nutch.segment.SegmentReader

This class holds together all data readers for an existing segment. Some convenience methods are also provided, to read from the segment and to reposition the current pointer.

segslice = net.nutch.segment.SegmentSlicer

This class reads data from one or more input segments, and outputs it to one or more output segments, optionally deleting the input segments when it's finished.

Data is read sequentially from input segments, and appended to output segment until it reaches the target count of entries, at which point the next output segment is created, and so on.

NOTE 1: this tool does NOT de-duplicate data - use SegmentMergeTool for that.

NOTE 2: this tool does NOT copy indexes. It is currently impossible to slice Lucene indexes. The proper procedure is first to create slices, and then to index them.

NOTE 3: if one or more input segments are in non-parsed format, the output segments will also use non-parsed format. This means that any parseData and parseText data from input segments will NOT be copied to the output segments.

server = 'net.nutch.searcher.DistributedSearch'

Implements the search API over IPC connnections.

updatedb = net.nutch.tools.UpdateDatabaseTool

This class takes the output of the fetcher and updates the page and link DBs accordingly. Eventually, as the database scales, this will broken into several phases, each consuming and emitting batch files, but, for now, we're doing it all here.

Version 0.3

Compiled by Rob Pettengill 05/07/07