See these pages for more details on how to install and use Apache Nutch:
https://wiki.apache.org/nutch/NutchTutorial
https://wiki.apache.org/nutch/CommandLineOptions

See these pages for more details on how to install MongoDB, Solr, Zookeeper:
http://www.mtitek.com/tutorials/tools/mongodb.php
http://www.mtitek.com/tutorials/solr
http://www.mtitek.com/tutorials/zookeeper

Download Apache Nutch: https://nutch.apache.org/downloads.html

Extract the file "apache-nutch-2.3.1-src.tar.gz" in the folder you want to install Nutch: /opt/apache-nutch-2.3.1

$ tar -xf ~/Downloads/apache-nutch-2.3.1-src.tar.gz -C /opt/

Note: In the following sections, the environment variable ${NUTCH_SRC_HOME} will refer to this location '/opt/apache-nutch-2.3.1'

You need to choose the Gora backend for your Nutch.
For example, to use MongoDB as the Gora backend you need to enable "gora-mongodb" dependency in '${NUTCH_SRC_HOME}/ivy/ivy.xml':

<!-- Uncomment this to use MongoDB as Gora backend. -->
<dependency org="org.apache.gora" name="gora-mongodb" rev="0.6.1" conf="*->default" />

You are ready to compile Nutch:

$ cd ${NUTCH_SRC_HOME}
$ ant runtime

This will create the Nutch runtime directory '${NUTCH_SRC_HOME}/runtime'.
The folder '${NUTCH_SRC_HOME}/runtime/local' is your Nutch home folder.

Note: In the following sections, the environment variable ${NUTCH_HOME} will refer to this location '${NUTCH_SRC_HOME}/runtime/local'

For all the required configuration you need to modify files in this folder '${NUTCH_HOME}/conf'.

Run this command 'ant clean' to delete Nutch runtime directory '${NUTCH_SRC_HOME}/runtime':

$ cd ${NUTCH_SRC_HOME}
$ ant clean

Use this file '${NUTCH_HOME}/conf/nutch-site.xml':
► to configure MongoDB as the GORA backend.
► to configure HTTP agent name.
► to use Nutch with Solr.

<configuration>
    <property>
        <name>storage.data.store.class</name>
        <value>org.apache.gora.mongodb.store.MongoStore</value>
    </property>

    <property>
        <name>http.agent.name</name>
        <value>mtitek nutch</value>
    </property>

    <property>
        <name>plugin.includes</name>
        <value>protocol-http|urlfilter-regex|parse-html|index-(basic|anchor)|indexer-solr|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
    </property>
</configuration>

Use this file '${NUTCH_HOME}/conf/gora.properties' to configure MongoDBStore as the default datastore:

############################
# MongoDBStore properties  #
############################
gora.datastore.default=org.apache.gora.mongodb.store.MongoStore
gora.mongodb.override_hadoop_configuration=false
gora.mongodb.mapping.file=/gora-mongodb-mapping.xml
gora.mongodb.servers=localhost:27017
gora.mongodb.db=nutchdb
#gora.mongodb.login=login
#gora.mongodb.secret=secret

This section provide quick instructions on how to configure Nutch with Solr.

Download Apache Solr: http://lucene.apache.org/solr

Download Apache ZooKeeper: http://zookeeper.apache.org

Extract the file "solr-7.3.1.zip" in the folder you want to install Solr: /opt/solr-7.3.1

$ unzip ~/Downloads/solr-7.3.1.zip -d /opt/

Extract the file "zookeeper-3.5.4-beta.tar.gz" in the folder you want to install Zookeeper: /opt/zookeeper-3.5.4-beta

$ tar -xf ~/Downloads/zookeeper-3.5.4-beta.tar.gz -C /opt/

Start Zookeeper:

$ cd /opt/zookeeper-3.5.4-beta
$ bin/zkServer.sh start

For the purpose of this tutorial you can use the sample solr configuration provided by Solr 'sample_techproducts_configs'.
You can use this sample Solr schema managed-schema (you need to copy it in this folder: /opt/solr-7.3.1/server/solr/configsets/sample_techproducts_configs/conf)

Upload Solr config to ZooKeeper and create a new Solr collection:

$ cd /opt/solr-7.3.1/server/scripts/cloud-scripts

$ ./zkcli.sh \
-zkhost "localhost:2181/solr" \
-cmd upconfig \
-confname solr_config1 \
-confdir /opt/solr-7.3.1/server/solr/configsets/sample_techproducts_configs/conf

$ cd /opt/solr-7.3.1/

$ bin/solr start -c -z "localhost:2181/solr"

$ bin/solr create_collection -c collection1 -n solr_config1

Create a seed directory and create a file that contains the list of web sites to be crawled by Nutch:

$ cd ${NUTCH_HOME}
$ mkdir seedDir
$ echo 'http://localhost:8080/' > seedDir/url.txt

Start a crawling job:

$ cd ${NUTCH_HOME}
$ bin/crawl seedDir/ mtitek http://localhost:8983/solr/collection1 1

You can also follow this Step-by-Step Nutch commands to crawl web sites:

#Initialize the crawldb with the selected URLs:
$ bin/nutch inject seedDir/

#Generate a fetch list from the crawldb:
$ bin/nutch generate -topN 100

#Fetch URLs for the generated batchIds:
$ bin/nutch fetch -all

#Parse all fetched URLs:
$ bin/nutch parse -all

#Update the crawldb:
$ bin/nutch updatedb -all

Here's an example of the execution of these commands and their results in MongoDB.

Initialize the crawldb with the selected URLs:

$ bin/nutch inject seedDir/

$ mongo

> use nutchdb
switched to db nutchdb

> show collections
webpage

> db.webpage.find()
{
    "_id": "localhost:http/",
    "fetchTime": "1111111111111",
    "fetchInterval": 2592000,
    "score": 1,
    "markers": {
        "dist": "0",
        "_injmrk_": "y"
    },
    "metadata": {
        "_csh_": 0,
        "AAAAAA=="
    }
}

Generate a fetch list from the crawldb:

$ bin/nutch generate -topN 100

$ mongo

> use nutchdb
switched to db nutchdb

> db.webpage.find()
{
    "_id": "localhost:http/",
    "fetchTime": "1111111111111",
    "fetchInterval": 2592000,
    "score": 1,
    "markers": {
        "dist": "0",
        "_injmrk_": "y",
        "_gnmrk_": "11111111-22222222"
    },
    "metadata": {
        "_csh_": 0,
        "AAAAAA=="
    },
    "batchId": "11111111-22222222"
}

Fetch URLs for the generated batchIds:

$ bin/nutch fetch -all

$ mongo

> use nutchdb
switched to db nutchdb

> db.webpage.find()
{
    "_id": "localhost:http/",
    "fetchTime": "1111111111111",
    "fetchInterval": 2592000,
    "score": 1,
    "markers": {
        "dist": "0",
        "_injmrk_": "y",
        "_gnmrk_": "11111111-22222222",
        "_ftcmrk_": "11111111-22222222"
    },
    "metadata": {
        "_rs_": 0,
        "AAAA2g=="
    },
    "batchId": "11111111-22222222",
    "baseUrl": "http://localhost:8080/",
    "status": 2,
    "prevFetchTime": "11111111111111",
    "protocolStatus": {
        "code": 1,
        "lastModified": 0
    },
    "content": 0,
    "contentType": "application/xhtml+xml",
    "headers": {
        "Connection": "close",
        "Server": "Apache",
        "Content-Type": "text/html"
    }
}

Parse all fetched URLs:

$ bin/nutch parse -all

$ mongo

> use nutchdb
switched to db nutchdb

> db.webpage.find()
{
    "_id": "localhost:http/",
    "fetchTime": "1111111111111",
    "fetchInterval": 2592000,
    "score": 1,
    "markers": {
        "_injmrk_": "y",
        "__prsmrk__": "111111111-2222222222",
        "dist": "0",
        "_gnmrk_": "111111111-2222222222",
        "_ftcmrk_": "111111111-2222222222"
    },
    "metadata": {
        "CharEncodingForConversion": 0,
        "OriginalCharEncoding": 0
    },
    "batchId": "111111111-2222222222",
    "baseUrl": "http://localhost:8080/",
    "status": 2,
    "protocolStatus": {
        "code": 1,
        "lastModified": 0
    },
    "contentType": "application/xhtml+xml",
    "headers": {
        "Connection": "close",
        "Server": "Apache",
        "Content-Type": "text/html"
    },
    "title": "Tutorials: Java, EJB, WebSphere AS, Oracle, Ubuntu, Subversion, Nexus, SonarQube, Jenkins, …",
    "text": "Tutorials: Java, EJB, WebSphere AS, Oracle, Ubuntu, Subversion, Nexus, SonarQube, Jenkins, … mtitek.com   Home   Samples   Install   Tutorials   Contact  © mtitek.com",
    "outlinks": {
        "http://localhost:8080/tools·php": "Install"
    }
}

Update the crawldb:

$ bin/nutch updatedb -all

$ mongo

> use nutchdb
switched to db nutchdb

> db.webpage.find({},{_id:1})
{ "_id" : "localhost:http/" }
{ "_id" : "localhost:http/tools.php" }

> db.webpage.find()

{
    "_id": "localhost:http/",
    "fetchTime": "1111111111111",
    "fetchInterval": 2592000,
    "score": 1,
    "markers": {
        "_updmrk_": "1111111-22222222",
        "dist": "0",
        "_injmrk_": "y",
        "_gnmrk_": null,
        "_ftcmrk_": null,
        "__prsmrk__": null
    },
    "metadata": {
        "CharEncodingForConversion": 0,
        "dXRmLTg=",
        "OriginalCharEncoding": 0,
        "dXRmLTg=",
        "_rs_": 0,
        "caching·forbidden": 0,
        "_csh_": 0
    },
    "batchId": "111111111-222222222",
    "baseUrl": "http://localhost:8080/",
    "status": 2,
    "protocolStatus": {
        "code": 1,
        "lastModified": 0
    },
    "content": 0,
    "contentType": "application/xhtml+xml",
    "headers": {
        "Connection": "close",
        "Server": "Apache",
        "Content-Type": "text/html"
    },
    "signature": 0,
    "title": "Tutorials: Java, EJB, WebSphere AS, Oracle, Ubuntu, Subversion, Nexus, SonarQube, Jenkins, …",
    "text": "Tutorials: Java, EJB, WebSphere AS, Oracle, Ubuntu, Subversion, Nexus, SonarQube, Jenkins, … mtitek.com   Home   Samples   Install   Tutorials   Contact mtitek.com",
    "parseStatus": {
        "majorCode": 1,
        "minorCode": 0
    },
    "outlinks": {
        "http://localhost:8080/tools·php": "Install"
    },
    "retriesSinceFetch": 0,
    "modifiedTime": 0,
    "prevModifiedTime": 0,
    "inlinks": {
        "http://localhost:8080/": "mtitek.com"
    }
}

{
    "_id": "localhost:http/tools.php",
    "status": 1,
    "fetchTime": "1111111111111",
    "fetchInterval": 2592000,
    "retriesSinceFetch": 0,
    "score": 0,
    "inlinks": {
        "http://localhost:8080/": "Install"
    },
    "markers": {
        "dist": "1"
    },
    "metadata": {
        "_csh_": 0,
        "AAAAAA=="
    }
}