找到以下幾行程式碼並註釋掉就OK了。 

if (!rules.isAllowed(fit.u)) {
                // unblock
                fetchQueues.finishFetchItem(fit, true);
                if (LOG.isDebugEnabled()) {
                  LOG.debug("Denied by robots.txt: " + fit.url);
                }
                output(fit.url, fit.datum, null, ProtocolStatus.STATUS_ROBOTS_DENIED, CrawlDatum.STATUS_FETCH_GONE);
                reporter.incrCounter("FetcherStatus", "robots_denied", 1);
                continue;
              }
 

後來發現請求部分頁面的連結返回的http狀態為301，跳轉之後才會到真正頁面，而nutch預設是不會抓取跳轉後的頁面的.這時需要修改nutch-site.xml，加入以下配置即可，nutch-default.xml裡的預設值是0，我們這裡改成一個大於0的值，nutch就會繼續抓取跳轉後的頁面了。 

<property>
  <name>http.redirect.max</name>
  <value>2</value>
  <description>The maximum number of redirects the fetcher will follow when
  trying to fetch a page. If set to negative or 0, fetcher won't immediately
  follow redirected URLs, instead it will record them for later fetching.
  </description>
</property>
 

3. 抽取的過程中發現某些屬性老是抽不到，而在不使用nutch抓取的情況下是能抽到的，進而懷疑nutch抓取的頁面不全。於是去google了一下"nutch content limit"，發現nutch有這麼一個配置項： 

<property>
  <name>http.content.limit</name>
  <value>65536</value>
  <description>The length limit for downloaded content using the http://
  protocol, in bytes. If this value is nonnegative (>=0), content longer
  than it will be truncated; otherwise, no truncation at all. Do not
  confuse this setting with the file.content.limit setting.
  </description>
</property>
 

需要注意的是nutch還有一個很容易混淆的配置項： 

<property>
  <name>file.content.limit</name>
  <value>65536</value>
  <description>The length limit for downloaded content using the file://
  protocol, in bytes. If this value is nonnegative (>=0), content longer
  than it will be truncated; otherwise, no truncation at all. Do not
  confuse this setting with the http.content.limit setting.
  </description>
</property>