1. ansj分詞原始碼及jar包下載地址

原始碼：
https://github.com/NLPchina/ansj_seg

jar包：
http://maven.nlpcn.org/org/ansj/
http://maven.nlpcn.org/org/nlpcn/nlp-lang

2. 在solr使用ansj分詞

(1) ansj的solr擴充套件及編譯

ansj支援了lucene的擴充套件，使用下面幾個jar包：
ansj_lucene4_plug-2.0.2.jar
ansj_seg-2.0.8-min.jar 
nlp-lang-0.3.jar

要在solr中使用ansj，可以在lucene外掛的原始碼上做一下擴充套件：
外掛的程式碼目錄(maven工程)為ansj_seg/plug/ansj_lucene4_plug，匯入此maven工程，配置好依賴關係，增加一個solr擴充套件類AnsjTokenizerFactory.
編譯後生成一個新的ansj_lucene4_plug-2.0.2.jar包，重新命名為ansj_lucene4_plug-2.0.2-solr.jar.

package org.ansj.solr;
 
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.Reader;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;


import org.ansj.lucene.util.AnsjTokenizer;
import org.ansj.splitWord.analysis.IndexAnalysis;
import org.ansj.splitWord.analysis.ToAnalysis;
import org.apache.lucene.analysis.Tokenizer;
import org.apache.lucene.analysis.util.TokenizerFactory;
import org.apache.lucene.util.AttributeFactory;
 
public class AnsjTokenizerFactory extends TokenizerFactory{
    boolean pstemming;
    boolean isQuery;
    private String stopwordsDir;
    public Set<String> filter; 
 
    public AnsjTokenizerFactory(Map<String, String> args) {
        super(args);
        assureMatchVersion();
        isQuery = getBoolean(args, "isQuery", true);
        pstemming = getBoolean(args, "pstemming", false);
        stopwordsDir = get(args,"stopwords");
        addStopwords(stopwordsDir);
    }
    
    //add stopwords list to filter
    private void addStopwords(String dir) {
        if (dir == null){
            System.out.println("no stopwords dir");
            return;
        }
        //read stoplist
        System.out.println("stopwords: " + dir);
        filter = new HashSet<String>();
        File file = new File(dir);
        InputStreamReader reader;
        try {
            reader = new InputStreamReader(new FileInputStream(file),"UTF-8");
            BufferedReader br = new BufferedReader(reader);
            String word = br.readLine(); 
            while (word != null) {
                filter.add(word);
                word = br.readLine();
            } 
        } catch (FileNotFoundException e) {
            System.out.println("No stopword file found");
        } catch (IOException e) {
            System.out.println("stopword file io exception");
        }
    }
    
    @Override
    public Tokenizer create(AttributeFactory factory, Reader input) {
        if(isQuery == true){
            //query
            return new AnsjTokenizer(new ToAnalysis(new BufferedReader(input)), input, filter, pstemming);
        } else {
            //index
            return new AnsjTokenizer(new IndexAnalysis(new BufferedReader(input)), input, filter, pstemming);
        }
    }
}
 

(2) 在tomcat+solr中使用ansj分詞擴充套件並進行配置

將下列jar包放在${tomcat}/webapps/solr/WEB-INF/lib/目錄下：
ansj_lucene4_plug-2.0.2-solr.jar
ansj_seg-2.0.8-min.jar
nlp-lang-0.3.jar

solr資料集的schema.xml中增加text_ansj分詞配置：
    <fieldType name="text_ansj" class="solr.TextField" positionIncrementGap="100">
      <analyzer type="index">
         <tokenizer class="org.ansj.solr.AnsjTokenizerFactory"  isQuery="false" stopwords="/xxx/tomcat/apache-tomcat-8.0.9/webapps/solr/WEB-INF/classes/stopwords.dic"/>
      </analyzer>
      <analyzer type="query">
        <tokenizer class="org.ansj.solr.AnsjTokenizerFactory" stopwords="/xxx/tomcat/apache-tomcat-8.0.9/webapps/solr/WEB-INF/classes/stopwords.dic"/>
      </analyzer>
    </fieldType>

(3) 使用ansj分詞配置及自定義詞典
將下列檔案放在${tomcat}/webapps/solr/WEB-INF/classes目錄下：
ansj_seg/library
ansj_seg/train_file

ansj_seg/library.properties  

注意library.properties中的配置路徑，通常需要使用絕對路徑，相對路徑是相對於tomcat的啟動目錄而言的。

------

相對ik分詞，ansj準確度更高。

IK分詞在細粒度模式下有些錯誤的詞被分出來，英文分詞時會將單詞切割開：如javascript可能會分成ja,java,javascript; nagios可能被切分成nagios,ios, 檢索時會出現問題。