solr的索引可以接受不同的資料來源，包括：xml，csv檔案，資料庫表和常見的檔案格式的檔案（word，PDF）

有三種常用的方式來載入資料到solr的索引中：

1、Solr Cell 框架提取二進位制檔案或結構化檔案，如office word，PDF等

2、向solr伺服器傳送HTTP請求來載入xml檔案

3、利用solr的java API

solr的索引包含一個或多個document，一個document可以包含多個field，field可以為空，利用field： unique ID （和資料庫的主鍵類似），但該field不是必須的。

field的一般都會在schema.xml 檔案中相應的field對應，然後按照該檔案裡定義的步驟進行解析，如果在schema.xml沒有此filed，則會被忽略或者對映到schema.xml

中的dynamic field。

1、index handler

index handler是一種request handler，用來新增、刪除和更新document。另外，匯入大量的document的使用Tika或者Data Import Handler(用於結構化資料)、

XML：引數：Content-type:application/xml /Content-type: text/xml

curl http://localhost:8983/solr/my_collection/update -H "Content-Type: text/xml"
--data-binary '
<add>
<doc>
<field name="authors">Patrick Eagar</field>
<field name="subject">Sports</field>
<field name="dd">796.35</field>
<field name="isbn">0002166313</field>
<field name="yearpub">1982</field>
<field name="publisher">Collins</field>
</doc>
</add>'

Json：引數：Content-Type:application/json or Content-Type: text/json.

curl -X POST -H 'Content-Type: application/json'
'http://localhost:8983/solr/my_collection/update' --data-binary '
{
"add": {
"doc": {
"id": "DOC1",
"my_boosted_field": { /* use a map with boost/value for a boosted field
*/
"boost": 2.3,
"value": "test"
},
"my_multivalued_field": [ "aaa", "bbb" ] /* Can use an array for a
multi-valued field */
}
},
"add": {
"commitWithin": 5000, /* commit this document within 5 seconds */
"overwrite": false, /* don't check for existing documents with the
same uniqueKey */
"boost": 3.45, /* a document boost */
"doc": {
"f1": "v1", /* Can use repeated keys for a multi-valued field
*/
"f1": "v2"
}
},
"commit": {},
"optimize": { "waitSearcher":false },
"delete": { "id":"ID" }, /* delete by ID */
"delete": { "query":"QUERY" } /* delete by query */
}'

----------------------------------------------------------------------------------------------------------------------

note: Comments are not allowed in JSON, but duplicate names are

2、Solr Cell using Apache Tika//Apache Tika 利用現有的解析類庫，從不同格式的文件中（例如HTML, PDF, Doc)，偵測和提取出元資料和結構化內容

ExtractingRequestHandler;

3、DataImpothandler（重點）

The Data Import Handler (DIH) provides a mechanism for importing content from a data store and
indexing it. In addition to relational databases, DIH can index content from HTTP based data sources such as
RSS and ATOM feeds, e-mail repositories, and structured XML where an XPath processor is used to generate
fields.

概念與術語

DataSource：資料來源，基於URL

Entity：實體，一個實體被處理為一串document，包含多個欄位，然後會被solr作為索引。對於關係型資料庫，一個實體就是一個檢視或一張表，然後被處理成一個或多個sql語句，對應裡一行或多行行記錄（一條記錄對應一個document）和一列或多列（一列對應一個field）

Processor：處理器。用來衝資料來源中提取資料，轉換成document新增到索引中。一個自定義的實體處理器可以被重寫來擴充套件或代替一個supplied。

Tranformer：轉換器，修改field，建立一個新的field或通過一行記錄形成多個document。主要用來修改日期和過濾HTML。可以使用公共可用介面自定義一個轉換器。

配置：

solrconfig.xm;

<requestHandler name="/dataimport"
class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">/path/to/my/DIHconfigfile.xml</str>
</lst>
</requestHandler>

--------------------------------------------------------------------------------------------

(example/example-DIH/solr/db/conf/db-data-config.xml).

<dataConfig>
<dataSource driver="org.hsqldb.jdbcDriver"
url="jdbc:hsqldb:./example-DIH/hsqldb/ex" user="sa" password="secret"/>

<document>
<entity name="item" query="select * from item"
deltaQuery="select id from item where last_modified >
'${dataimporter.last_index_time}'">
<field column="NAME" name="name" />

<entity name="feature"
query="select DESCRIPTION from FEATURE where ITEM_ID='${item.ID}'"
deltaQuery="select ITEM_ID from FEATURE where last_modified >
'${dataimporter.last_index_time}'"
parentDeltaQuery="select ID from item where ID=${feature.ITEM_ID}">
<field name="features" column="DESCRIPTION" />
</entity>
<entity name="item_category"
query="select CATEGORY_ID from item_category where
ITEM_ID='${item.ID}'"
deltaQuery="select ITEM_ID, CATEGORY_ID from item_category where
last_modified > '${dataimporter.last_index_time}'"
parentDeltaQuery="select ID from item where
ID=${item_category.ITEM_ID}">
<entity name="category"
query="select DESCRIPTION from category where ID =
'${item_category.CATEGORY_ID}'"

deltaQuery="select ID from category where last_modified >
'${dataimporter.last_index_time}'"
parentDeltaQuery="select ITEM_ID, CATEGORY_ID from item_category
where CATEGORY_ID=${category.ID}">
<field column="description" name="cat" />
</entity>
</entity>

</entity>
</document>
</dataConfig>

a、Data Import Handler Commands、

DIH通過HTTPrequest傳送到solr。

引數common：

abort（停止正在進行的操作）；

delta-import（增量匯入和變化檢測，也支援clean、commit、optimize and debug parameters as full-import command）（SqlEntityProcessor）

full-import::該操作會立即返回，也將開啟一個新的執行緒，並且status屬性為busy，會儲存操作開始時間在conf/dataimport。properties以用來以後的delta-import

reload-config:如果配置檔案更改後，又不想重啟solr

status：It returns statistics on the number of documents created, deleted,queries run, rows fetched, status, and so on.

show-config、

---------------------------------------------------------------

Property Writer：

<propertyWriter dateFormat="yyyy-MM-dd HH:mm:ss" type="SimplePropertiesWriter"
directory="data" filename="my_dih.properties" locale="en_US" />

資料來源：

你可以自定義一個數據源通過繼承org.apache.solr.handler.dataimport.DataSource

可以使用的資料來源型別：

ContentStreamDataSource：EntityProcesssor + DataSource<Reader>

Field ReaderDataSource:XML;XPathEntityProcessor + JDBC + FieldReader

<dataSource name="a1" driver="org.hsqldb.jdbcDriver" ... />
<dataSource name="a2" type=FieldReaderDataSource" />
<document>
<!-- processor for database -->
<entity name ="e1" dataSource="a1" processor="SqlEntityProcessor" pk="docid"
query="select * from t1 ...">
<!-- nested XpathEntity; the field in the parent which is to be used for
Xpath is set in the "datafield" attribute in place of the "url" attribute
-->
<entity name="e2" dataSource="a2" processor="XPathEntityProcessor"
dataField="e1.fieldToUseForXPath">
<!-- Xpath configuration follows --
...
</entity>

FileDataSource:和URLDataSource相同，但是它用來從磁碟抓取資料 屬性：basePath、encoding

JdbcDatasource：預設資料來源和SqlEntityProcessor聯合使用

URLDataSource：XPathEntityProcessor

<dataSource name="a"
type="URLDataSource"
baseUrl="http://host:port/"
encoding="UTF-8"
connectionTimeout="5000"
readTimeout="10000"/>

---------------------------------------------------------------------------------------

Entity Processors

屬性名：

datasource

name:requerid,唯一識別一個實體。

pk

processor：預設是SqlEntityProcessor當不是關係型資料庫時是必須的含有的屬性

onError：abort|skip|continue,廢棄|跳過|繼續使用該document

preimportDeleteQuery:在完全匯入資料之前，來清除索引

postImportDeleteQuery:匯入資料之後執行

rootEntity：

transformer：Optional。

cacheImpl"SortedMapBackedCache

cacheKey

cacheLookup

where

<entity name="product" query="select description,sku, manu from product" >
<entity name="manufacturer" query="select id, name from manufacturer"
cacheKey="id" cacheLookup="product.manu" cacheImpl="SortedMapBackedCache"/>
</entity>

the sqlentity processor

屬性：

query:requited

deltaQuery:

parentDeltaQuery

deletedPKQuery

deltaImportQuery

The XPathEntityProcessor

<dataConfig>
<dataSource type="HttpDataSource" />
<document>
<entity name="slashdot"
pk="link"
url="http://rss.slashdot.org/Slashdot/slashdot"
processor="XPathEntityProcessor"
<!-- forEach sets up a processing loop ; here there are two
expressions-->
forEach="/RDF/channel | /RDF/item"
transformer="DateFormatTransformer">
<field column="source" xpath="/RDF/channel/title" commonField="true" />
<field column="source-link" xpath="/RDF/channel/link" commonField="true"/>
<field column="subject" xpath="/RDF/channel/subject" commonField="true" />
<field column="title" xpath="/RDF/item/title" />
<field column="link" xpath="/RDF/item/link" />
<field column="description" xpath="/RDF/item/description" />
<field column="creator" xpath="/RDF/item/creator" />
<field column="item-subject" xpath="/RDF/item/subject" />
<field column="date" xpath="/RDF/item/date"
dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss" />
<field column="slash-department" xpath="/RDF/item/department" />
<field column="slash-section" xpath="/RDF/item/section" />
<field column="slash-comments" xpath="/RDF/item/comments" />
</entity>
</document>
</dataConfig>

The MailEntityProcessor

<dataConfig>
<document>
<entity processor="MailEntityProcessor"
user="[email protected]"
password="password"
host="imap.gmail.com"
protocol="imaps"
fetchMailsSince="2009-09-20 00:00:00"
batchSize="20"
folders="inbox"
processAttachement="false"
name="sample_entity"/>
</document>
</dataConfig>

The TikaEntityProcessor

<dataConfig>
<dataSource type="BinFileDataSource" />
<document>
<entity name="tika-test" processor="TikaEntityProcessor"
url="../contrib/extraction/src/test-files/extraction/solr-word.pdf"
format="text">
<field column="Author" name="author" meta="true"/>
<field column="title" name="title" meta="true"/>
<field column="text" name="text"/>
</entity>
</document>
</dataConfig>