Elasticsearch.net專案實戰

摘要： elasticsearch.net專案實戰 @智客幸達 目錄 Elasticsearch+kibana 環境搭建 windows 10環境配置 安裝Elasticsearch head...

elasticsearch.net專案實戰

@智客幸達

目錄

• Elasticsearch+kibana
  • 環境搭建
    • windows 10環境配置
    • 安裝Elasticsearch
    • head安裝(非必需)
    • 安裝kibana
  • DSL的基本使用
    • 增加
    • 修改
    • 查詢
    • 刪除
• Elasticsearch .Net
  • Low level client基本使用
  • 專案實戰
• 總結
• 參考

Elasticsearch是一個基於Apache Lucene(TM)的開源搜尋引擎。無論在開源還是專有領域，Lucene可以被認為是迄今為止最

先進、效能最好的、功能最全的搜尋引擎庫。

一說到全文搜尋,lucene久負盛名。早年間,因為專案需要,接觸過一個叫盤古分詞的開源專案,藉助其中的分詞實現了分詞搜尋的功能。而盤古分詞就是lucence的.NET版本。據說這個開源專案已經恢復更新並支援. NET Core,有興趣的童鞋可以去圍觀一下(https://github.com/LonghronShen/Lucene.Net.Analysis.PanGu/tree/netcore2.0)。

我想很多童鞋都聽過ELK,ELK是Elasticsearch、Logstash、Kibana。正好公司運維同事引入了這樣一套體系,用於建立集中式日誌收集系統，將所有節點上的日誌統一收集，管理，訪問。雖然能夠從一定程度上解決基本的問題,但是原生的kibana介面和查詢方式都不夠友好,很難推向廣大的開發人員。於是我在想，我們是否可以利用這個開源的庫整合到運維自動化平臺當中,讓這把利劍發揮出更大的價值。

一、環境搭建

本文是基於windows 10作業系統的es環境的搭建。

1. java環境安裝

由於es是java語言開發的,所以這裡要安裝java環境。

jdk下載:

https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

2.安裝Elasticsearch

Elasticsearch版本已經比較多,初學者可能比較懵。特別是在安裝head和Kibana的時候,如果版本不匹配,往往會導致無法使用。這裡使用的是elasticsearch-5.6.11版本。

elasticsearch-5.6.11下載:

https://www.elastic.co/downloads/past-releases/elasticsearch-5-6-11

解壓到C:\ELk 備用。

3.head安裝(非必需)

es 4.x 版本安裝head很簡單,只需下載head外掛解壓到指定目錄即可。es 5.x+需要藉助node安裝。

head下載:

https://github.com/mobz/elasticsearch-head

node下載:

https://nodejs.org/dist/v8.12.0/node-v8.12.0-win-x64.zip

path環境變數末尾 會自動增加 C:\Program Files\nodejs

安裝 phantomjs

官網:http://phantomjs.org/下載【配置環境變數】

安裝grunt

npm install -g grunt-cli

執行C:\ELk\elasticsearch-5.6.11\bin\elasticsearch.bat

4.安裝kibana

導致為止,其實elasticsearch自身已經安裝完成。通過Head就能很方便的操作es,但是kibana集成了head類似功能,並提供了更加友好的訪問介面。

kibana-5.6.9-windows-x86下載:

https://www.elastic.co/downloads/past-releases/kibana-5-6-9

下載之後,解壓到C:\ELk\kibana-5.6.9-windows-x86

執行C:\ELk\kibana-5.6.9-windows-x86\bin\kibana.bat

二、DSL的基本使用

elasticsearch也像mysql一樣提供了專門的語法來操作資料。Elasticsearch provides a full Query DSL (Domain Specific Language) based on JSON to define queries.

• 建立文件

PUT people/person/1?op_type=create
{
"user" : "kimchy",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}

• 修改

POST /user/guest/20/_update
{
"doc": {
"RealName":"LukyHuu20"
}
}

• 查詢

GET /user/guest/_search
{
"query": {
"match": {
"Id":22
}
}
}

• 刪除

DELETE /user/guest/15
{

}

三、Elasticsearch .Net

elasticsearch是以restfulAPI方式對外提供介面,並提供客戶端給多種語言使用。Elasticsearch uses standard RESTful APIs and JSON. We also build and maintain clients in many languages such as Java, Python, .NET, SQL, and PHP. Plus, our community has contributed many more. They’re easy to work with, feel natural to use, and, just like Elasticsearch, don't limit what you might want to do with them.

參考(https://www.elastic.co/products/elasticsearch)

1.Low level client基本使用

本文是介紹ES的.NET客戶端,Elasticsearch .Net - Low level client[5.x]

通過引入對應的版本的客戶端,便可通過C#操作ES。參考(https://www.elastic.co/guide/en/elasticsearch/client/net-api/5.x/elasticsearch-net.html)

連線

var settings = new ConnectionConfiguration(new Uri("http://example.com:9200"))
.RequestTimeout(TimeSpan.FromMinutes(2));

var lowlevelClient = new ElasticLowLevelClient(settings);

插入文件

var indexResponse = lowlevelClient.Index<byte[]>("user", "guest", user.Id.ToString(), user);
 byte[] responseBytes = indexResponse.Body;

更新文件

var searchResponse = lowlevelClient.Update<string>("user", "guest", id.ToString(), new
{
doc = new
{
RealName = realname,
Description = description
}
});

bool successful = searchResponse.Success;

查詢

var searchResponse = lowlevelClient.Search<string>("user", "guest", new
{
query = new
{
match = new
{
Id = id
}
}
});

bool successful = searchResponse.Success;

刪除

var searchResponse = lowlevelClient.Delete<string>("user", "guest", id.ToString());

bool successful = searchResponse.Success;

2.專案實戰

前面大致介紹了ES的安裝和基本使用。那麼,如何在專案中落地呢？

使用nuget安裝Elasticsearch.Net 5.6.4

Install-Package Elasticsearch.Net -Version 5.6.4

基本的增刪該查在專案中的實現上面已經有所介紹,這裡重點講一下查詢:

筆者使用的.NET MVC5 Web框架,對於返回的結果筆者做了一個簡單封裝：

public class ESearchRoot<T>
{
/// <summary>
/// 
/// </summary>
public int took { get; set; }
/// <summary>
/// 
/// </summary>
public string timed_out { get; set; }
/// <summary>
/// 
/// </summary>
public _shards _shards { get; set; }
/// <summary>
/// 
/// </summary>
public Hits<T> hits { get; set; }
}

public class _shards
{
/// <summary>
/// 
/// </summary>
public int total { get; set; }
/// <summary>
/// 
/// </summary>
public int successful { get; set; }
/// <summary>
/// 
/// </summary>
public int skipped { get; set; }
/// <summary>
/// 
/// </summary>
public int failed { get; set; }
}

public class HitsItem<T>
{
/// <summary>
/// 
/// </summary>
public string _index { get; set; }
/// <summary>
/// 
/// </summary>
public string _type { get; set; }
/// <summary>
/// 
/// </summary>
public string _id { get; set; }
/// <summary>
/// 
/// </summary>
public string _score { get; set; }
/// <summary>
/// 
/// </summary>
public T _source { get; set; }
/// <summary>
/// 
/// </summary>
public List<int> sort { get; set; }
/// <summary>
/// 
/// </summary>
public Highlight highlight { get; set; }
}

public class Hits<T>
{
/// <summary>
/// 
/// </summary>
public int total { get; set; }
/// <summary>
/// 
/// </summary>
public string max_score { get; set; }
/// <summary>
/// 
/// </summary>
public List<HitsItem<T>> hits { get; set; }
}

public class Highlight
{
/// <summary>
/// 
/// </summary>
public List<string> Description { get; set; }
}

因為soure返回的物件是不定的,所以使用了泛型。

本專案soure對應的類,user:

///<summary>
/// 
/// </summary>
public class User
{
/// <summary>
/// 
/// </summary>
public string Account { get; set; }
/// <summary>
/// 
/// </summary>
public string Phone { get; set; }
/// <summary>
/// 
/// </summary>
public string Email { get; set; }
/// <summary>
/// 
/// </summary>
public string RealName { get; set; }
/// <summary>
/// 
/// </summary>
public string CanReview { get; set; }
/// <summary>
/// 
/// </summary>
public string CanExcute { get; set; }
/// <summary>
/// 
/// </summary>
public string Avatar { get; set; }
/// <summary>
/// 
/// </summary>
public string IsUse { get; set; }
/// <summary>
/// 
/// </summary>
public int Id { get; set; }
/// <summary>
/// 
/// </summary>
public string Name { get; set; }
/// <summary>
/// 
/// </summary>
public string Description { get; set; }
/// <summary>
/// 
/// </summary>
public DateTime CreateTime { get; set; }
/// <summary>
/// 
/// </summary>
public DateTime ModifyTime { get; set; }
}

專案使用了帶條件的分頁查詢:

public List<AdminUser> GetBySomeWhere(string keyword, int limit, int pageSize, out int total)
{
List<AdminUser> users = new List<AdminUser>();

total = 0;
try
{
var settings = new ConnectionConfiguration(new Uri("http://localhost:9200/"))
.RequestTimeout(TimeSpan.FromMinutes(2));

var lowlevelClient = new ElasticLowLevelClient(settings);

//根據不同的引數 來構建不同的查詢條件
var request = new object();
if (!String.IsNullOrEmpty(keyword))
{
request = new
{
from = limit,
size = pageSize,
query = new
{
match = new
{
Description = keyword
}
},
highlight = new
{
fields = new
{
Description = new { }
}
},
sort = new
{
Id = new
{
order = "desc"
}
}
};
}
else
{
request = new
{
from = limit,
size = pageSize,
query = new
{
match_all = new
{

}
},
highlight = new
{
fields = new
{
Description = new { }
}
},
sort = new
{
Id = new
{
order = "desc"
}
}
};
}


var searchResponse = lowlevelClient.Search<string>("user", "guest", request);

bool successful = searchResponse.Success;
var responseJson = searchResponse.Body;

if (!successful)
{
return users;
}

ESearchRoot<User> root = JsonHelper.JSONStringObject<ESearchRoot<User>>(responseJson);
if (root != null)
{
total = root.hits.total;
foreach (HitsItem<User> item in root.hits.hits)
{
if (item._source != null)
{
string highlightDescription = String.Empty;
StringBuilder sbDs = new StringBuilder();
if (item.highlight != null && item.highlight.Description.Count > 0)
{
//ighlightDescription = item.highlight.Description[0];
foreach (var d in item.highlight.Description)
{
sbDs.Append(d);
}
highlightDescription = sbDs.ToString();
}

AdminUser user = new AdminUser
{
Id = item._source.Id,
RealName = item._source.RealName,
Account = item._source.Account,
Email = item._source.Email,
Phone = item._source.Phone,
//IsUse=item._source.IsUse,
Avatar = item._source.Avatar,
Description = item._source.Description,
HighlightDescription = highlightDescription,
CreateTime = item._source.CreateTime,
ModifyTime = item._source.ModifyTime
};
users.Add(user);
}
}
}

return users;
}
catch (ElasticsearchClientException ex)
{
//Log4Helper.Error
}
return users;
}

專案最終的效果如下:

四、總結

elasticsearch是很強大的開源工具,在實現全文搜尋上有其獨到之處,也是大資料的分析方面利器,值得大家深入去研究和實踐。

五、參考

• Elasticsearch權威指南
• ofollow,noindex" target="_blank">Elasticsearch.Net 5.x
• Elasticsearch Reference [5.6] » Document APIs
• elasticsearch-net-example

• ElasticSearch安裝及HEAD外掛配置