Fluentd並非是專用於日誌檔案收集的，而是一個通用的資訊收集、整理、轉發的流式資料處理工具，日誌收集只是它十分典型的一個運用場景。重要的是，Fluentd的日誌收集功能對容器支援十分完備，遠遠勝於Logstash等傳統日誌收集工具。一方面得益於Fluentd社群開發了幾種專用於Docker日誌檔案的收集外掛，這些外掛能夠在Fluentd收集完Docker日誌以後自動為它加上容器相關的資訊，比較推薦其中的fluent-plugin-docker-metadata-filter這一款外掛，它提供的資訊頗為齊全。Logstash對於這方面依然比較空缺，GitHub上唯一能夠找到的一款社群外掛也已經在一年前就停止開發。另一方面，當前Docker官方支援的日誌驅動除了預設的使用本地目錄，還可以直接傳送到遠端的日誌儲存或日誌採集服務，而其中日誌採集服務目前僅僅支援Splunk和Fluentd，同樣沒有Logstash等老一輩開源日誌工具的蹤影。

安裝elasticsearch

安裝JDK或者openJDK（這裡以openJDK為例）

[[email protected] elk]# yum install java-1.8.0-openjdk -y

安裝elasticsearch

[root@elk elk]# wget -c https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.3.3/elasticsearch-2.3.3.rpm
[root@elk elk]# yum localinstall elasticsearch-2.3.3.rpm -y
 

.....
  Installing : elasticsearch-2.3.3-1.noarch                                                                                             1/1
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 

 sudo systemctl start elasticsearch.service
  Verifying  : elasticsearch-2.3.3-1.noarch                                                                                             1/1

Installed:
  elasticsearch.noarch 0:2.3.3-1
[root@elk elk]# systemctl daemon-reload
[root@elk elk]# systemctl enable elasticsearch
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
[root@elk elk]# systemctl start elasticsearch

修改防火牆，開放9200和9300埠

[[email protected] elk]# firewall-cmd --permanent --add-port={9200/tcp,9300/tcp}
success
[[email protected] elk]# firewall-cmd --reload
success
[[email protected] elk]# firewall-cmd  --list-all
public (default, active)
  interfaces: eno16777984 eno33557248
  sources:
  services: dhcpv6-client ssh
  ports: 9200/tcp 9300/tcp
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

安裝kibana

安裝kibana的rpm包

[root@elk elk]# wget https://download.elastic.co/kibana/kibana/kibana-4.5.1-1.x86_64.rpm
[root@elk elk]# yum localinstall kibana-4.5.1-1.x86_64.rpm –y
[root@elk elk]# systemctl enable kibana
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /usr/lib/systemd/system/kibana.service.
[root@elk elk]# systemctl start kibana

修改防火牆，對外開放tcp/5601

[[email protected] elk]# firewall-cmd --permanent --add-port=5601/tcp
Success
[[email protected] elk]# firewall-cmd --reload
success
[[email protected] elk]# firewall-cmd --list-all
public (default, active)
  interfaces: eno16777984 eno33557248
  sources:
  services: dhcpv6-client ssh
  ports: 9200/tcp 9300/tcp 5601/tcp
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

開啟瀏覽器測試訪問kibana的首頁http://192.168.10.143:5601/

安裝fluent

安裝前步奏

檢視當前最大開啟檔案數：

$ ulimit -n
1024

如果檢視到的是1024，那麼這個數值是不足的，則需要修改配置檔案提高數值

vi /etc/security/limits.conf

設定值如下

root soft nofile 65536
root hard nofile 65536
* soft nofile 65536
* hard nofile 65536

修改network引數

vi /etc/sysctl.conf

設定值如下

net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240    65535

然後重啟系統

安裝fluentd

執行如下命令（命令將會自動安裝td-agent，td-agent即為fluentd）

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

啟動td-agent

$ /etc/init.d/td-agent start 
Starting td-agent: [  OK  ]
$ /etc/init.d/td-agent status
td-agent (pid  21678) is running...

或者

$ systemctl start td-agent
$ systemctl status td-agent

簡單demo測試HTTP logs

$ curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test

安裝必要的外掛

$ /usr/sbin/td-agent-gem install fluent-plugin-elasticsearch
$ /usr/sbin/td-agent-gem install fluent-plugin-typecast
$ /usr/sbin/td-agent-gem install fluent-plugin-secure-forward

$ systemctl restart td-agent

配置td-agent，使docker生成的日誌輸出到elasticsearch

修改td-agent的配置

$ vi /etc/td-agent/td-agent.conf

修改內容為如下（這裡我沒有填寫埠，預設使用9200埠）

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match alpine**>
  @type elasticsearch
  logstash_format true
  flush_interval 10s # for testing
  host 127.0.0.1
</match>

重啟td-agent

$ systemctl restart td-agent

啟動docker映象，這裡我使用alpine做測試

$ docker run --log-driver=fluentd --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" alpine:3.3 echo "helloWorld"

檢查fluentd是否正常轉發資料

檢查td-agent日誌，如果最後出現如下一行，則表明正常連線elasticsearch

2016-09-02 12:01:10 +0800 [info]: Connection opened to Elasticsearch cluster => {:host=>"127.0.0.1", :port=>9200, :scheme=>"http"}

檢查elasticsearch儲存，使用如下命令

$ curl localhost:9200/_search?pretty
{
  "took" : 472,
  "timed_out" : false,
  "_shards" : {
    "total" : 6,
    "successful" : 6,
    "failed" : 0
  },
  "hits" : {
    "total" : 3,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : ".kibana",
      "_type" : "config",
      "_id" : "4.5.4",
      "_score" : 1.0,
      "_source" : {
        "buildNum" : 10000
      }
    }, {
      "_index" : "logstash-2016.09.02",
      "_type" : "fluentd",
      "_id" : "AVbpDs1fOf1yFl8aApNq",
      "_score" : 1.0,
      "_source" : {
        "log" : "helloWorld",
        "container_id" : "8c58cf5be57e9905215ea8f8cd677e48594a384c30a2842facb6563c3f32097c",
        "container_name" : "/sleepy_elion",
        "source" : "stdout",
        "@timestamp" : "2016-09-02T12:00:59+08:00"
      }
    }, {
      "_index" : "logstash-2016.09.02",
      "_type" : "fluentd",
      "_id" : "AVbpDskPOf1yFl8aApNp",
      "_score" : 1.0,
      "_source" : {
        "log" : "helloWorld",
        "container_id" : "8c58cf5be57e9905215ea8f8cd677e48594a384c30a2842facb6563c3f32097c",
        "container_name" : "/sleepy_elion",
        "source" : "stdout",
        "@timestamp" : "2016-09-02T12:00:59+08:00"
      }
    } ]
  }
}

檢視到剛剛插入的資料，則表明資料已經寫入到elasticsearch了

進入kibana建立索引

點選“create”，即可看到如下設定

修改查詢條件，則可檢視到寫入的資料

至此，EFK的安裝與測試則已經完成