威脅獵殺實戰（一）：平臺

摘要： 在國內Threat Hunting常被翻譯成威脅追蹤或威脅狩獵，我們認為：“未知攻焉知防，未知防焉知攻”。藍方並不一定要處於被動防守的狀態，完全可以主動獵殺對手！ 本文是威脅獵殺實戰系列的第一篇，按照本文的操作步驟，只需幾次Copy&Paste即可搭建一套基於Elastic Sta...

在國內Threat Hunting常被翻譯成威脅追蹤或威脅狩獵，我們認為：“未知攻焉知防，未知防焉知攻”。藍方並不一定要處於被動防守的狀態，完全可以主動獵殺對手！

本文是威脅獵殺實戰系列的第一篇，按照本文的操作步驟，只需幾次Copy&Paste即可搭建一套基於Elastic Stack的威脅獵殺平臺。在後面的文章我們會進一步完善我們的平臺。

NSM架構

目錄：

1.部署Elastic Stack（容器化）

$ echo "nameserver 9.9.9.9" > /etc/resolv.conf
$ git clone https://github.com/Zer0d0y/docker-elk.git
$ docker-compose build && docker-compose up -d
訪問Kibana web UI：http://localhost:5601
完整指南參考
https://github.com/Zer0d0y/docker-elk

2.部署Bro

2.1 安裝

方式一：使用官方提供的Binary軟體包

Ubuntu 16.04：
$ wget https://github.com/tianyulab/Threat_Hunting_with_ELK/raw/master/Bro/Install_Bro_from_binary.sh$ chmod +x Install_Bro_from_binary.sh
&& ./Install_Bro_from_binary.sh

Bro repository提供5個Binary軟體包：

• Bro，包含meta-package

• bro-core，包含Bro core和scripts

• broctl，包含Bro control

• libbroccoli和libbroccoli-dev，包含libbroccoli及其開發標頭檔案

Ubuntu 16.04：
$ wget -nv http://download.opensuse.org/repositories/network:bro/xUbuntu_16.04/Release.key -O Release.key
$ sudo apt-key add - < Release.key
$ sudo apt-get update
$ sudo sh -c "echo 'deb http://download.opensuse.org/repositories/network:/bro/xUbuntu_16.04/ /' > /etc/apt/sources.list.d/bro.list"
$ sudo apt-get update
$ sudo apt-get install bro
# 注意：官方同時提供nightly binary builds：https://www.bro.org/download/nightly-packages.html

方式二：原始碼安裝

依賴軟體包：

$ cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev

其他依賴軟體包（可選）：

參考：https://www.bro.org/sphinx/install/install.html#id6

Ubuntu 16.04：
$ wget https://github.com/tianyulab/Threat_Hunting_with_ELK/raw/master/Bro/Install_Bro_from_source.sh
$ chmod +x Install_Bro_from_source.sh && ./Install_Bro_from_source.sh# 
注意：也可以安裝Bro開發版：https://www.bro.org/sphinx/install/install.html#id9

方式三：容器化方式（Docker）

參考：https://github.com/bro/bro-docker

2.2 配置

2.2.1 Bro配置檔案

$PREFIX == 預設值：/opt/bro或/usr/local/bro
配置監聽網路介面：$PREFIX/etc/node.cfg
配置本地網路地址：$PREFIX/etc/networks.cfg
主配置檔案：$PREFIX/etc/broctl.cfg
# 完整配置參考：https://www.zer0d0y.info/post/Bro-plus-ELK/

2.2.2 使用systemd管理Bro

# 修改Bro介面名稱
$ INAME=$(ip -o link show | sed -rn '/^[0-9]+: en/{s/.: ([^:]*):.*/\1/p}')
$ sed -i "s/eth0/$INAME/g" /usr/local/bro/etc/node.cfg
$ wget https://github.com/tianyulab/Threat_Hunting_with_ELK/raw/master/Bro/Bro_systemd.service -O /etc/systemd/system/bro.service
$ systemctl daemon-reload
$ systemctl enable bro
$ systemctl start bro

3.整合Elastic Stack，[Kafka]和Bro

3.1 Bro日誌101

conn.log -- IP, TCP, UDP, ICMP
dhcp.log -- DHCPdns.log -- DNS查詢/響應
ftp.log -- FTP請求/響應
http.log -- HTTP請求/響應
files.log -- 檔案還原
mysql.log -- MySQL
irc.log -- IRC
radius.log -- RADIUS認證
kerberos.log -- Kerberos認證
sip.log -- SIP協議
smtp.log -- SMTP事務
ssl.log -- SSL握手
ssh.log -- SSH握手
syslog.log -- Syslog訊息
tunnel.log -- 封裝隧道的細節
Microsoft相關的日誌
dce_rpc.log -- DCE/RPC訊息
ntlm.log -- NTLMrdp.log -- 遠端桌面 (RDP)
smb_files.log -- SMB檔案傳輸
smb_mapping.log -- SMB管道

# 詳細解釋：https://github.com/corelight/bro-cheatsheets

3.2 使用Elastic Stack直接處理Bro的csv格式日誌

# 注意事項
1.埠開放（--> 防火牆）：
elasticsearch:9200
Logstash:5044
Kibana:5061

2."index => "bro_logs-%{+YYYY.MM.dd}"",其中index名稱必須小寫
3.建立Index Patterns前必須有對應Bro的日誌，否則會導致Field不全
# 注意事項

# 軟體環境
Elastic Stack 6.4
bro version 2.5.4


# 方式一：使用Filebeat處理Bro日誌，
資料流：
Bro --> Filebeat --> ELK(Logstash)


1.安裝Filebeat

Ubuntu 16.04：
$ wget https://github.com/tianyulab/Threat_Hunting_with_ELK/raw/master/Bro/Install_Filebeat.sh
$ chmod +x Install_Filebeat.sh && ./Install_Filebeat.sh

2.配置ELK(Logstash)接收來自FileBeat收集的Bro日誌
# 注意：此命令在ELK主機上執行
$ wget https://github.com/tianyulab/Threat_Hunting_with_ELK/raw/master/Bro/Deploy_Bro_Filebeat_Logstash.sh
$ chmod +x Deploy_Bro_Filebeat_Logstash.sh && ./Deploy_Bro_Filebeat_Logstash.sh
$ sed -i 's/8.8.8.8/ELK IP/g' Bro_Filebeat_Logstash.conf
$ systemctl start logstash.service


3.配置Filebeat處理Bro日誌
$ wget https://github.com/tianyulab/Threat_Hunting_with_ELK/raw/master/Bro/Deploy_Filebeat.sh
$ chmod +x Deploy_Filebeat.sh && ./Deploy_Filebeat.sh
$ sed -i 's/8.8.8.8/ELK logstash IP/g' /etc/filebeat/filebeat.yml
$ service filebeat start

4.訪問Kibana web UI：http://localhost:5601，新增"Index Patterns"
正常情況下，欄位(Fields) >= 218


# 方式二：使用Logstash處理Bro日誌，
資料流：
Bro --> Logstash --> ELK(Elasticsearch)

1.安裝Logstash
Ubuntu 16.04：
$ wget https://github.com/tianyulab/Threat_Hunting_with_ELK/raw/master/Bro/Install_Logstash.sh
$ chmod +x Install_Logstash.sh && ./Install_Logstash.sh

2.配置Logstash處理Bro日誌
# 注意：如ELK和Bro不在同一臺伺服器上，需要修改配置檔案中elasticsearch的值，如： hosts => ["ELK IP:9200"]
# sed -i 's/localhost/ELK IP/g' bro*.conf

$ cd /etc/logstash/conf.d
$ wget https://github.com/tianyulab/Threat_Hunting_with_ELK/raw/master/Bro/Deploy_Logstash.sh
$ chmod +x Deploy_Logstash.sh && ./Deploy_Logstash.sh
$ rm -f Deploy_Logstash.sh

3.訪問Kibana web UI：http://localhost:5601，新增“Index Patterns”

# 除錯&排錯
## Logstash
$ mkdir -p /root/xxx/logs && cd /root/xxx
$ /usr/share/logstash/bin/logstash -f xxx.conf --path.logs /root/xxx/logs --log.level=debug --config.debug --config.test_and_exit

$ /usr/share/logstash/bin/logstash -f nmap-logstash.conf --path.logs /root/xxx/logs/ --log.level=debug --config.debug 2>&1 | tee /root/xxx/logs/101
## FileBeat
$ filebeat -e -d "*" -c /etc/filebeat/filebeat.yml

# 容器化ELK專案對應配置（https://github.com/Zer0d0y/docker-elk）
1.docker-elk/docker-compose.yml
logstash:
ports:
- "5044:5044"

2.docker-elk/logstash/pipeline/bro_logs.conf
3.docker-compose build

3.3 使用Elastic Stack + Kafka處理Bro的json格式日誌

資料流：Bro --> Kafka --> Logstash --> ELK(Elasticsearch)

3.3.1 安裝Kafka

# 軟體環境：
# Ubuntu 16.04
# Elastic Stack 6.4
# Bro 2.5.5
# Kafka 2.12
# librdkafka-0.9.4

# 1.安裝Kafka
# 建立臨時目錄
mkdir /src && cd /src

# 下載&驗證kafka
wget https://archive.apache.org/dist/kafka/1.0.0/kafka_2.12-1.0.0.tgz
wget https://archive.apache.org/dist/kafka/1.0.0/kafka_2.12-1.0.0.tgz.asc
gpg --recv-keys3B417B9B 
gpg -v kafka_2.12-1.0.0.tgz.asc

# 安裝&啟動kafka服務
tar -xf kafka_2.12-1.0.0.tgz
sudo mv kafka_2.12-1.0.0 /opt/kafka
sudo sed -i '/^log.dirs/{s/=.*//;}' /opt/kafka/config/server.properties
sudo sed -i 's/^log.dirs/log.dirs=\/var\/lib\/kafka/' /opt/kafka/config/server.properties
sudo sed -i '$alisteners=bro://BRO所在機器的IP地址:9092' /opt/kafka/config/server.properties 

cat > /etc/systemd/system/kafka.service << EOF
[Unit]
Description=Kafka Service
Wants=network.target
After=zookeeper.target

[Service]
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.pr
ExecReload=on-failure
Restart=always
User=root
Group=root
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=multi-user.target
EOF

# 
sudo apt-get -y install zookeeperd
sudo systemctl enable zookeeper 
sudo systemctl start zookeeper
sudo systemctl daemon-reload
sudo systemctl enable kafka 
sudo systemctl start kafka

3.3.2 安裝kafka外掛（metron-bro-plugin-kafka）

## 安裝librdkafka
curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz | tar
cd librdkafka-0.9.4/ 
./configure --enable-sasl 
make 
sudo make install 
## 構建外掛
### 先安裝Bro 2.5.5
cd /src
wget https://github.com/tianyulab/Threat_Hunting_with_ELK/raw/master/Bro/Install_Bro_from_source.sh
chmod +x Install_Bro_from_source.sh && ./Install_Bro_from_source.sh


git clone https://github.com/apache/metron-bro-plugin-kafka.git
cd metron-bro-plugin-kafka
./configure --bro-dist=/src/bro-2.5.5/
make 
sudo make install
## 驗證
/usr/local/bro/bin/bro -N Apache::Kafka

3.3.3 配置Bro把日誌傳送到Kafka

$ vi /usr/local/bro/share/bro/site/local.bro
@load /usr/local/bro/lib/bro/plugins/APACHE_KAFKA/scripts/Apache/Kafka/logs-to-kafka.bro
redef Kafka::topic_name = "";
redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG, SMTP::LOG, SSL::LOG, Software::LOG, DHCP::LOG, FTP::LOG, IRC::LOG, Notice::LOG, X509::LOG, SSH::LOG,
redef Kafka::kafka_conf = table(["metadata.broker.list"] = "BRO所在機器的IP地址:9092");
redef Kafka::tag_json = T;

3.3.4 配置Logstash接收Kafka日誌

## 先安裝Logstash
$ wget https://github.com/tianyulab/Threat_Hunting_with_ELK/raw/master/Bro/Install_Logstash.sh
$ chmod +x Install_Logstash.sh && ./Install_Logstash.sh
$ echo config.reload.automatic: true |sudo tee -a /etc/logstash/logstash.yml
$ echo config.reload.interval: 3s |sudo tee -a /etc/logstash/logstash.yml
# 以Bro conn日誌為例：
$ cat > /etc/logstash/conf.d/bro-conn.conf << EOF
input {
kafka {
topics => ["conn"]
group_id => "bro_logstash"
bootstrap_servers => "10.42.94.92:9092"
codec => json
type => "conn"
auto_offset_reset => "earliest"
}
}

output {
if [type] == "conn" {
elasticsearch {
hosts => ["192.168.8.112:9200"]
index => "bro-conn-%{+YYYY.MM.dd}"
}
}
}
EOF

3.3.5 一鍵部署指令碼

$ wget 
# 修改10.42.94.92 --> 為Kafka監聽IP
$ sed -i 's/10.42.94.92/Kafka監聽IP/g' Deploy_Kafka_for_Bro.sh
# 修改192.168.8.112 --> 為Elasticsearch監聽IP
$ sed -i 's/192.168.8.112/Elasticsearch監聽IP/g' Deploy_Kafka_for_Bro.sh
# 修改"BRO所在機器的IP地址"為BRO所在機器的IP地址
$ sed -i 's/BRO所在機器的IP地址/BRO所在機器的IP地址/g' Deploy_Kafka_for_Bro.sh
$ sh -x Deploy_Kafka_for_Bro.sh

# 驗證
$ sudo systemctl status zookeeper
$ sudo systemctl status kafka
$ systemctl status logstash
$ /usr/local/bro/bin/bro -N Apache::Kafka
$ /usr/local/bro/bin/broctl status
$ netstat -tunlp| grep -E '2181|9092|9600'

# 安裝過程排錯
$ watch tail log.out
$ cat log.out | grep error
$ cat log.out | grep -B 10 "Configuring incomplete, errors occurred"
$ cat log.out | grep -i "cd librdkafka-0.9.4" -A 50 | more

# Kafka 排錯
$ apt-get install kafkacat
$ kafkacat -b 192.168.8.115:9092 -t http -o end # "http"為Bro的kafka外掛定義的"topics"
或
$ /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.8.115:9092 --topic http

致謝：

@HardenedLinux 團隊

@Rock NSM團隊

@Security Onion團隊