第一：背景介紹

常見的開源資料收集系統

• 非結構化日誌（資料）收集
  
  • flume
• 結構化日誌（資料）收集
  
  • sqoop全量匯入
  • canal（Alibaba）增量匯入
  • Databus（linkedin）增量匯入

第二：Flume（NG）介紹

- Event

• flume以事件的形式傳輸資料單元
• 事件由一個header和載有資料的byte array構成
• header是一個字典結構的資料，可以在上下文路由中擴充套件使用

- Client

• client是一個將原始log包裝成event並且傳送他們到一個或者多個agent的實體
• client不是必須的

- Agent

• source

  1. 接收或者產生event，並且批量傳送到一個或者多個channel

  2. 不同型別的source

     • 與系統整合的source：syslog，netCat
     • 自動生成事件的source：Exce
     • 監聽資料夾下檔案變化的：Spolling Directory Source，Talidir Source
     • 用於Agent與Agent通訊的IPC Source：Avro，Thrift

• channel

  1. 位於Source和Sink之間，用於快取event
  2. 支援事務
  3. 不同型別的channel
     
     • Memory Channel：volatile
     • File Channel：
     • JDBC Channel：
• sink
  
  1. 將event傳輸到下一步或者最終目的地，成功後將event從Channel中清除
  2. 不同型別的sink
     
     • 儲存event到最終終端的sink：HDFS，HBASE
     • 自動消耗的Channel：Null sink
     • 用於Agent之間通訊的IPC sink：Avro

第三：Sqooq介紹

• 傳統關係型資料庫和Hadoop之間的橋樑
  
  • 把關係型資料的資料匯入到hadoop系統
  • 把hadoop系統的資料匯入到關係型資料庫中
• 利用MapReduce加快資料傳輸速度
• 批處理方式進行資料傳輸

第四：CDC介紹

1. canal：
2. databus

第五：專案實踐

一，專案說明

將命名為record.list裡面不斷生成的內容，收集到hadoop叢集中。其中source採用exec的sourc，Channel採用file的Channel，sink採用hdfs的sink
 

二，安裝flume

[hadoop@hadoopa ~]$ tar -zxvf apache-flume-1.7.0-bin.tar.gz

三，配置flume

1. 配置 flume-conf-logAnalysis.properties

[hadoop@hadoopa conf]$ pwd
/home/hadoop/apache-flume-1.7.0-bin/conf
[hadoop@hadoopa conf]$ vi flume-conf-logAnalysis.properties 

[[email protected] conf]$ cat flume-conf-logAnalysis.properties
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#  http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.


# The configuration file needs to define the sources,
# the channels and the sinks.
# Sources, channels and sinks are defined per agent,
# in this case called 'agent'

logAgent.sources = logSource
logAgent.channels = fileChannel
logAgent.sinks = hdfsSink

# For each one of the sources, the type is defined
logAgent.sources.logSource.type = exec
logAgent.sources.logSource.command = tail -F /home/hadoop/hadooptraining/datasource/record.list

# The channel can be defined as follows.
logAgent.sources.logSource.channels = fileChannel

# Each sink's type must be defined
logAgent.sinks.hdfsSink.type = hdfs
logAgent.sinks.hdfsSink.hdfs.path = hdfs://hadoopA:8020/flume/record/%Y-%m-%d/%H%M
logAgent.sinks.hdfsSink.hdfs.filePrefix= transaction_log
logAgent.sinks.hdfsSink.hdfs.rollInterval= 600
logAgent.sinks.hdfsSink.hdfs.rollCount= 10000
logAgent.sinks.hdfsSink.hdfs.rollSize= 0
logAgent.sinks.hdfsSink.hdfs.round = true
logAgent.sinks.hdfsSink.hdfs.roundValue = 10
logAgent.sinks.hdfsSink.hdfs.roundUnit = minute
logAgent.sinks.hdfsSink.hdfs.fileType = DataStream
logAgent.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
#Specify the channel the sink should use
logAgent.sinks.hdfsSink.channel = fileChannel

# Each channel's type is defined.
logAgent.channels.fileChannel.type = file
logAgent.channels.fileChannel.checkpointDir= /home/hadoop/apache-flume-1.7.0-bin/dataCheckpointDir
logAgent.channels.fileChannel.dataDirs= /home/hadoop/apache-flume-1.7.0-bin/dataDir

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel

1. 配置 flume-env.sh

[hadoop@hadoopa conf]$ pwd
/home/hadoop/apache-flume-1.7.0-bin/conf
[hadoop@hadoopa conf]$ vi flume-env.sh

[[email protected] conf]$ cat flume-env.sh
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
export JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"

# Let Flume write raw event data and configuration information to its log files for debugging
# purposes. Enabling these flags is not recommended in production,
# as it may result in logging sensitive user information or encryption secrets.
# $JAVA_OPTS="$JAVA_OPTS -Dorg.apache.flume.log.rawdata=true -Dorg.apache.flume.log.printconfig=true "

# Foll. classpath will be included in Flume's classpath.
# Note that the Flume conf directory is always included in the classpath.
FLUME_CLASSPATH="$HADOOP_HOME/share/hadoop/common/hadoop-common-2.7.3.jar"   # Example:  "path1;path2;path3"

四，執行flume

[[email protected] conf]$ flume-ng agent --conf /home/hadoop/apache-flume-1.7.0-bin/conf --conf-file /home/hadoop/apache-flume-1.7.0-bin/conf/flume-conf-logAnalysis.properties --name logAgent -Dflume.root.logger=DEBUG,console -Dflume.monitoring.type=http -Dflume.monitoring.port=34545

五，驗證結果

通過瀏覽器訪問hdfs：

http://192.168.1.201:50070/explorer.html#/flume/record/