排查logstash2.4升級到5.0版本後kafka不相容問題
排查升級到版本後kafka不相容問題
參考文件:
/usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/ /usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/ /usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/ /usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/lib/logstash/inputs/ /usr/share/logstash/vendor/bundle/jruby//gems/logstash-output-kafka-/lib/logstash/outputs/
緣由:
之前對ELKB環境從版本升級到最新的穩定版本,主要升級步驟可以參考/3002256/1870205,後來發現kafka叢集執行報錯,現在把排查過程記錄如下,僅供參考
之前環境:
logstash-input-kafka-
logstash-output-kafka-
kafka_-
升級後環境:
logstash-input-kafka-
logstash-output-kafka-
報錯資訊:
[2016-11-16T14:35:44,739][ERROR][ ] Unknown setting 'zk_connect' for kafka [2016-11-16T14:35:44,741][ERROR][ ] Unknown setting 'topic_id' for kafka [2016-11-16T14:35:44,741][ERROR][ ] Unknown setting 'reset_beginning' for kafka
實施步驟:
1,根據錯誤檢視程式哪裡報錯
grep "Unknown setting" /usr/share/logstash/ -R /usr/share/logstash/logstash-core/lib/logstash/config/: ("Unknown setting '#{name}' for #{@plugin_name}")
2,檢視程式相關程式碼,發現需要檢視plugins的config定義檔案等
def validate_check_invalid_parameter_names(params) invalid_params = # Filter out parameters that match regexp keys. # These are defined in plugins like this: # config /foo.*/ => ... @_key do |config_key| if config__a?(Regexp) invalid_! { |k| k =~ config_key } elsif config__a?(String) invalid_! { |k| k == config_key } end end if invalid_ > 0 invalid_ do |name| ("Unknown setting '#{name}' for #{@plugin_name}") end return false end # if invalid_ > 0 return true end # def validate_check_invalid_parameter_names
3,進入外掛總目錄檢視具體資訊
cd /usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-
發現重點檢視如下檔案
grep config ./* -R |awk '{print $1}' |uniq ./: ./:See ./lib/logstash/inputs/:# ./lib/logstash/inputs/: ./:- Binary
1)首先看,就有發現logstash-input-開始就不在向後相容,且剔除了jruby-kafka,注意這裡有個坑2)會講到,版本說開始支援kafka ,又說開始
支援切不向後相容,這破壞性更新也是夠了。看來問題找到了我的kafka版本是kafka_-,kafka版本不相容導致的。
部分文件如下:
## - Update to Kafka version for bug fixes ## - Support for Kafka which is not backward compatible with broker. ## - Republish all the gems under jruby. - Update the plugin to the version of the plugin api, this change is required for Logstash compatibility. See https:///elastic/logstash/issues/5141 - Support for Kafka for LS ## - Refactor to use new Java based consumer, bypassing jruby-kafka - Breaking: Change configuration to match Kafka's configuration. This version is not backward compatible
2)之前我看文件時,看配置語法都正確,還以為是卻少依賴關係jruby-kafka library呢,這個再是在用的(另外對比發現5版本少了不少外掛。另外
kafka版本寫的是,看來這個沒有及時更新(與後面檔案不一致),誰要是看到了麻煩及時更新啊,雖是小問題但是也可能誤導我等屁民。當然也有可能是我沒
有全面看文件導致的。
文件結尾如下:
Dependencies ==================== * Apache Kafka version * jruby-kafka library
3)開始看文件,特意看了下kafka的相容性 看來logstas-input-和logstash-output-只能用了。如果你想用還想用,你的
logstash-input-kafka和logstash-output-kafka只能降級版本到了,這裡都說他是中間過渡版本了,所以還是隨大流吧。
## Kafka Compatibility Here's a table that describes the compatibility matrix for Kafka Broker support. Please remember that it is good advice to upgrade brokers before consumers/producers since brokers target backwards compatibility. The broker will work with both the consumer and consumer APIs but not the other way around. | Kafka Broker Version | Logstash Version | Input Plugin | Output Plugin | Why? | |:---------------:|:------------------:|:--------------:|:---------------:|:------| | | - | < | < | Legacy, is still popular | | | - | | | Intermediate release before that works with old Ruby Event API `[]` | | | , | | | Intermediate release before with new get/set API | | | , | | | Track latest Kafka release. Not compatible with broker |
4)現在看來只能升級kafka版本了。最後我看了下jar-dependencies發現了kafka-clients-
ls /usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/vendor/jar-dependencies/runtime-jars/ kafka-clients- log4j- lz4- slf4j-api- slf4j-log4j12- snappy-java-
5)還有一個檔案沒有看,懷著好奇心我看了一眼,發現之前都白費力氣了,這裡才是最有參考價值的的主參考文件啊,是捷徑啊,隱藏的夠深的,差點錯過了,汗!
/usr/share/logstash/vendor/bundle/jruby//gems/logstash-input-kafka-/lib/logstash/inputs/
部分文件如下:
# This input will read events from a Kafka topic. It uses the the newly designed # version of consumer API provided by Kafka to read messages from the broker. # # Here's a compatibility matrix that shows the Kafka client versions that are compatible with each combination # of Logstash and the Kafka input plugin: # # [options="header"] # |========================================================== # |Kafka Client Version |Logstash Version |Plugin Version |Security Features |Why? # | | - |< | |Legacy, is still popular # | | - | |Basic Auth, SSL |Works with the old Ruby Event API (`event['product']['price'] = 10`) # | | - | |Basic Auth, SSL |Works with the new getter/setter APIs (`('[product][price]', 10)`) # | | - | |Basic Auth, SSL |Not compatible with the broker # |========================================================== # # NOTE: We recommended that you use matching Kafka client and broker versions. During upgrades, you should # upgrade brokers before clients because brokers target backwards compatibility. For example, the broker # is compatible with both the consumer and consumer APIs, but not the other way around.
6)升級kafka_-為kafka_- (我看kafka-clients-,所以沒有用最新的kafka_-)
大概步驟
關閉老kafka
/usr/local/kafka/bin/kafka-server- /usr/local/kafka/config/
備份老配置檔案
和
刪除kafka
rm -rf /usr/local/kafka/
rm -rf /data/kafkalogs/*
安裝配置新kafka
wget /apache/kafka//kafka_-
tar zxvf kafka_- -C /usr/local/
ln -s /usr/local/kafka_- /usr/local/kafka
diff了下和變動不大可以直接使用
啟動新kafka
/usr/local/kafka/bin/kafka-server- /usr/local/kafka/config/ &
7)注意幾個關鍵配置需要修改
config :bootstrap_servers, :validate => :string, :default => "localhost:9092"
config :group_id, :validate => :string, :default => "logstash"
config :topics, :validate => :array, :default => ["logstash"]
config :consumer_threads, :validate => :number, :default => 1
除了上面的幾個關鍵配置外,kafka的topic分片資訊需要重新create一份,否則KafkaMonitor監控不出Active Topic Consumer圖形,但實際是在工作中。