supervisor無法正常執行Caused by: java.io.EOFException: null
阿新 • • 發佈：2019-01-07
告警發現一個supervisor停止，重啟啟動服務，但無法正常執行，檢視日誌報錯如下：
2015-09-27 10:21:53 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-09-27 10:21:53 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=dn11.21.abc:2181,dn12.21.abc:2181,dn13.21.abc:2181,dn14.21.abc:2181,dn15.21.abc:2181 sessionTimeout=20000 [email protected]
2015-09-27 10:21:53 o.a.z.ClientCnxn [WARN] SASL configuration failed: javax.security.auth.login.LoginException: Zookeeper client cannot authenticate using the 'Client' section of the supplied JAAS configuration: '/etc/storm/conf/storm_jaas.conf' because of a RuntimeException: java.lang.SecurityException: /etc/storm/conf/storm_jaas.conf (No such file or directory) Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2015-09-27 10:21:53 o.a.z.ClientCnxn [INFO] Opening socket connection to server dn18.21.abc/192.168.12.119:2181
2015-09-27 10:21:53 b.s.zookeeper [INFO] Zookeeper state update: :auth-failed:none
2015-09-27 10:21:53 c.n.c.ConnectionState [ERROR] Authentication failed
2015-09-27 10:21:53 o.a.z.ClientCnxn [INFO] Socket connection established to dn18.21.abc/192.168.12.119:2181, initiating session
2015-09-27 10:21:53 o.a.z.ClientCnxn [INFO] Session establishment complete on server dn18.21.abc/192.168.12.119:2181, sessionid = 0x74f284f080e3366, negotiated timeout = 20000
2015-09-27 10:21:53 c.n.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-09-27 10:21:53 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2015-09-27 10:21:53 c.n.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered.
2015-09-27 10:21:53 o.a.z.ZooKeeper [INFO] Session: 0x74f284f080e3366 closed
2015-09-27 10:21:53 o.a.z.ClientCnxn [INFO] EventThread shut down
2015-09-27 10:21:53 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-09-27 10:21:53 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=dn11.21.abc:2181,dn12.21.abc:2181,dn13.21.abc:2181,dn14.21.abc:2181,dn15.21.abc:2181/storm sessionTimeout=20000
2015-09-27 10:21:53 o.a.z.ClientCnxn [WARN] SASL configuration failed: javax.security.auth.login.LoginException: Zookeeper client cannot authenticate using the 'Client' section of the supplied JAAS configuration: '/etc/storm/conf/storm_jaas.conf' because of a RuntimeException: java.lang.SecurityException: /etc/storm/conf/storm_jaas.conf (No such file or directory) Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2015-09-27 10:21:53 o.a.z.ClientCnxn [INFO] Opening socket connection to server kafka-3/192.168.12.82:2181
2015-09-27 10:21:53 b.s.cluster [WARN] Received event :auth-failed::none: with disconnected Zookeeper.
2015-09-27 10:21:53 o.a.z.ClientCnxn [INFO] Socket connection established to kafka-3/192.168.12.82:2181, initiating session
2015-09-27 10:21:53 c.n.c.ConnectionState [ERROR] Authentication failed
2015-09-27 10:21:53 o.a.z.ClientCnxn [INFO] Session establishment complete on server kafka-3/192.168.12.82:2181, sessionid = 0x94f882b7e5d2e38, negotiated timeout = 20000
2015-09-27 10:21:53 c.n.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-09-27 10:21:53 c.n.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered.
2015-09-27 10:21:54 b.s.d.supervisor [INFO] Starting supervisor with id 59cf6aaf-72b6-4878-88d6-fbdc409f7ed4 at host dn23.21.abc
2015-09-27 10:21:54 b.s.event [ERROR] Error when processing event
java.lang.RuntimeException: java.io.EOFException
at backtype.storm.utils.Utils.deserialize(Utils.java:86) ~[storm-core-0.9.1.2.1.5.0-695.jar:0.9.1.2.1.5.0-695]
at backtype.storm.utils.LocalState.snapshot(LocalState.java:45) ~[storm-core-0.9.1.2.1.5.0-695.jar:0.9.1.2.1.5.0-695]
at backtype.storm.utils.LocalState.get(LocalState.java:56) ~[storm-core-0.9.1.2.1.5.0-695.jar:0.9.1.2.1.5.0-695]
at backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:207) ~[storm-core-0.9.1.2.1.5.0-695.jar:0.9.1.2.1.5.0-695]
at clojure.lang.AFn.applyToHelper(AFn.java:161) [clojure-1.4.0.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
at clojure.core$apply.invoke(core.clj:603) ~[clojure-1.4.0.jar:na]
at clojure.core$partial$fn__4070.doInvoke(core.clj:2343) ~[clojure-1.4.0.jar:na]
at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.4.0.jar:na]
at backtype.storm.event$event_manager$fn__2647.invoke(event.clj:39) ~[storm-core-0.9.1.2.1.5.0-695.jar:0.9.1.2.1.5.0-695]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.io.EOFException: null
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) ~[na:1.7.0_45]
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794) ~[na:1.7.0_45]
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) ~[na:1.7.0_45]
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) ~[na:1.7.0_45]
at backtype.storm.utils.Utils.deserialize(Utils.java:81) ~[storm-core-0.9.1.2.1.5.0-695.jar:0.9.1.2.1.5.0-695]
... 11 common frames omitted
2015-09-27 10:21:54 b.s.util [INFO] Halting process: ("Error when processing an event")
提示什麼null值之類的，沒有特別明確的內容，後來在網上搜索了下，解決方法：
刪除storm.yaml中配置的storm.local.dir指向的目錄中的supervisor和workers兩個目錄，再次啟動即可。
找不到的可以使用命令搜尋下：
find / -name supervisor
find / -name workers
mv supervisor supervisor.bak
mv workers supervisor.bak
rm -rf supervisor
rm -rf workers
重新命名是為了以防萬一，如果supervisor執行正常了，在刪除.bak檔案目錄
2015-09-27 10:21:53 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-09-27 10:21:53 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=dn11.21.abc:2181,dn12.21.abc:2181,dn13.21.abc:2181,dn14.21.abc:2181,dn15.21.abc:2181 sessionTimeout=20000 [email protected]
2015-09-27 10:21:53 o.a.z.ClientCnxn [WARN] SASL configuration failed: javax.security.auth.login.LoginException: Zookeeper client cannot authenticate using the 'Client' section of the supplied JAAS configuration: '/etc/storm/conf/storm_jaas.conf' because of a RuntimeException: java.lang.SecurityException: /etc/storm/conf/storm_jaas.conf (No such file or directory) Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2015-09-27 10:21:53 o.a.z.ClientCnxn [INFO] Opening socket connection to server dn18.21.abc/192.168.12.119:2181
2015-09-27 10:21:53 b.s.zookeeper [INFO] Zookeeper state update: :auth-failed:none
2015-09-27 10:21:53 c.n.c.ConnectionState [ERROR] Authentication failed
2015-09-27 10:21:53 o.a.z.ClientCnxn [INFO] Socket connection established to dn18.21.abc/192.168.12.119:2181, initiating session
2015-09-27 10:21:53 o.a.z.ClientCnxn [INFO] Session establishment complete on server dn18.21.abc/192.168.12.119:2181, sessionid = 0x74f284f080e3366, negotiated timeout = 20000
2015-09-27 10:21:53 c.n.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-09-27 10:21:53 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2015-09-27 10:21:53 c.n.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered.
2015-09-27 10:21:53 o.a.z.ZooKeeper [INFO] Session: 0x74f284f080e3366 closed
2015-09-27 10:21:53 o.a.z.ClientCnxn [INFO] EventThread shut down
2015-09-27 10:21:53 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-09-27 10:21:53 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=dn11.21.abc:2181,dn12.21.abc:2181,dn13.21.abc:2181,dn14.21.abc:2181,dn15.21.abc:2181/storm sessionTimeout=20000
2015-09-27 10:21:53 o.a.z.ClientCnxn [WARN] SASL configuration failed: javax.security.auth.login.LoginException: Zookeeper client cannot authenticate using the 'Client' section of the supplied JAAS configuration: '/etc/storm/conf/storm_jaas.conf' because of a RuntimeException: java.lang.SecurityException: /etc/storm/conf/storm_jaas.conf (No such file or directory) Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2015-09-27 10:21:53 o.a.z.ClientCnxn [INFO] Opening socket connection to server kafka-3/192.168.12.82:2181
2015-09-27 10:21:53 b.s.cluster [WARN] Received event :auth-failed::none: with disconnected Zookeeper.
2015-09-27 10:21:53 o.a.z.ClientCnxn [INFO] Socket connection established to kafka-3/192.168.12.82:2181, initiating session
2015-09-27 10:21:53 c.n.c.ConnectionState [ERROR] Authentication failed
2015-09-27 10:21:53 o.a.z.ClientCnxn [INFO] Session establishment complete on server kafka-3/192.168.12.82:2181, sessionid = 0x94f882b7e5d2e38, negotiated timeout = 20000
2015-09-27 10:21:53 c.n.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-09-27 10:21:53 c.n.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered.
2015-09-27 10:21:54 b.s.d.supervisor [INFO] Starting supervisor with id 59cf6aaf-72b6-4878-88d6-fbdc409f7ed4 at host dn23.21.abc
2015-09-27 10:21:54 b.s.event [ERROR] Error when processing event
java.lang.RuntimeException: java.io.EOFException
at backtype.storm.utils.Utils.deserialize(Utils.java:86) ~[storm-core-0.9.1.2.1.5.0-695.jar:0.9.1.2.1.5.0-695]
at backtype.storm.utils.LocalState.snapshot(LocalState.java:45) ~[storm-core-0.9.1.2.1.5.0-695.jar:0.9.1.2.1.5.0-695]
at backtype.storm.utils.LocalState.get(LocalState.java:56) ~[storm-core-0.9.1.2.1.5.0-695.jar:0.9.1.2.1.5.0-695]
at backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:207) ~[storm-core-0.9.1.2.1.5.0-695.jar:0.9.1.2.1.5.0-695]
at clojure.lang.AFn.applyToHelper(AFn.java:161) [clojure-1.4.0.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
at clojure.core$apply.invoke(core.clj:603) ~[clojure-1.4.0.jar:na]
at clojure.core$partial$fn__4070.doInvoke(core.clj:2343) ~[clojure-1.4.0.jar:na]
at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.4.0.jar:na]
at backtype.storm.event$event_manager$fn__2647.invoke(event.clj:39) ~[storm-core-0.9.1.2.1.5.0-695.jar:0.9.1.2.1.5.0-695]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.io.EOFException: null
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) ~[na:1.7.0_45]
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794) ~[na:1.7.0_45]
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) ~[na:1.7.0_45]
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) ~[na:1.7.0_45]
at backtype.storm.utils.Utils.deserialize(Utils.java:81) ~[storm-core-0.9.1.2.1.5.0-695.jar:0.9.1.2.1.5.0-695]
... 11 common frames omitted
2015-09-27 10:21:54 b.s.util [INFO] Halting process: ("Error when processing an event")
提示什麼null值之類的，沒有特別明確的內容，後來在網上搜索了下，解決方法：
刪除storm.yaml中配置的storm.local.dir指向的目錄中的supervisor和workers兩個目錄，再次啟動即可。
找不到的可以使用命令搜尋下：
find / -name supervisor
find / -name workers
mv supervisor supervisor.bak
mv workers supervisor.bak
rm -rf supervisor
rm -rf workers
重新命名是為了以防萬一，如果supervisor執行正常了，在刪除.bak檔案目錄