Kafka：ZK+Kafka+Spark Streaming集群環境搭建（二十三）Structured Streaming遇到問題：Set(TopicName-0) are gone. Some data may have been missed
阿新 • • 發佈：2018-08-24
ack loss set div top 過程 pan check use
事情經過：之前該topic(M_A)已經存在，而且正常消費了一段時間，後來刪除了topic(M_A)，重新創建了topic（M-B），程序使用新創建的topic（M-B）進行實時統計操作，執行過程中拋出了一下異常：
18/08/24 10:20:42 INFO utils.AppInfoParser: Kafka version : 0.10.0-kafka-2.1.0 18/08/24 10:20:42 INFO utils.AppInfoParser: Kafka commitId : unknown 18/08/24 10:20:42 INFO internals.AbstractCoordinator: Discovered coordinator vmxx.xx.xx.xx.com.cn:9092(id: 2147483417 rack: null) for group spark-kafka-source-165bc430-5cbc-4cfc-8327-9af01fd02fcc-616947503-driver-0. 18/08/24 10:20:42 INFO internals.ConsumerCoordinator: Revoking previously assigned partitions [] for group spark-kafka-source-165bc430-5cbc-4cfc-8327-9af01fd02fcc-616947503-driver-0 18/08/24 10:20:42 INFO internals.AbstractCoordinator: (Re-)joining group spark-kafka-source-165bc430-5cbc-4cfc-8327-9af01fd02fcc-616947503-driver-0 18/08/24 10:20:45 INFO internals.AbstractCoordinator: Successfully joined group spark-kafka-source-165bc430-5cbc-4cfc-8327-9af01fd02fcc-616947503-driver-0 with generation 1 18/08/24 10:20:45 INFO internals.ConsumerCoordinator: Setting newly assigned partitions [M-B-0] for group spark-kafka-source-165bc430-5cbc-4cfc-8327-9af01fd02fcc-616947503-driver-0 18/08/24 10:20:46 WARN kafka010.KafkaSource: Set(M_A-0) are gone. Some data may have been missed. Some data may have been lost because they are not available in Kafka any more; either the data was aged out by Kafka or the topic may have been deleted before all the data in the topic was processed. If you want your streaming query to fail on such cases, set the source option "failOnDataLoss" to "true".
錯誤原因，在structured streaming編程時，使用checkpoint，此時忘記了刪除checkpoint,因此導致操作。
Kafka：ZK+Kafka+Spark Streaming集群環境搭建（二十三）Structured Streaming遇到問題：Set(TopicName-0) are gone. Some data may have been missed