1.UDF造成的compile 編譯失敗

class GetDay() extends ScalarFunction{
// 這個變數千萬不能定義在這裡，否則除錯沒問題，on yarn執行會編譯出錯
// val simpleDateFormat = new SimpleDateFormat("yyyyMMdd")
  def eval():String = {
    val simpleDateFormat = new SimpleDateFormat("yyyyMMdd")
    val calendar = Calendar.getInstance
    simpleDateFormat.
 
format(calendar.getTime)
  }
}

2.類載入順序導致的問題

Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.commons.collections.map.LinkedMap to 
field org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.pendingOffsetsToCommit of type 
org.apache.commons.collections.
 
map.LinkedMap in instance of org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer010

修改yarn-conf.yml解決

classloader.resolve-order: parent-first

3. 型別轉換cast string報錯

Caused by: org.apache.calcite.sql.validate.SqlValidatorException: Unknown datatype name 'string'
  at sun.reflect.NativeConstructorAccessorImpl.
 
newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
  at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572)

4. kafka 依賴問題

Caused by: org.apache.flink.table.api.NoMatchingTableFactoryException: Could not find a suitable table factory for 'org.apache.flink.table.factories.DeserializationSchemaFactory' in
the classpath.

Reason: No factory implements 'org.apache.flink.table.factories.DeserializationSchemaFactory'.

將依賴放在lib下。 09和10的。10依賴09. 還有base的包。
和flink-json的。

5. 寫ES報錯衝突

Caused by: [es_index_poc/A8kEJoRzQHeGpqlN_0JdOA][[es_index_poc][0]] VersionConflictEngineException[[es_type][32698248814793]: version conflict, current version [20] is different than the one provided [19]]

主鍵要唯一