Spark筆記(一):錯誤總結
1.轉義字符:
常見的replaceAll,split,mkstring中涉及到特殊字符的都要加上轉義字符,比如str.split("\\|"),str.replaceAll("\\[", "")
報錯信息:
java.util.regex.PatternSyntaxException: Unclosed character class near index 0
java.util.regex.PatternSyntaxException: Unexpected internal error near index 1
2.kafka中數據還沒來得及消費,數據就已經丟失或者過期了:
[https://blog.csdn.net/yxgxy270187133/article/details/53666760]
報錯信息:
org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions:{...}
3.spark submit 內存參數太小 --executor-memory 8G \ --driver-memory 8G \
報錯信息:
Application application_1547156777102_0243 failed 2 times due to AM Container for appattempt_1547156777102_0243_000002 exited with exitCode: -104
For more detailed output, check the application tracking page:https://host-10-31-4-246:26001/cluster/app/application_1547156777102_0243 Then click on links to logs of each attempt.
Diagnostics: Container [pid=5064,containerID=container_e62_1547156777102_0243_02_000001] is running beyond physical memory limits. Current usage: 4.6 GB of 4.5 GB physical memory used; 6.3 GB of 22.5 GB virtual memory used. Killing container.
Spark筆記(一):錯誤總結