1. 程式人生 > >hdfs遠端連線hadoop問題與解決

hdfs遠端連線hadoop問題與解決

問題:使用hdfs客戶端在本地連線阿里雲伺服器上部署的hadoop,操作hdfs時候出現異常:could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. 並且,在管理網頁上檢視檔案檔案大小全部為0;
原因:在百度找了很久都沒有找答案都是說dataNode沒啟動,可是伺服器上jps檢視dataNode已經正常啟動了,百思不得其解,後來在stackoverflow找到一段話: It take me a week to figure out the problem in my situation. When the client(your program) ask the nameNode for data operation, the nameNode picks up a dataNode and navigate the client to it, by giving the dataNode's ip to the client. But, when the dataNode host is configured to has multiple ip, and the nameNode gives you the one your client CAN'T ACCESS TO, the client would add the dataNode to exclude list and ask the nameNode for a new one, and finally all dataNode are excluded, you get this error. So check node's ip settings before you try everything!!! 大致的意思是:客戶端操作hdfs時候先連線namenode,然後namenode分配給客戶端一個datanoe的ip地址,如果這個ip地址客戶端無法訪問到,就會被客戶端新增到排除列表中。 而我的阿里雲伺服器是多ip地址,所以分配給我了一個不可達到的地址;從而出現了這個問題; 解決:
將客戶端程式放在hadoop伺服器上執行,該錯誤就會消失。在hadoop監管頁面檢視dataNode的節點是 iZ234nvolhdZ (10.253.102.93:50010),我本地10.253.102.93地址可以ping通,而telnet埠號50010不通,說明是埠號的問題把50010埠號開通後本地就可以訪問了。 參考: https://stackoverflow.com/questions/5293446/hdfs-error-could-only-be-replicated-to-0-nodes-instead-of-1 http://blog.csdn.net/cht0112/article/details/72911307?utm_source=itdadao&utm_medium=referral


問題:ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: iZ234nvolhdZ:50010:DataXceiver error processing WRITE_BLOCK operation src: /10.253.102.93:59704 dst: /10.253.102.93:50010 java.io.IOException: Premature EOF from inputStream 原因:客戶端上傳檔案流操作後忘記關掉 解決:輸出流關閉和重新整理 outputStream.flush(); outputStream.close(); 參考:
http://blog.csdn.net/menghuannvxia/article/details/44591619