1. 程式人生 > >本地搭建hadoop叢集--kylin的安裝(叢集部署)

本地搭建hadoop叢集--kylin的安裝(叢集部署)

kylin官方安裝文件

http://kylin.apache.org/cn/docs/install/configuration.html

總結:

1、關於麒麟高可用涉及負載均衡官網都說超出範圍了,安裝只考慮安裝的事情,其餘的先忽略

2、kylin在啟動的時候會自動檢測你本地hadoop,hbase,hive,spark,kafka 的環境變數。配置路徑

3、逐一安裝不像CDH或者HDP。所有元件自動整合,有的地方會單獨踩坑

4、麒麟web-ui 訪問路徑

http://host:7070/kylin

admin/KYLIN

一、安裝步驟

從官網下載

apache-kylin-2.4.0-bin-hbase1x.tar.gz

前提是你已經成功安裝了 zookeeper,hadoop,hive,hbase

-hbase1x 指的是相容hbase 1.X 所有版本

tar -zxvf apache-kylin-2.4.0-bin-hbase1x.tar.gz -C /opt/module

cd apach-kylin-2.4.0 進入解壓的目錄

cd conf  進入配置目錄

vim kylin.properties

末尾新增如下每個節點都要寫進去

kylin.server.mode=all
kylin.rest.servers=host1:7070,host2:7070

:wq 儲存

分發到各個節點 

切記:mode 模式分為3種,第一種是job,第二種是all,第三種是query

一個例項中最多隻能允許一個 job

我在主節點 寫的 all

其餘slave 節點寫的都是 query

all顧名思義既有job,又有query

 

進入目錄 bin

kylin.sh start

啟動成功

二、打地鼠模式

報錯

2018-08-24 17:11:30,471 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:java.library.path=/opt/module/hadoop-2.7.6/lib/native
2018-08-24 17:11:30,472 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:java.io.tmpdir=/tmp
2018-08-24 17:11:30,473 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:java.compiler=<NA>
2018-08-24 17:11:30,473 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:os.name=Linux
2018-08-24 17:11:30,473 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:os.arch=amd64
2018-08-24 17:11:30,474 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:os.version=2.6.32-642.el6.x86_64
2018-08-24 17:11:30,474 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:user.name=root
2018-08-24 17:11:30,474 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:user.home=/root
2018-08-24 17:11:30,474 INFO  [main] zookeeper.ZooKeeper:100 : Client environment:user.dir=/opt/module/apache-kylin-2.3.1-bin/bin
2018-08-24 17:11:30,477 INFO  [main] zookeeper.ZooKeeper:438 : Initiating client connection, connectString=master:2181,slave1:2181,slave2:2181,slave3:2181 sessionTimeout=90000 watcher=hconnection-0x1ee47300x0, quorum=master:2181,slave1:2181,slave2:2181,slave3:2181, baseZNode=/hbase
2018-08-24 17:11:30,512 INFO  [main-SendThread(slave3:2181)] zookeeper.ClientCnxn:975 : Opening socket connection to server slave3/192.168.77.103:2181. Will not attempt to authenticate using SASL (unknown error)
2018-08-24 17:11:30,522 INFO  [main-SendThread(slave3:2181)] zookeeper.ClientCnxn:852 : Socket connection established to slave3/192.168.77.103:2181, initiating session
2018-08-24 17:11:30,548 INFO  [main-SendThread(slave3:2181)] zookeeper.ClientCnxn:1235 : Session establishment complete on server slave3/192.168.77.103:2181, sessionid = 0x400002542ed0004, negotiated timeout = 40000
2018-08-24 17:11:31,022 DEBUG [main] hbase.HBaseConnection:181 : Using the working dir FS for HBase: hdfs://master:9000
2018-08-24 17:11:31,061 INFO  [main] imps.CuratorFrameworkImpl:224 : Starting
2018-08-24 17:11:31,063 INFO  [main] zookeeper.ZooKeeper:438 : Initiating client connection, connectString=master:2181:2181,slave1:2181:2181,slave2:2181:2181,slave3:2181:2181

sessionTimeout=120000 [email protected]
2018-08-24 17:11:41,074 ERROR [main] imps.CuratorFrameworkImpl:546 : Background exception was not retry-able or retry gave up
java.net.UnknownHostException: master:2181: 域名解析暫時失敗
    at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
    at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
    at java.net.InetAddress.getAllByName(InetAddress.java:1192)
    at java.net.InetAddress.getAllByName(InetAddress.java:1126)
    at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
    at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
    at org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
    at org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:146)
    at org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
    at org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55)
    at org.apache.curator.ConnectionState.reset(ConnectionState.java:218)
    at org.apache.curator.ConnectionState.start(ConnectionState.java:102)
    at org.apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.java:189)
    at org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:248)
    at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock$Factory.getZKClient(ZookeeperDistributedLock.java:81)
    at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock$Factory.<init>(ZookeeperDistributedLock.java:105)
    at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock$Factory.<init>(ZookeeperDistributedLock.java:101)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.lang.Class.newInstance(Class.java:442)
    at org.apache.kylin.common.util.ClassUtil.newInstance(ClassUtil.java:84)
    at org.apache.kylin.common.KylinConfigBase.getDistributedLockFactory(KylinConfigBase.java:354)
    at org.apache.kylin.storage.hbase.HBaseConnection.createHTableIfNeeded(HBaseConnection.java:328)
    at org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:111)
    at org.apache.kylin.storage.hbase.HBaseResourceStore.<init>(HBaseResourceStore.java:92)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:90)
    at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:108)
    at org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:94)
    at org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41)
2018-08-24 17:11:41,077 DEBUG [main] util.ZookeeperDistributedLock:143 : [email protected] trying to lock /kylin/kylin_metadata/create_htable/kylin_metadata/lock
2018-08-24 17:11:56,112 ERROR [main] curator.ConnectionState:200 : Connection timed out for connection string (master:2181:2181,slave1:2181:2181,slave2:2181:2181,slave3:2181:2181) and timeout (15000) / elapsed (25046)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
解決方案:

master:2181:2181

重複了

去hbase的conf目錄下 找到 hbase-site.xml

把 master:2181,slave1:2181,slave2:2181 後的 2181 全部去掉,只留主機名稱如下:

<property>

             <name>hbase.zookeeper.property.clientPort</name>

             <value>2181</value>

</property>

<property>

            <name>hbase.zookeeper.quorum</name>

             <value>master,slave1,slave2,slave3</value>

</property>

儲存退出分發檔案

重啟hbase

再重啟麒麟