Phoenix的安裝使用與SQL查詢HBase
阿新 • • 發佈：2018-08-26
saving home sum rip sna ima 事務性 timeout exists
一. Phoenix的簡介
1. 什麽是phoenix
現有hbase的查詢工具有很多如：Hive，Tez，Impala，Shark/Spark，Phoenix等。今天主要說Phoenix。phoenix是一個在hbase上面實現的基於hadoop的OLTP技術，具有低延遲、事務性、可使用sql、提供jdbc接口的特點。 而且phoenix還提供了hbase二級索引的解決方案，豐富了hbase查詢的多樣性，繼承了hbase海量數據快速隨機查詢的特點。但是在生產環境中，不可以用在OLTP中。在線事務處理的環境中，需要低延遲，而Phoenix在查詢HBase時，雖然做了一些優化，但延遲還是不小。所以依然是用在OLAT中，再將結果返回存儲下來。
Phoenix完全使用Java編寫，作為HBase內嵌的JDBC驅動。Phoenix查詢引擎會將SQL查詢轉換為一個或多個HBase掃描，並編排執行以生成標準的JDBC結果集。直接使用HBase API、協同處理器與自定義過濾器，對於簡單查詢來說，其性能量級是毫秒，對於百萬級別的行數來說，其性能量級是秒。
2. phoenix的架構
3. phoenix的下載
選擇對應的hbase版本進行下載，我這裏使用的是hbase-1.2.6版本，下載地址：http://mirror.bit.edu.cn/apache/phoenix/
[admin@node21 ~]$ cd /opt/software/
[admin@node21 software]$ wget http://mirror.bit.edu.cn/apache/phoenix/apache-phoenix-4.14.0-HBase-1.2/bin/apache-phoenix-4.14.0-HBase-1.2-bin.tar.gz
[admin@node21 software]$ ll
-rw-rw-r-- 1 admin admin 346171547 Jun 5 06:16 apache-phoenix-4.14.0-HBase-1.2-bin.tar.gz
4. 集群部署
| 節點IP | 節點名稱 | HBase | Phoniex | Zookeeper | Hadoop |
| 192.168.100.21 | node21 | √ | √ | √ | √ |
| 192.168.100.22 | node22 | √ | √ | √ | |
| 192.168.100.23 | node23 | √ | √ | √ |
Hadoop集群搭建參考： CentOS7.5搭建Hadoop2.7.6完全分布式集群
zookeeper集群搭建參考：CentOS7.5搭建Zookeeper3.4.12集群
HBase集群搭建參考： CentOS7.5搭建HBase1.2.6HA集群
二. Phoenix服務端安裝
官網安裝文檔參考：http://phoenix.apache.org/installation.html
1. 解壓縮安裝包
[admin@node21 software]$ tar zxvf apache-phoenix-4.14.0-HBase-1.2-bin.tar.gz -C /opt/module/
[admin@node21 software]$ cd /opt/module/
[admin@node21 module]$ mv apache-phoenix-4.14.0-HBase-1.2-bin phoenix-4.14.0-HBase-1.2
[admin@node21 module]$ ll
drwxr-xr-x 5 admin admin 4096 Jun 5 05:41 phoenix-4.14.0-HBase-1.2
2. 拷貝jar包到hbase/lib包
進入到phoenix的安裝目錄把phoenix-4.12.0-HBase-1.2-server.jar 拷貝到集群中每個節點( 主節點也要拷貝 )的 hbase 的 lib 目錄下
[admin@node21 module]$ cd phoenix-4.14.0-HBase-1.2/
[admin@node21 phoenix-4.14.0-HBase-1.2]$ cp phoenix-4.14.0-HBase-1.2-server.jar /opt/module/hbase-1.2.6/lib/
[admin@node21 phoenix-4.14.0-HBase-1.2]$ scp phoenix-4.14.0-HBase-1.2-server.jar admin@node22:/opt/module/hbase-1.2.6/lib/
[admin@node21 phoenix-4.14.0-HBase-1.2]$ scp phoenix-4.14.0-HBase-1.2-server.jar admin@node23:/opt/module/hbase-1.2.6/lib/
3. 配置phoenix環境變量
[admin@node21 phoenix-4.14.0-HBase-1.2]$ sudo vi /etc/profile export PHOENIX_HOME=/opt/module/phoenix-4.14.0-HBase-1.2 export PATH=$PATH:$PHOENIX_HOME/bin [admin@node21 phoenix-4.14.0-HBase-1.2]$ source /etc/profile
4. 重啟Hbase
$ zkServer.sh start
[admin@node21 ~]$ start-dfs.sh
[admin@node22 ~]$ start-yarn.sh
[admin@node21 ~]$ start-hbase.sh
5. 啟動phoenix
啟動命令：phoenix-4.14.0-HBase-1.2/bin/sqlline.py zookeeper
[admin@node21 phoenix-4.14.0-HBase-1.2]$ ./bin/sqlline.py 192.168.100.21,192.168.100.22,192.168.100.23:2181
Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix:192.168.100.21,192.168.100.22,192.168.100.23:2181 none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:192.168.100.21,192.168.100.22,192.168.100.23:2181
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/phoenix-4.14.0-HBase-1.2/phoenix-4.14.0-HBase-1.2-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/hadoop-2.7.6/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
18/08/21 02:28:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Connected to: Phoenix (version 4.14)
Driver: PhoenixEmbeddedDriver (version 4.14)
Autocommit status: true
Transaction isolation: TRANSACTION_READ_COMMITTED
Building list of tables and columns for tab-completion (set fastconnect to true to skip)...
133/133 (100%) Done
Done
sqlline version 1.2.0
0: jdbc:phoenix:192.168.100.21,192.168.100.22>
6. 驗證phoenix
1）查看所有表
輸入 !tables ，查看都有哪些表。以下顯示均為Phoenix系統表，系統表中維護了用戶表的元數據信息。
2）退出Phoenix
輸入 !exit 命令(PS：Phoenix早期版本如(2.11版本)需輸入!quilt才可退出，目前高版本已改為!exit命令)
三. Phoenix基本shell命令
提示：以下，可能有部分命令在Phoenix更高版本中已失效，改為其他命令代替，請註意。
0: jdbc:phoenix:192.168.100.21,192.168.100.22> help
!all Execute the specified SQL against all the current
connections
!autocommit Set autocommit mode on or off
!batch Start or execute a batch of statements
!brief Set verbose mode off
!call Execute a callable statement
!close Close the current connection to the database
!closeall Close all current open connections
!columns List all the columns for the specified table
!commit Commit the current transaction (if autocommit is off)
!connect Open a new connection to the database.
!dbinfo Give metadata information about the database
!describe Describe a table
!dropall Drop all tables in the current database
!exportedkeys List all the exported keys for the specified table
!go Select the current connection
!help Print a summary of command usage
!history Display the command history
!importedkeys List all the imported keys for the specified table
!indexes List all the indexes for the specified table
!isolation Set the transaction isolation for this connection
!list List the current connections
!manual Display the SQLLine manual
!metadata Obtain metadata information
!nativesql Show the native SQL for the specified statement
!outputformat Set the output format for displaying results
(table,vertical,csv,tsv,xmlattrs,xmlelements)
!primarykeys List all the primary keys for the specified table
!procedures List all the procedures
!properties Connect to the database specified in the properties file(s)
!quit Exits the program
!reconnect Reconnect to the database
!record Record all output to the specified file
!rehash Fetch table and column names for command completion
!rollback Roll back the current transaction (if autocommit is off)
!run Run a script from the specified file
!save Save the current variabes and aliases
!scan Scan for installed JDBC drivers
!script Start saving a script to a file
!set Set a sqlline variable
Variable Value Description
=============== ========== ================================
autoCommit true/false Enable/disable automatic
transaction commit
autoSave true/false Automatically save preferences
color true/false Control whether color is used
for display
fastConnect true/false Skip building table/column list
for tab-completion
force true/false Continue running script even
after errors
headerInterval integer The interval between which
headers are displayed
historyFile path File in which to save command
history. Default is
$HOME/.sqlline/history (UNIX,
Linux, Mac OS),
$HOME/sqlline/history (Windows)
incremental true/false Do not receive all rows from
server before printing the first
row. Uses fewer resources,
especially for long-running
queries, but column widths may
be incorrect.
isolation LEVEL Set transaction isolation level
maxColumnWidth integer The maximum width to use when
displaying columns
maxHeight integer The maximum height of the
terminal
maxWidth integer The maximum width of the
terminal
numberFormat pattern Format numbers using
DecimalFormat pattern
outputFormat table/vertical/csv/tsv Format mode for
result display
propertiesFile path File from which SqlLine reads
properties on startup; default is
$HOME/.sqlline/sqlline.properties
(UNIX, Linux, Mac OS),
$HOME/sqlline/sqlline.properties
(Windows)
rowLimit integer Maximum number of rows returned
from a query; zero means no
limit
showElapsedTime true/false Display execution time when
verbose
showHeader true/false Show column names in query
results
showNestedErrs true/false Display nested errors
showWarnings true/false Display connection warnings
silent true/false Be more silent
timeout integer Query timeout in seconds; less
than zero means no timeout
trimScripts true/false Remove trailing spaces from
lines read from script files
verbose true/false Show verbose error messages and
debug info
!sql Execute a SQL command
!tables List all the tables in the database
!typeinfo Display the type map for the current connection
!verbose Set verbose mode on
Comments, bug reports, and patches go to ???
四. SQuirrel客戶端安裝
如果您更願意使用客戶端GUI與Phoenix進行交互，請下載並安裝SQuirrel。由於Phoenix是一個JDBC驅動程序，因此與此類工具的集成是無縫的。通過SQuirrel，您可以在SQL選項卡中發出SQL語句（創建表，插入數據，運行查詢），並在“對象”選項卡中檢查表元數據（即列表，列，主鍵和類型）。
1. 下載SQuirrel
官網下載地址：http://squirrel-sql.sourceforge.net/
下載完成：
2. 安裝Squirrel
進入到squirrel-sql-3.8.1-standard.jar所在目錄，cmd，執行如下命令：
java -jar squirrel-sql-3.8.1-standard.jar
然後，除了選擇安裝路徑，其它的可按需選擇，否則一路“next”即可。
3. 配置Squirrel客戶端
（1）將服務器上Phoenix安裝目錄下的phoenix-4.14.0-HBase-1.2-client.jar文件，下載到Windows下squirrel-sql安裝目錄的lib文件夾下，並點擊squirrel-sql.bat啟動。
（2）添加新的驅動(Drivers -> New Driver)，如下圖所示：
（3）在彈出的“Add Driver”窗口中，設置Driver的“Name”為：Phoenix，“Phoenix”為：jdbc:phoenix:192.168.100.21，其中，192.168.100.21為你Phoenix服務器的地址；“ClassName”為：org.apache.phoenix.jdbc.PhoenixDriver，最終結果如下圖：
設置完成後，點擊OK關閉。
（4）點擊“Drivers”邊上的“Alias”選項卡，和第（2）步的新建驅動一樣新建Alias (Aliases -> New Aliases)，在彈出的窗口中，填寫Name，UserName，Password，這三項可任意填寫，在Driver選項的下拉列表中查找我們剛添加的Phoenix驅動，然後點擊“Test”按鈕測試連接，然後點擊“Connect”按鈕，彈出“Connection successful”表示配置成功。結果如下圖：
點擊OK按鈕，關閉彈出窗口。
（5）雙擊新建的test Alias ，然後點擊“Connect”按鈕，打開SQL命令行，在“SQL”選項卡下，即可編寫sql命令，新建一張person表進行測試
create table person (id integer not null primary key,name varchar,age integer)
插入一條數據：
upsert into person values (1,‘zhangsan‘ ,18)
查詢數據：
select * from person
查詢結果如下圖所示：
（6）到Hbase Shell下去查看我們剛剛添加的數據
可以看到我們新建的person表已經添加進去了，至此squirrel也安裝並配置成功了！！
註意：通過squirrel添加的表，會自動轉換為大寫，由於Phoenix是區分大小寫的，所以在Hbase Shell下，表名都要用大寫，如：desc ‘PERSON‘，如果用desc ‘person‘是會報錯的。
五. Phoenix的使用
1. 命令行方式
從命令行執行SQL的終端接口現在與Phoenix捆綁在一起。要啟動它，請從bin目錄執行以下命令：
[admin@node21 phoenix-4.14.0-HBase-1.2]$ ./bin/sqlline.py 192.168.100.21,192.168.100.22,192.168.100.23:2181
1）建表插入數據：這裏演示官方案例，安裝包目錄下examples包下STOCK_SYMBOL.sql 內容如下：
CREATE TABLE IF NOT EXISTS STOCK_SYMBOL (SYMBOL VARCHAR NOT NULL PRIMARY KEY, COMPANY VARCHAR);
UPSERT INTO STOCK_SYMBOL VALUES (‘CRM‘,‘SalesForce.com‘);
SELECT * FROM STOCK_SYMBOL;
要從命令行執行SQL腳本，可以包含如下所示的SQL文件參數：
[admin@node21 phoenix-4.14.0-HBase-1.2]$ sqlline.py 192.168.100.21,192.168.100.22,192.168.100.23:2181 ./examples/STOCK_SYMBOL.sql
Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
Setting property: [run, ./examples/STOCK_SYMBOL.sql]
issuing: !connect jdbc:phoenix:192.168.100.21,192.168.100.22,192.168.100.23:2181 none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:192.168.100.21,192.168.100.22,192.168.100.23:2181
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/phoenix-4.14.0-HBase-1.2/phoenix-4.14.0-HBase-1.2-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/hadoop-2.7.6/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
18/08/21 07:53:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Connected to: Phoenix (version 4.14)
Driver: PhoenixEmbeddedDriver (version 4.14)
Autocommit status: true
Transaction isolation: TRANSACTION_READ_COMMITTED
Building list of tables and columns for tab-completion (set fastconnect to true to skip)...
136/136 (100%) Done
Done
1/4 CREATE TABLE IF NOT EXISTS STOCK_SYMBOL (SYMBOL VARCHAR NOT NULL PRIMARY KEY, COMPANY VARCHAR);
No rows affected (3.818 seconds)
2/4 UPSERT INTO STOCK_SYMBOL VALUES (‘CRM‘,‘SalesForce.com‘);
1 row affected (0.166 seconds)
3/4 SELECT * FROM STOCK_SYMBOL;
+---------+-----------------+
| SYMBOL | COMPANY |
+---------+-----------------+
| CRM | SalesForce.com |
+---------+-----------------+
1 row selected (0.184 seconds)
4/4
Closing: org.apache.phoenix.jdbc.PhoenixConnection
sqlline version 1.2.0
examples/STOCK_SYMBOL.csv
AAPL,APPLE Inc.
CRM,SALESFORCE
GOOG,Google
HOG,Harlet-Davidson Inc.
HPQ,Hewlett Packard
INTC,Intel
MSFT,Microsoft
WAG,Walgreens
WMT,Walmart
2）導入數據：此外，您可以使用bin / psql.py加載CSV數據或執行SQL腳本。例如：
[admin@node21 phoenix-4.14.0-HBase-1.2]$ ./bin/psql.py -t STOCK_SYMBOL 192.168.100.21,192.168.100.22,192.168.100.23:2181 ./examples/STOCK_SYMBOL.csv
PS：其中 -t 後面是表名， ../examples/STOCK_SYMBOL.csv是csv數據（註意數據的分隔符需要是逗號）。
3）查詢數據
[admin@node21 phoenix-4.14.0-HBase-1.2]$ ./bin/sqlline.py 192.168.100.21,192.168.100.22,192.168.100.23:2181 0: jdbc:phoenix:192.168.100.21,192.168.100.22> select * from STOCK_SYMBOL;
查詢總條數
2. 客戶端方式
SQuirrel是用來連接Phoenix的客戶端，前面已做介紹。
六. Phoenix的api操作HBase
1. 參考資料鏈接
常見問題參考：http://phoenix.apache.org/faq.html#What_is_the_Phoenix_JDBC_URL_syntax
幫助文檔參考：https://help.aliyun.com/document_detail/69105.html?spm=a2c4g.11186623.6.591.420a48cbSK2wy8
2. 開發環境準備
IDEA、jdk1.8、window10、hadoop2.7.6、hbase1.2.6、phoenix4.14.0
1、從集群拷貝以下文件：core-site.xml、hbase-site.xml、hdfs-site.xml文件放到工程src下
2、把phoenix的phoenix-4.14.0-client.jar和phoenix-core-4.14.0.jar添加到工程classpath
3、配置集群中各節點的hosts文件，把客戶端的hostname：IP添加進去
4、在客戶端host文件中加入集群的hostname和IP
3. 例一
package com.xyg.phoenix;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
/**
* Author: Mr.Deng
* Date: 2018/8/25
* Desc: create table, create index, insert data, select table.
*/
public class TestPhoenixJDBC {
private static String driver = "org.apache.phoenix.jdbc.PhoenixDriver";
public static void main(String[] args) throws SQLException {
Connection conn = null;
Statement stmt = null;
ResultSet rs = null;
try {
Class.forName(driver);
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
conn = DriverManager.getConnection("jdbc:phoenix:node21,node22,node23:2181");
stmt = conn.createStatement();
stmt.execute("drop table if exists test");
stmt.execute("create table test (mykey integer not null primary key, mycolumn varchar)");
stmt.execute("create index test_idx on test(mycolumn)");
stmt.executeUpdate("upsert into test values (1,‘World!‘)");
stmt.executeUpdate("upsert into test values (2,‘Hello‘)");
stmt.executeUpdate("upsert into test values (3,‘World!‘)");
conn.commit();
rs = stmt.executeQuery("select mykey from test where mycolumn=‘Hello‘");
while (rs.next()) {
System.out.println(rs.getInt(1));
}
stmt.close();
rs.close();
conn.close();
}
}
Phoenix的安裝使用與SQL查詢HBase