PostgreSQL on Azure.cn : 性能測試及調優
上一篇我們知道怎麽一步步的安裝並部署PostgreSQL，接下來我們就要測試一下在Azure上PostgreSQL可以達成什麽樣的性能，並且嘗試修改數據庫的參數，看看怎麽優化數據庫性能。
對於數據庫新手來說，想要做數據庫的性能測試遇到的第一個問題就是：我可以用什麽測試工具？萬幸的是安裝了PostgreSQL後，我們可以使用自帶的pgbench來進行性能測試。
測試所使用的虛擬機配置如下：
Standard DS4 v2 (8 vcpus, 28 GB memory), 4 X 1023G, 4 X 5000 IOPS, Raid0, 記做 VM-SSD
Standard DS4 v2 (8 vcpus, 28 GB memory), 16 X 128G, 16 X 500 IOPS, Raid0, 記做 VM-HDD
兩臺虛機放在同一個VNET，有對外發布的公網地址。
通過分別測試這兩臺虛擬機，我們來看看在各種情況下PostgreSQL的性能。
首先建立測試數據庫，並創建5000萬條的記錄：
創建測試數據庫
$ createdb pgbench
初始化測試數據庫，會創建4張表
$ pgbench -i pgbench
插入5000萬條記錄
$ pgbench -i -s 500 pgbench
第一步測試：PostgreSQL安裝部署完成，不作任何優化
VM-SSD：
$ pgbench -r -c 16 -j 16 -n -T 180 pgbench
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 500
query mode: simple
number of clients: 16
number of threads: 16
duration: 180 s
number of transactions actually processed: 333504
latency average = 8.636 ms
tps = 1852.672083 (including connections establishing)
tps = 1852.742252 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
0.002 \set aid random(1, 100000 * :scale)
0.001 \set bid random(1, 1 * :scale)
0.001 \set tid random(1, 10 * :scale)
0.000 \set delta random(-5000, 5000)
0.071 BEGIN;
0.248 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE a id = :aid;
0.150 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
0.171 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
0.265 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
0.117 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
7.607 END;
VM-HDD：
$ pgbench -r -c 16 -j 16 -n -T 180 pgbench
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 500
query mode: simple
number of clients: 16
number of threads: 16
duration: 180 s
number of transactions actually processed: 223638
latency average = 12.879 ms
tps = 1242.328936 (including connections establishing)
tps = 1242.365821 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
0.002 \set aid random(1, 100000 * :scale)
0.001 \set bid random(1, 1 * :scale)
0.001 \set tid random(1, 10 * :scale)
0.000 \set delta random(-5000, 5000)
0.070 BEGIN;
0.250 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
0.150 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
0.174 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
0.331 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
0.116 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
11.679 END;
可以看到使用SSD的虛機比使用HDD的虛機下性能好了接近50%，時延也差了近50%，差距還是很明顯的。
第二步測試：優化PostgreSQL參數
修改數據庫目錄下的postgresql.conf文件
max_connections = 2000
shared_buffers = 4GB
temp_buffers = 2GB
max_prepared_transactions = 2000
work_mem = 512MB
maintenance_work_mem = 2GB
我們來挨個解釋下這幾個參數的含義：
max_connections : 決定和數據庫連接的並發連接數目的最大值。 缺省通常是 100，但是如果你的內核設置不支持這麽大（在 initdb 的時候判斷），可能會比這個數少。這個參數的上限收到操作系統並發連接數的限制。修改本參數需要重啟PostgreSQL服務。
shared_buffers：這是最重要的參數，postgresql對數據操作時都要先將數據從磁盤讀取到內存中，然後進行更新，最後再將數據寫回磁盤。因此應該盡量大，讓更多的數據緩存在shared_buffers中。按照文檔建議設置範圍一般在實際內存的25%~40%之間。修改本參數需要重啟PostgreSQL服務。
temp_buffers：臨時緩沖區，用於數據庫會話訪問臨時表數據，系統默認值為8M。可以在單獨的session中對該參數進行設置，尤其是需要訪問比較大的臨時表時，將會有顯著的性能提升。
max_prepared_transactions ：設置可以同時處於"準備好"狀態的事務的最大數目。 把這個參數設置為零則關閉準備好的事務的特性。max_prepared_transactions 設置成至少和 max_connections 一樣大， 以避免在準備步驟的失敗。 修改本參數需要重啟PostgreSQL服務。
work_mem：工作內存或者操作內存。負責內部的sort和hash操作，合適的work_mem大小能夠保證這些操作在內存中進行。定義太小的話，sort或者hash操作將需要與硬盤進行swap，這樣會極大的降低系統的性能；太大的話致使在能夠在內存中完成的操作數量減少，其他的部分需要與磁盤進行swap操作，增加IO降低性能。系統提供的默認值是1M，在實際的生產環境中，要對系統監控數據進行分析，作出最好的選擇。對於work_mem內存分配時還要考慮數據庫的並發情況，不論如何調整work_mem都要考慮max_connections*work_mem+shared_buffers+temp_buffers+maintenance_work_mem+操作系統所需內存不能夠超過整個的RAM大小
maintenance_work_mem：維護工作內存，主要是針對數據庫的維護操作或者語句。盡量的將這些操作在內存中進行。主要針對VACUUM，CREATE INDEX，REINDEX等操作。
可以看到修改的這幾個參數的主要目的就是盡可能將所有的數據庫和操作放在內存裏進行，通過這種方式來提高性能。
經過參數調整，得到測試結果如下：
VM-SSD
$ pgbench -r -c 16 -j 16 -n -T 180 pgbench
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 500
query mode: simple
number of clients: 16
number of threads: 16
duration: 180 s
number of transactions actually processed: 457226
latency average = 6.300 ms
tps = 2539.724092 (including connections establishing)
tps = 2539.832161 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
0.002 \set aid random(1, 100000 * :scale)
0.001 \set bid random(1, 1 * :scale)
0.001 \set tid random(1, 10 * :scale)
0.001 \set delta random(-5000, 5000)
0.077 BEGIN;
0.204 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
0.155 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
0.168 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
0.231 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
0.120 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
5.340 END;
VM-HDD
$ pgbench -r -c 16 -j 16 -n -T 180 pgbench
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 500
query mode: simple
number of clients: 16
number of threads: 16
duration: 180 s
number of transactions actually processed: 313329
latency average = 9.193 ms
tps = 1740.403152 (including connections establishing)
tps = 1740.455620 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
0.002 \set aid random(1, 100000 * :scale)
0.001 \set bid random(1, 1 * :scale)
0.001 \set tid random(1, 10 * :scale)
0.000 \set delta random(-5000, 5000)
0.073 BEGIN;
0.195 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
0.155 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
0.169 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
0.273 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
0.118 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
8.208 END;
可以看到，經過參數優化，性能有了明顯的提高，不過基於SSD的虛機性能還是遠遠好於基於HDD的虛機
第三步測試：Client/Server模式測試
之前的測試都是在虛擬機上，測試本機的性能。實際工作中，Client通常和數據庫不在同一臺服務器上，所以接下來就要測試下通過網絡訪問的性能。
上一篇說過，安裝部署好的PostgreSQL默認只有本機可以訪問，因此在開始測試前，先要配置允許對外服務，而且出於安全考慮，只有指定的IP地址可以訪問數據庫。
修改postgresql.conf, 設置監聽地址為所有地址：listen_addresses = ‘0.0.0.0‘
修改pg_hba.conf，按提示的格式設置允許訪問的IP地址，註意要加上掩碼。
設置完成後重啟PostgreSQL服務，開始測試。
首先測試通過VNET內網地址互相訪問的結果。
VM-SSD 作為Client訪問VM-HDD private address
$ pgbench -h 10.3.0.5 -p 1999 -r -c 16 -j 16 -n -T 180 pgbench
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 500
query mode: simple
number of clients: 16
number of threads: 16
duration: 180 s
number of transactions actually processed: 340623
latency average = 8.456 ms
tps = 1892.056710 (including connections establishing)
tps = 1892.119631 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
0.002 \set aid random(1, 100000 * :scale)
0.001 \set bid random(1, 1 * :scale)
0.001 \set tid random(1, 10 * :scale)
0.001 \set delta random(-5000, 5000)
0.308 BEGIN;
0.432 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
0.335 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
0.348 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
0.412 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
0.290 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
6.329 END;
此時從VM-SDD ping VM-HDD的時延為：
64 bytes from 10.3.0.5: icmp_seq=1 ttl=64 time=0.459 ms
64 bytes from 10.3.0.5: icmp_seq=2 ttl=64 time=0.447 ms
64 bytes from 10.3.0.5: icmp_seq=3 ttl=64 time=0.226 ms
64 bytes from 10.3.0.5: icmp_seq=4 ttl=64 time=0.231 ms
64 bytes from 10.3.0.5: icmp_seq=5 ttl=64 time=0.264 ms
64 bytes from 10.3.0.5: icmp_seq=6 ttl=64 time=0.300 ms
64 bytes from 10.3.0.5: icmp_seq=7 ttl=64 time=0.260 ms
64 bytes from 10.3.0.5: icmp_seq=8 ttl=64 time=0.302 ms
64 bytes from 10.3.0.5: icmp_seq=9 ttl=64 time=0.226 ms
VM-HDD 作為Client訪問VM-SSD private address
$ pgbench -h 10.3.0.4 -p 1999 -r -c 16 -j 16 -n -T 180 pgbench
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 500
query mode: simple
number of clients: 16
number of threads: 16
duration: 180 s
number of transactions actually processed: 429534
latency average = 6.708 ms
tps = 2385.353651 (including connections establishing)
tps = 2385.425889 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
0.002 \set aid random(1, 100000 * :scale)
0.001 \set bid random(1, 1 * :scale)
0.001 \set tid random(1, 10 * :scale)
0.001 \set delta random(-5000, 5000)
0.294 BEGIN;
0.442 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
0.336 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
0.349 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
0.391 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
0.289 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
4.599 END;
此時從VM-SDD ping VM-HDD的時延為：
64 bytes from 10.3.0.4: icmp_seq=1 ttl=64 time=0.228 ms
64 bytes from 10.3.0.4: icmp_seq=2 ttl=64 time=0.219 ms
64 bytes from 10.3.0.4: icmp_seq=3 ttl=64 time=0.240 ms
64 bytes from 10.3.0.4: icmp_seq=4 ttl=64 time=0.353 ms
64 bytes from 10.3.0.4: icmp_seq=5 ttl=64 time=0.445 ms
64 bytes from 10.3.0.4: icmp_seq=6 ttl=64 time=0.278 ms
64 bytes from 10.3.0.4: icmp_seq=7 ttl=64 time=0.407 ms
64 bytes from 10.3.0.4: icmp_seq=8 ttl=64 time=0.196 ms
64 bytes from 10.3.0.4: icmp_seq=9 ttl=64 time=0.271 ms
從VNET內部訪問的性能已經知道了，那麽如果用公網地址互訪的結果會怎麽樣呢？
VM-SSD 訪問 VM-HDD public address
$ pgbench -h 139.219.110.250 -p 1999 -r -c 16 -j 16 -n -T 180 pgbench
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 500
query mode: simple
number of clients: 16
number of threads: 16
duration: 180 s
number of transactions actually processed: 353836
latency average = 8.141 ms
tps = 1965.306199 (including connections establishing)
tps = 1965.368417 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
0.002 \set aid random(1, 100000 * :scale)
0.001 \set bid random(1, 1 * :scale)
0.001 \set tid random(1, 10 * :scale)
0.001 \set delta random(-5000, 5000)
0.312 BEGIN;
0.446 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
0.340 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
0.351 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
0.413 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
0.292 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
5.985 END;
VM-HDD 訪問 VM-SSD public address
$ pgbench -h 139.219.96.68 -p 1999 -r -c 16 -j 16 -n -T 180 pgbench
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 500
query mode: simple
number of clients: 16
number of threads: 16
duration: 180 s
number of transactions actually processed: 433479
latency average = 6.646 ms
tps = 2407.470097 (including connections establishing)
tps = 2407.548375 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
0.002 \set aid random(1, 100000 * :scale)
0.001 \set bid random(1, 1 * :scale)
0.001 \set tid random(1, 10 * :scale)
0.001 \set delta random(-5000, 5000)
0.291 BEGIN;
0.431 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
0.331 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
0.344 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
0.385 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
0.287 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
4.570 END;
總結測試結果如下表：
|
|
VM-SSD未優化 |
VM-HDD未優化 |
VM-SSD優化 |
VM-HDD優化 |
SSD to HDD Private IP |
HDD to SSD Private IP |
SSD to HDD Public IP |
HDD to SSD Private IP |
|
latency average |
8.636 ms |
12.879 ms |
6.300 ms |
9.193 ms |
8.456 ms |
6.708 ms |
8.141 ms |
6.646 ms |
|
tps |
1852 |
1242 |
2539 |
1740 |
1892 |
2385 |
1965 |
2407 |
其實仔細閱讀了postgresql的文檔後，可以發現work_mem是十分重要的一個參數，系統提供的默認值是1M，在實際的生產環境中，需要對系統監控數據進行分析，作出最好的選擇。推薦使用以下兩種方式：估計方法與計算方法。
第一種是可以根據業務量的大小和類型，一般語句運行時間，來粗略的估計一下。第二種方式是通過對數據庫的監控，數據采集，然後計算其大小。總之合適的大小對系統的性能至關重要。
在實際的維護中可以通過explain
analyze分析語句的work_mem大小是否合適。在語句中設置work_mem參數的大小可以充分利用內存，提高語句的執行效率。
work_mem參數對系統的性能是如此的重要，讓其實時的適應數據庫的運行狀況顯的不太可能，但是可以通過對數據庫運行周期的監控，總結相應的數據，然後定制一個專用的腳本，專門用來修改work_mem的大小，使其階段性的更加適應系統的狀況，不失為一種好的方法。
最後補充一點，如果禁用 synchronous_commit參數，性能會有驚人的提高，不過這樣關閉了日誌的實時同步，雖然對性能有極大的提高，但是可能會造成數據庫在意外時無法保持數據一致性，因此並不建議修改這個參數。
$ pgbench -r -c 16 -j 16 -n -T 180 pgbench
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 500
query mode: simple
number of clients: 16
number of threads: 16
duration: 180 s
number of transactions actually processed: 1667171
latency average = 1.728 ms
tps = 9260.473969 (including connections establishing)
tps = 9260.967802 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
0.002 \set aid random(1, 100000 * :scale)
0.001 \set bid random(1, 1 * :scale)
0.001 \set tid random(1, 10 * :scale)
0.001 \set delta random(-5000, 5000)
0.064 BEGIN;
0.209 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
0.252 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
0.315 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
0.411 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
0.317 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
0.148 END;
PostgreSQL on Azure.cn : 性能測試及調優