在Glusterfs上創建distributed volume,replicated volume,dispersed volume,combined volume
阿新 • • 發佈：2017-07-25
glusterfs
/* */ ~]# for i in {1..5}; do lvcreate -V 2G -T vg_bricks/pool$i -n thinvol$i; done
Logical volume "thinvol1" created.
Logical volume "thinvol2" created.
Logical volume "thinvol3" created.
Logical volume "thinvol4" created.
Logical volume "thinvol5" created.前面一篇寫到了在CentOS上如何安裝glusterfs,以及簡單創建了一個volume並實現了native-mount,今天我們重點看一下在glusterfs上都可以創建哪種類型的volume.
1. 首先還是先介紹下實驗環境，今天共用到了5臺虛擬機，其中4個虛擬機做server端,分別是：
servera.lab.example.com
serverb.lab.example.com
serverc.lab.example.com
serverd.lab.example.com
1個虛擬機做client端：
workstation.lab.example.com
五臺虛擬機之間都打開了防火墻，可以相互訪問，並且可以相互解析hostname,4臺server上已經安裝好了glusterfs相關的package,並組成了一個trusted-storage-pool:
[[email protected] ~]# gluster pool list UUID Hostname State d61aaee4-efe5-4f60-9216-c65fdb0b65f8 serverb.lab.example.com Connected be6e1fe5-ae7d-40d1-901e-8b58fd0a4be3 serverc.lab.example.com Connected 25dda180-3285-47f6-8595-3cb9bdaab92f serverd.lab.example.com Connected 860cd46c-390b-430f-8a01-d3433fa2775c localhost Connected
2.在4臺server上分別創建5個bricks,並掛載到/bricks/thinvol$n目錄下，依次執行以下命令（如非特殊說明，以下命令均為在4臺server上執行，文中以servera為示例）：
a. 實驗環境中每個server上已經提前做好了一個lvm group vg_bricks
[[email protected] ~]# vgs
VG #PV #LV #SN Attr VSize VFree
vg_bricks 1 10 0 wz--n- 20.00g 9.97g
b. 創建瘦邏輯卷
[[email protected] ~]# for i in {1..5}; do lvcreate -L 2G -T vg_bricks/pool$i; done
Logical volume "pool1" created.
Logical volume "pool2" created.
Logical volume "pool3" created.
Logical volume "pool4" created.
Logical volume "pool5" created.
[[email protected]c. 格式化邏輯卷並掛載到指定目錄上，生產環境中掛載時建議將掛載命令寫入/etc/fstab中以實現開機自動掛載
[[email protected] ~]# for i in {1..5}; do mkfs -t xfs -i size=512 /dev/mapper/vg_bricks-thinvol$i; done
[[email protected] ~]# mkdir -p /bricks/thinvol{1..5}
[[email protected] ~]# for i in {1..5}; do mount -t xfs /dev/mapper/vg_bricks-thinvol$i /bricks/thinvol$i; doned. 創建brick並設置selinux安全上下文，這裏默認selinux是打開的
[[email protected] ~]# for i in {1..5}; do mkdir /bricks/thinvol$i/brick; done
[[email protected] ~]# chcon -R -t glusterd_brick_t /bricks/thinvol{1..5}至此，準備工作可以告一段落了
3. 創建分布式卷Distributed volume
[[email protected] ~]# gluster volume create Test01 > servera.lab.example.com:/bricks/thinvol1/brick > serverb.lab.example.com:/bricks/thinvol1/brick > serverc.lab.example.com:/bricks/thinvol1/brick > serverd.lab.example.com:/bricks/thinvol1/brick
volume create: Test01: success: please start the volume to access data
查看volume的狀態，可以看到剛創建完後，volume處在“not started”狀態
[[email protected] ~]# gluster volume status Test01 Volume Test01 is not started
start volume
[[email protected] ~]# gluster volume start Test01 volume start: Test01: success
此時查看volume的詳細信息發現volume已started，volume 類型為distribute
[[email protected] ~]# gluster volume info Test01 Volume Name: Test01 Type: Distribute Volume ID: f40beb82-81ae-42d2-bd1a-a7b9a24abe63 Status: Started Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: servera.lab.example.com:/bricks/thinvol1/brick Brick2: serverb.lab.example.com:/bricks/thinvol1/brick Brick3: serverc.lab.example.com:/bricks/thinvol1/brick Brick4: serverd.lab.example.com:/bricks/thinvol1/brick Options Reconfigured: performance.readdir-ahead: on
在workstation.lab.example.com上掛載volume Test01
[[email protected] ~]# yum install -y glusterfs-fuse [[email protected] ~]# mkdir /mnt/Test01 [[email protected] ~]# mount -t glusterfs servera.lab.example.com:Test01 /mnt/Test01 [[email protected] ~]# df -h | grep Test01 servera.lab.example.com:Test01 8.0G 131M 7.9G 2% /mnt/Test01
這裏可以看到一些distributed volume的特點，有點類似raid0,volume的大小是組成volume的4個brick之和（每個brick大小為2G，共計2G*4=8G）
在/mnt/Test01下創建100個測試文件，會發現文件會“隨機”分配到4個server的brick下
[[email protected] ~]# cd /mnt/Test01/
[[email protected] Test01]# touch {1..100}.file[[email protected] ~]# ls /bricks/thinvol1/brick/ 100.file 13.file 18.file 29.file 32.file 37.file 40.file 47.file 61.file 6.file 76.file 7.file 97.file 12.file 17.file 28.file 2.file 34.file 39.file 44.file 54.file 62.file 75.file 77.file 94.file
[[email protected] ~]# ls /bricks/thinvol1/brick/ 10.file 1.file 25.file 38.file 42.file 52.file 56.file 59.file 66.file 80.file 86.file 98.file 14.file 24.file 26.file 41.file 50.file 53.file 57.file 5.file 74.file 83.file 87.file ...
4. 創建復制卷Replicated volume
[[email protected] ~]# gluster volume create Test02 replica 4 > servera.lab.example.com:/bricks/thinvol2/brick > serverb.lab.example.com:/bricks/thinvol2/brick > serverc.lab.example.com:/bricks/thinvol2/brick > serverd.lab.example.com:/bricks/thinvol2/brick volume create: Test02: success: please start the volume to access data
start volume Test02並查看volume的詳細信息，可以看到volume的類型為Replicate即復制卷
[[email protected] ~]# gluster volume start Test02 volume start: Test02: success [[email protected] ~]# gluster volume info Test02 Volume Name: Test02 Type: Replicate Volume ID: 305f2c2f-8550-40e0-9d75-a2bc8149a333 Status: Started Number of Bricks: 1 x 4 = 4 Transport-type: tcp Bricks: Brick1: servera.lab.example.com:/bricks/thinvol2/brick Brick2: serverb.lab.example.com:/bricks/thinvol2/brick Brick3: serverc.lab.example.com:/bricks/thinvol2/brick Brick4: serverd.lab.example.com:/bricks/thinvol2/brick Options Reconfigured: performance.readdir-ahead: on
將Test02掛載到client下的/mnt/Test02目錄下
[[email protected] Test01]# mkdir /mnt/Test02 [[email protected] Test01]# mount -t glusterfs servera.lab.example.com:Test02 /mnt/Test02 [[email protected] Test02]# df -h | grep Test02
servera.lab.example.com:Test02 2.0G 33M 2.0G 2% /mnt/Test02
這裏可以看到Test02掛載後的大小為2G，即和組成Test02的每個brick一樣大小，其特點類似raid1
在/mnt/Test02 下創建100個測試文件，會發現每個server的brick下都創建了100個測試文件
[[email protected] Test02]# touch {1..100}.file[[email protected] ~]# ls /bricks/thinvol2/brick/ 100.file 16.file 22.file 29.file 35.file 41.file 48.file 54.file 60.file 67.file 73.file 7.file 86.file 92.file 99.file 10.file 17.file 23.file 2.file 36.file 42.file 49.file 55.file 61.file 68.file 74.file 80.file 87.file 93.file 9.file 11.file 18.file 24.file 30.file 37.file 43.file 4.file 56.file 62.file 69.file 75.file 81.file 88.file 94.file 12.file 19.file 25.file 31.file 38.file 44.file 50.file 57.file 63.file 6.file 76.file 82.file 89.file 95.file 13.file 1.file 26.file 32.file 39.file 45.file 51.file 58.file 64.file 70.file 77.file 83.file 8.file 96.file 14.file 20.file 27.file 33.file 3.file 46.file 52.file 59.file 65.file 71.file 78.file 84.file 90.file 97.file 15.file 21.file 28.file 34.file 40.file 47.file 53.file 5.file 66.file 72.file 79.file 85.file 91.file 98.file
[[email protected] ~]# ls /bricks/thinvol2/brick/ | wc -w 100
[[email protected] ~]# ls /bricks/thinvol2/brick/ | wc -w 100
[[email protected] ~]# ls /bricks/thinvol2/brick/ | wc -w 100
5. 創建“分散卷”Dispersed volume
[[email protected] ~]# gluster volume create Test03 > disperse-data 2 redundancy 1 > servera.lab.example.com:/bricks/thinvol3/brick > serverb.lab.example.com:/bricks/thinvol3/brick > serverc.lab.example.com:/bricks/thinvol3/brick volume create: Test03: success: please start the volume to access data [[email protected] ~]# gluster volume start Test03 volume start: Test03: success [[email protected] ~]# gluster volume info Test03 Volume Name: Test03 Type: Disperse Volume ID: 78d23935-8d67-4eb1-86b6-e94c7c8866c9 Status: Started Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: servera.lab.example.com:/bricks/thinvol3/brick Brick2: serverb.lab.example.com:/bricks/thinvol3/brick Brick3: serverc.lab.example.com:/bricks/thinvol3/brick Options Reconfigured: performance.readdir-ahead: on
將dispersed volume Test03 掛載到client server /mnt/Test03目錄下
[[email protected] Test02]# mkdir /mnt/Test03 [[email protected] Test02]# mount -t glusterfs servera.lab.example.com:Test03 /mnt/Test03 [[email protected] Test02]# df -h | grep Test03 servera.lab.example.com:Test03 4.0G 66M 4.0G 2% /mnt/Test03
從上可以看出Test03的可用空間大小為4G，為2個brick（
servera.lab.example.com:/bricks/thinvol3/brick serverb.lab.example.com:/bricks/thinvol3/brick
）的大小
在/mnt/Test03下創建100個測試文件，觀察數據分布##這裏測試的結果和我原來理解的有些區別
[[email protected] Test03]# touch {1..100}.file[[email protected] ~]# ls /bricks/thinvol3/brick/ 100.file 16.file 22.file 29.file 35.file 41.file 48.file 54.file 60.file 67.file 73.file 7.file 86.file 92.file 99.file 10.file 17.file 23.file 2.file 36.file 42.file 49.file 55.file 61.file 68.file 74.file 80.file 87.file 93.file 9.file 11.file 18.file 24.file 30.file 37.file 43.file 4.file 56.file 62.file 69.file 75.file 81.file 88.file 94.file 12.file 19.file 25.file 31.file 38.file 44.file 50.file 57.file 63.file 6.file 76.file 82.file 89.file 95.file 13.file 1.file 26.file 32.file 39.file 45.file 51.file 58.file 64.file 70.file 77.file 83.file 8.file 96.file 14.file 20.file 27.file 33.file 3.file 46.file 52.file 59.file 65.file 71.file 78.file 84.file 90.file 97.file 15.file 21.file 28.file 34.file 40.file 47.file 53.file 5.file 66.file 72.file 79.file 85.file 91.file 98.file
[[email protected] ~]# ls /bricks/thinvol3/brick/ | wc -w 100
[[email protected] ~]# ls /bricks/thinvol3/brick/ | wc -w 100
6. 創建復合卷Combined volume
復合卷就是前面三種卷的組合，可以是distributed-replicated,也可以是distributed-dispersed,但是貌似沒有replicated-dispersed的組合
[[email protected] ~]# gluster volume create Test04 replica 3 > servera.lab.example.com:/bricks/thinvol4/brick > serverb.lab.example.com:/bricks/thinvol4/brick > serverc.lab.example.com:/bricks/thinvol4/brick > servera.lab.example.com:/bricks/thinvol5/brick > serverb.lab.example.com:/bricks/thinvol5/brick > serverc.lab.example.com:/bricks/thinvol5/brick volume create: Test04: success: please start the volume to access data [[email protected] ~]# gluster volume start Test04 volume start: Test04: success [[email protected] ~]# gluster volume info Test04 Volume Name: Test04 Type: Distributed-Replicate Volume ID: 45df998d-2d14-4af2-83c7-e497bd5a8dd0 Status: Started Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: servera.lab.example.com:/bricks/thinvol4/brick Brick2: serverb.lab.example.com:/bricks/thinvol4/brick Brick3: serverc.lab.example.com:/bricks/thinvol4/brick Brick4: servera.lab.example.com:/bricks/thinvol5/brick Brick5: serverb.lab.example.com:/bricks/thinvol5/brick Brick6: serverc.lab.example.com:/bricks/thinvol5/brick Options Reconfigured: performance.readdir-ahead: on
從上面可以看出volume Test04的類型為Distributed-Replicate
在client上掛載Test04，Test04的可用空間為4G
[[email protected] Test03]# mkdir /mnt/Test04 [[email protected] Test03]# mount -t glusterfs servera.lab.example.com:Test04 /mnt/Test04 [[email protected] Test03]# df -h | grep Test04 servera.lab.example.com:Test04 4.0G 66M 4.0G 2% /mnt/Test04
在/mnt/Test04下創建100個測試文件，並查看下數據分布
[[email protected] Test04]# touch {1..100}.file[[email protected] thinvol4]# ls /bricks/thinvol4/brick/ | wc -w 48 [[email protected] thinvol4]# ls /bricks/thinvol5/brick/ | wc -w 52 [[email protected] thinvol4]#
[[email protected] ~]# ls /bricks/thinvol4/brick/ | wc -w 48 [[email protected] ~]# ls /bricks/thinvol5/brick/ | wc -w 52
[[email protected] ~]# ls /bricks/thinvol4/brick/ | wc -w 48 [[email protected] ~]# ls /bricks/thinvol5/brick/ | wc -w 52
可以發現
Brick1: servera.lab.example.com:/bricks/thinvol4/brick Brick2: serverb.lab.example.com:/bricks/thinvol4/brick Brick3: serverc.lab.example.com:/bricks/thinvol4/brick
組成了一個3副本，每個brick上的數據一致
Brick4: servera.lab.example.com:/bricks/thinvol5/brick Brick5: serverb.lab.example.com:/bricks/thinvol5/brick Brick6: serverc.lab.example.com:/bricks/thinvol5/brick
組成了一個3副本，每個brick上的數據一致
當然了，這裏的例子限於實驗環境的局限性，每個server上選取了兩個brick，生產環境中還是要盡量杜絕這種情況的。Ok，今天就到這裏了。
本文出自 “不積跬步，無以至千裏” 博客，請務必保留此出處http://jiaxiaolei.blog.51cto.com/3117381/1950629
在Glusterfs上創建distributed volume,replicated volume,dispersed volume,combined volume