Ceph叢集部署實戰
建議系統核心升級到4.x，不然容易出各種問題，不要給自己找不必要的麻煩。 參考文章:Centos7 核心編譯升級到指定的版本
https://blog.csdn.net/happyfreeangel/article/details/85088706
[[email protected] ceph-ansible-3.1.7]# more hosts
[admins]
10.20.4.10
[osds]
10.20.4.21
10.20.4.22
10.20.4.23
[rgws]
10.20.4.11
10.20.4.12
10.20.4.13
[mds]
10.20.4.31
[clients]
10.20.4.51
[rbdmirrors]
10.20.4.51
[mons]
10.20.4.1
10.20.4.2
10.20.4.3
[mgrs]
10.20.4.1
10.20.4.2
10.20.4.3
[agents]
10.20.4.51
1.自動建立虛擬機器 （OK）
我是用vmware Esxi 做實驗的，全場採用自動化建立虛擬機器。
1.1 確保虛擬機器有2個磁碟 一個是系統本身的,來自虛擬機器模版, 預設是/dev/sda,
另外是/dev/sdb
/dev/sdc
為了實驗，我建立了/dev/sdb 我是都在osd 主機上建立的，每個建立500GB 也就是 /dev/sdb 容量500GB .
這個可以通過ansible 自動化完成.
2.設定免密碼登陸. 詳細請參考文章:
#下面這個操作會建立{{ceph_username}}帳號,下面的group, owner 等建立資料夾時會用到. - name: "安裝不同主機間使用者{{hostconfig['hadoop_config']['the_ceph_username']}}之間免密碼登陸 passwordless-ssh-login" include_role: name: passwordless-ssh-login vars: user_host_list: "{{passwordless_host_list}}" username: "root" password: "{{ceph_salt_password}}" sudo_privilege: False auto_generate_etc_host_list: True debug_mode: False #下面這個操作會建立{{ceph_username}}帳號,下面的group, owner 等建立資料夾時會用到. - name: "安裝不同主機間使用者{{hostconfig['hadoop_config']['the_ceph_username']}}之間免密碼登陸 passwordless-ssh-login" include_role: name: passwordless-ssh-login vars: user_host_list: "{{passwordless_host_list}}" username: "{{ceph_username}}" password: "{{ceph_salt_password}}" sudo_privilege: True debug_mode: False
3.關閉所有yum repo 源頭,設定yum 源頭,請把docker-ce.repo 重新命名一下,這個連線到國外,老卡住,容易導致安裝失敗.
- ceph-admin 安裝ansible 安裝虛擬環境
4.1 確保你的虛擬機器已經有了python
4.2 安裝虛擬機器環境 virtual-env
pip install virtualenv
virtualenv -p python2 venv #可以指定python版本 python3
#python3 -m venv venv #如果你有安裝python3的話，也可以這樣操作.
5.切換到虛擬環境
source venv/bin/activate
- 在ceph-admin 虛擬機器上 安裝指定版本的ansible
6.1 切換目錄 cd /home/ceph
6.2 用pip執行安裝ansible #務必嚴格按照官方文件的要求來，ansible版本過新或舊都會有各種報錯。 這裡注意不要用yum install ansible
pip install ansible==2.4.2
或者用git 下載
#6.2 下載ceph-ansible 部署指令碼
wget -c https://github.com/ceph/ceph-ansible/archive/v3.1.7.tar.gz
tar xf v3.1.7.tar.gz
cd ceph-ansible-3.1.7
或 git clone https://github.com/ceph/ceph-ansible.git
cd ceph-ansible
git checkout remotes/origin/stable-3.2 # 這裡剛好沒有3.1.7 版本。
#在虛擬環境裡安裝依賴包
pip install -r requirements.txt
- 準備好配置檔案
cp group_vars/all.yml.sample group_vars/all.yml
cp group_vars/osds.yml.sample group_vars/osds.yml
cp site.yml.sample site.yml
vim group_vars/all.yml
[[email protected] group_vars]# more all.yml
ceph_origin: repository
ceph_repository: community
ceph_mirror: http://mirrors.163.com/ceph
ceph_stable_key: http://mirrors.163.com/ceph/keys/release.asc
ceph_stable_release: luminous
ceph_stable_repo: “{{ ceph_mirror }}/rpm-{{ ceph_stable_release }}”
#fsid: 82D6CE06-6E92-4C2A-AB26-11FF63B7E67D ##通過uuidgen生成
fsid: 82d6ce06-6e92-4c2a-ab26-11ff63b7e67d ##通過uuidgen生成 建議改成小寫的,大寫的之前執行指令碼有問題會報錯.
generate_fsid: false
cephx: true
public_network: 10.20.0.0/16
cluster_network: 10.20.0.0/16
monitor_interface: ens160
ceph_conf_overrides:
global:
rbd_default_features: 7
auth cluster required: cephx
auth service required: cephx
auth client required: cephx
osd journal size: 2048
osd pool default size: 3
osd pool default min size: 1
mon_pg_warn_max_per_osd: 1024
osd pool default pg num: 128
osd pool default pgp num: 128
max open files: 131072
osd_deep_scrub_randomize_ratio: 0.01
mgr:
mgr modules: dashboard
mon:
mon_allow_pool_delete: true
client:
rbd_cache: true
rbd_cache_size: 335544320
rbd_cache_max_dirty: 134217728
rbd_cache_max_dirty_age: 10
osd:
osd mkfs type: xfs
# osd mount options xfs: “rw,noexec,nodev,noatime,nodiratime,nobarrier”
ms_bind_port_max: 7100
osd_client_message_size_cap: 2147483648
osd_crush_update_on_start: true
osd_deep_scrub_stride: 131072
osd_disk_threads: 4
osd_map_cache_bl_size: 128
osd_max_object_name_len: 256
osd_max_object_namespace_len: 64
osd_max_write_size: 1024
osd_op_threads: 8
osd_recovery_op_priority: 1
osd_recovery_max_active: 1
osd_recovery_max_single_start: 1
osd_recovery_max_chunk: 1048576
osd_recovery_threads: 1
osd_max_backfills: 4
osd_scrub_begin_hour: 23
osd_scrub_end_hour: 7
# bluestore block create: true
# bluestore block db size: 73014444032
# bluestore block db create: true
# bluestore block wal size: 107374182400
# bluestore block wal create: true
vim group_vars/osds.yml
devices:
- /dev/sdb
- /dev/vdd
- /dev/vde
osd_scenario: collocated
osd_objectstore: bluestore
#osd_scenario: non-collocated
#osd_objectstore: bluestore
#devices:
- /dev/sdc
- /dev/sdd
- /dev/sde
#dedicated_devices:
- /dev/sdf
- /dev/sdf
- /dev/sdf
#bluestore_wal_devices:
- /dev/sdg
- /dev/sdg
- /dev/sdg
#monitor_address: 192.168.66.125
註釋不需要的元件
vim site.yml
Defines deployment design and assigns role to server groups
- hosts:
- mons
- agents
- osds
- mdss
- rgws
- nfss
- restapis
- rbdmirrors
- clients
- mgrs
- iscsigws
- iscsi-gws # for backward compatibility only!
[[email protected] group_vars]# more osds.yml
devices:
- /dev/sdb
osd_scenario: collocated
osd_objectstore: bluestore
#osd_scenario: non-collocated
#osd_objectstore: bluestore
#devices:
- /dev/sdc
- /dev/sdd
- /dev/sde
#dedicated_devices:
- /dev/sdf
- /dev/sdf
- /dev/sdf
#bluestore_wal_devices:
- /dev/sdg
- /dev/sdg
- /dev/sdg
#monitor_address: 192.168.66.125
註釋不需要的元件
vim site.yml
Defines deployment design and assigns role to server groups
- hosts:
- mons
- agents
- osds
- mdss
- rgws
- nfss
- restapis
- rbdmirrors
- clients
- mgrs
- iscsigws
- iscsi-gws # for backward compatibility only!
客戶端解除安裝pip中安裝的urllib3，不然會失敗
pip freeze|grep urllib3
pip uninstall urllib3
ansible-playbook -i hosts site.yml
至此ceph部署完成，登陸ceph節點檢查狀態。
清空叢集
如果部署過程中出現報錯，建議先清空叢集 再進行部署操作
cp infrastructure-playbooks/purge-cluster.yml purge-cluster.yml # 必須copy到專案根目錄下
ansible-playbook -i hosts purge-cluster.yml
建立客戶端訪問環境:
[[email protected] ~]# ceph auth get-or-create client.rbd mon ‘allow r’ osd ‘allow class-read object_prefix rbd_children,allow rwx pool=rbd’
[client.rbd]
key = AQAMyxxc1zSRBRAAhp8HTqmXXky8azntPF7gdQ==
[[email protected] ~]#
[[email protected] ~]# ceph auth get-or-create client.rbd | ssh [email protected] sudo tee /etc/ceph/ceph.client.rbd.keyring
The authenticity of host ‘client1 (10.20.4.51)’ can’t be established.
ECDSA key fingerprint is SHA256:1Li21Z7NsbSmDsHubXT2R7tYFNM003JYfEuSuV8mSJ4.
ECDSA key fingerprint is MD5:b9:d0:23:5d:12:26:84:b9:76:a6:d6:7a:e9:9a:b0:46.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘client1,10.20.4.51’ (ECDSA) to the list of known hosts.
[email protected]’s password:
[client.rbd]
key = AQAMyxxc1zSRBRAAhp8HTqmXXky8azntPF7gdQ==
[[email protected] ~]#
#在client1 上操作
cat /etc/ceph/ceph.client.rbd.keyring >> /etc/ceph/keyring
Since we are not using the default user client.admin we
need to supply username that will connect to the Ceph cluster
ceph -s --name client.rbd