1. 程式人生 > >Oracle RAC環境下ASM磁碟組擴容

Oracle RAC環境下ASM磁碟組擴容

生產環境注意調整以下引數:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ 1.節點間滾動新增UDEV磁碟
+ 2.ASM 記憶體大小
+ 3.POWER_LIMIT別弄太大
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

實驗環境
--系統環境:Centos 6.10 --資料庫:Oracle 11g RAC --磁碟繫結:udev 實驗目的 --儲存空間不足,需要給磁碟組DATA加一塊磁碟 準備工作 1.檢查各個節點database、asm及grid日誌是否有錯誤資訊 2.檢查各個節點服務狀態(切換到grid使用者) --列出資料庫名 [[email protected] ~]$ srvctl config database RacNode [[email protected] ~]$ --例項狀態 [
[email protected]
~]$ srvctl status database -d RacNode Instance RacNode1 is running on node node1 Instance RacNode2 is running on node node2 [[email protected] ~]$ --叢集狀態: [[email protected] ~]$ crsctl check cluster -all ************************************************************** node1: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** node2: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** [
[email protected]
~]$ --叢集資源狀態 [[email protected] ~]$ crsctl status res -t 3.ASM磁碟組及磁碟檢查 [[email protected] ~]$ export ORACLE_SID=+ASM1 [[email protected] ~]$ sqlplus /nolog SQL> conn /as sysasm --顯示使用ASM磁碟組的資料庫 SQL> col INSTANCE_NAME format a20 SQL> col SOFTWARE_VERSION format a20 SQL> select * from gv$asm_client order by 1,2; INST_ID GROUP_NUMBER INSTANCE_NAME DB_NAME STATUS SOFTWARE_VERSION COMPATIBLE_VERSION ---------- ------------ -------------------- -------- ------------ -------------------- --------------------- 1 1 RacNode1 RacNode CONNECTED 11.2.0.4.0 11.2.0.4.0 1 1 +ASM1 +ASM CONNECTED 11.2.0.4.0 11.2.0.4.0 1 3 +ASM1 +ASM CONNECTED 11.2.0.4.0 11.2.0.4.0 2 1 RacNode2 RacNode CONNECTED 11.2.0.4.0 11.2.0.4.0 2 1 +ASM2 +ASM CONNECTED 11.2.0.4.0 11.2.0.4.0 2 3 +ASM2 +ASM CONNECTED 11.2.0.4.0 11.2.0.4.0 SQL> --顯示磁碟組 SQL> select group_number,name,state,type,total_mb,free_mb,usable_file_mb,allocation_unit_size/1024/1024 unit_mb from v$asm_diskgroup order by 1; GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB USABLE_FILE_MB UNIT_MB ------------ ------------------------------ ----------- ------ ---------- ---------- -------------- ---------- 1 DATA MOUNTED NORMAL 4096 477 -273 1 2 FLASH MOUNTED EXTERN 2048 1951 1951 1 3 OCRVOTE MOUNTED NORMAL 3072 2146 561 1 SQL> --顯示磁碟 SQL> col NAME format a25 SQL> col PATH format a40 SQL> col FAILGROUP format a25 SQL> select group_number,name,path,mount_status,state,redundancy,total_mb,free_mb,failgroup,create_date from v$asm_disk order by 1,2; GROUP_NUMBER NAME PATH MOUNT_S STATE REDUNDA TOTAL_MB FREE_MB FAILGROUP CREATE_DA ------------ ------------------------- ---------------------------------------- ------- -------- ------- ---------- ---------- ------------------------- --------- 1 DATA_0000 /dev/asm-data01 CACHED NORMAL UNKNOWN 1024 118 DATA_0000 11-DEC-18 1 DATA_0001 /dev/asm-data02 CACHED NORMAL UNKNOWN 1024 125 DATA_0001 11-DEC-18 1 DATA_0002 /dev/asm-data03 CACHED NORMAL UNKNOWN 1024 115 DATA_0002 11-DEC-18 1 DATA_0003 /dev/asm-data04 CACHED NORMAL UNKNOWN 1024 119 DATA_0003 11-DEC-18 2 FLASH_0000 /dev/asm-arch01 CACHED NORMAL UNKNOWN 1024 975 FLASH_0000 11-DEC-18 2 FLASH_0001 /dev/asm-arch02 CACHED NORMAL UNKNOWN 1024 976 FLASH_0001 11-DEC-18 3 OCRVOTE_0000 /dev/asm-ocrvote01 CACHED NORMAL UNKNOWN 1024 715 OCRVOTE_0000 11-DEC-18 3 OCRVOTE_0001 /dev/asm-ocrvote02 CACHED NORMAL UNKNOWN 1024 715 OCRVOTE_0001 11-DEC-18 3 OCRVOTE_0002 /dev/asm-ocrvote03 CACHED NORMAL UNKNOWN 1024 716 OCRVOTE_0002 11-DEC-18 SQL> 4.系統層面UDEV資訊 [[email protected] ~]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB132e6928-d49d18d4", NAME="asm-ocrvote01", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB58647ee7-b466963a", NAME="asm-ocrvote02", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB4c771d58-f17105b9", NAME="asm-ocrvote03", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB88c685cb-3a4633f4", NAME="asm-data01", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB1b30fd18-af14e003", NAME="asm-data02", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB67a4423a-e151f28b", NAME="asm-data03", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB81f04366-170fc910", NAME="asm-data04", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB9a0f6e52-bcfcb52c", NAME="asm-arch01", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB1f2bac75-512a46ae", NAME="asm-arch02", OWNER="grid", GROUP="asmadmin", MODE="0660" [[email protected] ~]# [[email protected] ~]# ll /dev/asm-* brw-rw---- 1 grid asmadmin 8, 16 Dec 12 17:22 /dev/asm-arch01 brw-rw---- 1 grid asmadmin 8, 32 Dec 12 16:21 /dev/asm-arch02 brw-rw---- 1 grid asmadmin 8, 48 Dec 12 17:22 /dev/asm-data01 brw-rw---- 1 grid asmadmin 8, 64 Dec 12 17:22 /dev/asm-data02 brw-rw---- 1 grid asmadmin 8, 80 Dec 12 17:22 /dev/asm-data03 brw-rw---- 1 grid asmadmin 8, 96 Dec 12 17:22 /dev/asm-data04 brw-rw---- 1 grid asmadmin 8, 112 Dec 12 17:22 /dev/asm-ocrvote01 brw-rw---- 1 grid asmadmin 8, 128 Dec 12 17:22 /dev/asm-ocrvote02 brw-rw---- 1 grid asmadmin 8, 144 Dec 12 17:22 /dev/asm-ocrvote03 [[email protected] ~]# 進入實驗階段 1.停止對應資料庫的業務 2.檢查資料庫會話及停止監聽 --檢查各個節點監聽狀態 [[email protected] ~]$ srvctl status listener -n node1 Listener LISTENER is enabled on node(s): node1 Listener LISTENER is running on node(s): node1 [[email protected] ~]$ srvctl status listener -n node2 Listener LISTENER is enabled on node(s): node2 Listener LISTENER is running on node(s): node2 [[email protected] ~]$ --禁止監聽自啟動 [[email protected] ~]$ srvctl disable listener -n node1 [[email protected] ~]$ srvctl disable listener -n node2 --停止監聽 [[email protected] ~]$ srvctl stop listener -n node1 [[email protected] ~]$ srvctl stop listener -n node2 --檢視停止及關閉自啟後的監聽狀態 [[email protected] ~]$ srvctl status listener -n node1 Listener LISTENER is disabled on node(s): node1 Listener LISTENER is not running on node(s): node1 [[email protected] ~]$ srvctl status listener -n node2 Listener LISTENER is disabled on node(s): node2 Listener LISTENER is not running on node(s): node2 [[email protected] ~]$ 3.關閉資料庫 --檢查資料庫配置 [[email protected] ~]$ srvctl config database -d RacNode Database unique name: RacNode Database name: RacNode Oracle home: /u01/app/oracle/11.2.0/dbhome_1 Oracle user: oracle Spfile: +DATA/RacNode/spfileRacNode.ora Domain: Start options: open Stop options: immediate Database role: PRIMARY Management policy: AUTOMATIC Server pools: RacNode Database instances: RacNode1,RacNode2 Disk Groups: DATA Mount point paths: Services: Type: RAC Database is administrator managed [[email protected] ~]$ --禁止資料庫自啟動(需切換root使用者) [[email protected] ~]# cd /u01/app/11.2.0/grid/bin [[email protected] bin]# ./srvctl disable database -d RacNode [[email protected] bin]# --關閉資料庫 [[email protected] ~]$ srvctl stop database -d RacNode [[email protected] ~]$ --檢查關閉後資料庫狀態 [[email protected] ~]$ srvctl status database -d RacNode Instance RacNode1 is not running on node node1 Instance RacNode2 is not running on node node2 [[email protected] ~]$ 4.關閉叢集軟體 --檢視各個節點叢集是否為自啟動 [[email protected] bin]# ./crsctl config has CRS-4622: Oracle High Availability Services autostart is enabled. [[email protected] bin]# [[email protected] bin]# ./crsctl config has CRS-4622: Oracle High Availability Services autostart is enabled. --禁止各個節點的自啟動 [[email protected] bin]# ./crsctl disable has CRS-4621: Oracle High Availability Services autostart is disabled. [[email protected] bin]# [[email protected] bin]# ./crsctl disable has CRS-4621: Oracle High Availability Services autostart is disabled. [[email protected] bin]# --檢視各個節點禁止自啟動是否生效 [[email protected] bin]# ./crsctl config has CRS-4621: Oracle High Availability Services autostart is disabled. [[email protected] bin]# [[email protected] bin]# ./crsctl config has CRS-4621: Oracle High Availability Services autostart is disabled. [[email protected] bin]# --停止各個節點叢集 [[email protected] bin]# ./crsctl stop has [[email protected] bin]# ./crsctl stop has 5.系統層面新增磁碟(儲存工程師協助完成) 6.各個節點使用UDEV新增磁碟 --確定碟符 fdisk -l /dev/sdk /dev/sdl /dev/sdm /dev/sdn --獲取繫結規則 for i in k l m n do echo "KERNEL==\"sd*\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/\$name\", RESULT==\"`/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sd$i`\", NAME=\"asm-disk$i\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\"" done KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB00e4d091-5990307e", NAME="asm-data05", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB8bc8125c-72dacc92", NAME="asm-data06", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VBc195c913-00f6c68e", NAME="asm-data07", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB720e4cca-32c20936", NAME="asm-data08", OWNER="grid", GROUP="asmadmin", MODE="0660" --各個節點將規則新增到rule檔案99-oracle-asmdevices.rules --重啟udev [[email protected] rules.d]# start_udev [[email protected] rules.d]# start_udev --確認新增磁碟,確保各個節點可以識別到磁碟且一致 [[email protected] rules.d]# ll /dev/asm-* brw-rw---- 1 grid asmadmin 8, 16 Dec 13 16:02 /dev/asm-arch01 brw-rw---- 1 grid asmadmin 8, 32 Dec 13 16:02 /dev/asm-arch02 brw-rw---- 1 grid asmadmin 8, 48 Dec 13 16:02 /dev/asm-data01 brw-rw---- 1 grid asmadmin 8, 64 Dec 13 16:02 /dev/asm-data02 brw-rw---- 1 grid asmadmin 8, 80 Dec 13 16:02 /dev/asm-data03 brw-rw---- 1 grid asmadmin 8, 96 Dec 13 16:02 /dev/asm-data04 brw-rw---- 1 grid asmadmin 8, 160 Dec 13 16:13 /dev/asm-data05 brw-rw---- 1 grid asmadmin 8, 176 Dec 13 16:13 /dev/asm-data06 brw-rw---- 1 grid asmadmin 8, 192 Dec 13 16:13 /dev/asm-data07 brw-rw---- 1 grid asmadmin 8, 208 Dec 13 16:13 /dev/asm-data08 brw-rw---- 1 grid asmadmin 8, 112 Dec 13 16:02 /dev/asm-ocrvote01 brw-rw---- 1 grid asmadmin 8, 128 Dec 13 16:02 /dev/asm-ocrvote02 brw-rw---- 1 grid asmadmin 8, 144 Dec 13 16:02 /dev/asm-ocrvote03 [[email protected] rules.d]# 7.啟動叢集 [[email protected] bin]# ./crsctl start has CRS-4123: Oracle High Availability Services has been started. [[email protected] bin]# ./crsctl start has CRS-4123: Oracle High Availability Services has been started. [[email protected] bin]# --檢查叢集的各個元件是否啟動正常 [[email protected] ~]$ crsctl status res -t 此時,監聽和資料庫服務是停掉的 8.ASM擴容 --檢查asm是否識別到未新增的磁碟 SQL> set line 200 SQL> col NAME format a25 SQL> col PATH format a40 SQL> col FAILGROUP format a25 SQL> select group_number,name,path,mount_status,state,redundancy,total_mb,free_mb,failgroup,create_date from v$asm_disk order by 1,2; GROUP_NUMBER NAME PATH MOUNT_S STATE REDUNDA TOTAL_MB FREE_MB FAILGROUP CREATE_DA ------------ ------------------------- ---------------------------------------- ------- -------- ------- ---------- ---------- ------------------------- --------- 0 /dev/asm-data07 CLOSED NORMAL UNKNOWN 0 0 0 /dev/asm-data08 CLOSED NORMAL UNKNOWN 0 0 0 /dev/asm-data06 CLOSED NORMAL UNKNOWN 0 0 0 /dev/asm-data05 CLOSED NORMAL UNKNOWN 0 0 1 DATA_0000 /dev/asm-data01 CACHED NORMAL UNKNOWN 1024 118 DATA_0000 11-DEC-18 1 DATA_0001 /dev/asm-data02 CACHED NORMAL UNKNOWN 1024 125 DATA_0001 11-DEC-18 1 DATA_0002 /dev/asm-data03 CACHED NORMAL UNKNOWN 1024 115 DATA_0002 11-DEC-18 1 DATA_0003 /dev/asm-data04 CACHED NORMAL UNKNOWN 1024 119 DATA_0003 11-DEC-18 2 FLASH_0000 /dev/asm-arch01 CACHED NORMAL UNKNOWN 1024 975 FLASH_0000 11-DEC-18 2 FLASH_0001 /dev/asm-arch02 CACHED NORMAL UNKNOWN 1024 976 FLASH_0001 11-DEC-18 3 OCRVOTE_0000 /dev/asm-ocrvote01 CACHED NORMAL UNKNOWN 1024 715 OCRVOTE_0000 11-DEC-18 3 OCRVOTE_0001 /dev/asm-ocrvote02 CACHED NORMAL UNKNOWN 1024 715 OCRVOTE_0001 11-DEC-18 3 OCRVOTE_0002 /dev/asm-ocrvote03 CACHED NORMAL UNKNOWN 1024 716 OCRVOTE_0002 11-DEC-18 SQL> --給磁碟組DATA新增磁碟 SQL> alter diskgroup DATA add disk '/dev/asm-data05' rebalance power 5; SQL> alter diskgroup DATA add disk '/dev/asm-data06' rebalance power 5; 注:rebalance power的級別從1到11中選擇一個數值;數值越大,rebalance速度越快,對現有執行系統影響也越大。需要根據當時業務權衡選擇適合的級別; --監控磁碟組rebalance完成情況 select * from v$asm_operation; --檢查新增結果(本次實驗有2塊盤未新增) SQL> col name format a20 SQL> set line 200 SQL> col name format a20 SQL> col path format a40 SQL> select group_number,name,path,mount_status,state,redundancy,total_mb,free_mb,failgroup,create_date from v$asm_disk order by 1,2; GROUP_NUMBER NAME PATH MOUNT_S STATE REDUNDA TOTAL_MB FREE_MB FAILGROUP CREATE_DA ------------ -------------------- ---------------------------------------- ------- -------- ------- ---------- ---------- ------------------------------ --------- 0 /dev/asm-data08 CLOSED NORMAL UNKNOWN 0 0 0 /dev/asm-data06 CLOSED NORMAL UNKNOWN 0 0 0 /dev/asm-data07 CLOSED NORMAL UNKNOWN 0 0 1 DATA_0000 /dev/asm-data01 CACHED NORMAL UNKNOWN 1024 300 DATA_0000 11-DEC-18 1 DATA_0001 /dev/asm-data02 CACHED NORMAL UNKNOWN 1024 297 DATA_0001 11-DEC-18 1 DATA_0002 /dev/asm-data03 CACHED NORMAL UNKNOWN 1024 298 DATA_0002 11-DEC-18 1 DATA_0003 /dev/asm-data04 CACHED NORMAL UNKNOWN 1024 298 DATA_0003 11-DEC-18 1 DATA_0004 /dev/asm-data05 CACHED NORMAL UNKNOWN 1024 306 DATA_0004 13-DEC-18 2 FLASH_0000 /dev/asm-arch01 CACHED NORMAL UNKNOWN 1024 975 FLASH_0000 11-DEC-18 2 FLASH_0001 /dev/asm-arch02 CACHED NORMAL UNKNOWN 1024 976 FLASH_0001 11-DEC-18 3 OCRVOTE_0000 /dev/asm-ocrvote01 CACHED NORMAL UNKNOWN 1024 715 OCRVOTE_0000 11-DEC-18 3 OCRVOTE_0001 /dev/asm-ocrvote02 CACHED NORMAL UNKNOWN 1024 715 OCRVOTE_0001 11-DEC-18 3 OCRVOTE_0002 /dev/asm-ocrvote03 CACHED NORMAL UNKNOWN 1024 716 OCRVOTE_0002 11-DEC-18 SQL> 9.啟動資料庫 --啟動資料庫自啟動服務(ora.racnode.db),否則無法用srvctl啟動資料庫 [[email protected] bin]# ./srvctl enable database -d RacNode [[email protected] bin]# [grid[email protected] ~]$ srvctl start database -d RacNode [[email protected] ~]$ srvctl status database -d RacNode Instance RacNode1 is running on node node1 Instance RacNode2 is running on node node2 [[email protected] ~]$ 10.啟動監聽 --啟動監聽自啟動服務,否則無法用srvctl啟動資料庫 [[email protected] ~]$ srvctl enable listener -n node1 [[email protected] ~]$ srvctl enable listener -n node2 [[email protected] ~]$ srvctl start listener -n node1 [[email protected] ~]$ srvctl start listener -n node2