模擬主庫創建數據文件，dg備庫空間不足時問題處理
本篇文檔測試目的：
模擬實際環境中，主庫對表空間添加數據文件，備庫空間不足，最終導致MRP進程自動斷開，處理方式。
1.問題環境模擬
1）正常情況下的dg 主庫創建數據文件，備庫接受日誌，自動創建表空間及數據文件。 RFS[49]: Selected log 4 for thread 1 sequence 115 dbid 699220720 branch 994543603 Fri Feb 22 23:20:36 2019 Media Recovery Log /u01/app/oracle/oradata/arch/1_112_994543603.arc Recovery created file /u01/app/oracle/oradata/adg1/test001.dbf Successfully added datafile 10 to media recovery Datafile #10: ‘/u01/app/oracle/oradata/adg1/test001.dbf‘ Media Recovery Log /u01/app/oracle/oradata/arch/1_113_994543603.arc Media Recovery Log /u01/app/oracle/oradata/arch/1_114_994543603.arc Media Recovery Waiting for thread 1 sequence 115 (in transit) Fri Feb 22 23:20:52 2019 RFS[49]: Selected log 5 for thread 1 sequence 116 dbid 699220720 branch 994543603 Fri Feb 22 23:20:52 2019 Archived Log entry 51 added for thread 1 sequence 115 ID 0x29aceaf0 dest 1: Fri Feb 22 23:20:52 2019 Media Recovery Log /u01/app/oracle/oradata/arch/1_115_994543603.arc Media Recovery Waiting for thread 1 sequence 116 (in transit) 2）異常情況
備庫文件系統
[oracle@11gtest arch]$ df -h
文件系統 容量 已用 可用 已用% 掛載點
/dev/mapper/VolGroup00-Lvroot
34G 22G 11G 69% / 空間剩余11G
主庫創建測試表空間，數據文件大小12G（測試環境ssd）
SQL> create tablespace test_tbs datafile ‘/home/oracle/test_tbs01.dbf‘ size 12g;
Tablespace created.
Elapsed: 00:01:50.86
SQL> alter system switch logfile
;
備庫Alert日誌
Fri Feb 22 23:28:52 2019
RFS[49]: Selected log 4 for thread 1 sequence 117 dbid 699220720 branch 994543603
Fri Feb 22 23:28:52 2019
Archived Log entry 52 added for thread 1 sequence 116 ID 0x29aceaf0 dest 1:
Fri Feb 22 23:28:52 2019
Media Recovery Log /u01/app/oracle/oradata/arch/1_116_994543603.arc
-------------------------對於DG備庫而言，最開始是接受日誌，MRP進程應用日誌，隨後空間確實不足後，停止創建數據文件
SQL>
select process,client_process,sequence#,status,BLOCK#,BLOCKS from v$managed_standby
PROCESS CLIENT_P SEQUENCE# STATUS BLOCK# BLOCKS
--------- -------- ---------- ------------ ---------- ----------
MRP0 N/A 116 APPLYING_LOG 3314 4568
[root@11gtest etc]# df -h
文件系統 容量 已用 可用 已用% 掛載點
/dev/mapper/VolGroup00-Lvroot
34G 33G 7.2M 100% /
[root@11gtest etc]# df -h 從11g可用空間，將為0，最後又還原為可用空間11g，都是自動進行。
文件系統 容量 已用 可用 已用% 掛載點
/dev/mapper/VolGroup00-Lvroot
34G 22G 11G 69% /
---Alert報錯
Fri Feb 22 23:28:52 2019
RFS[49]: Selected log 4 for thread 1 sequence 117 dbid 699220720 branch 994543603
Fri Feb 22 23:28:52 2019
Archived Log entry 52 added for thread 1 sequence 116 ID 0x29aceaf0 dest 1:
Fri Feb 22 23:28:52 2019
Media Recovery Log /u01/app/oracle/oradata/arch/1_116_994543603.arc
Fri Feb 22 23:29:51 2019
Errors in file /u01/app/oracle/diag/rdbms/adg1/adg1/trace/adg1_pr00_7496.trc:
ORA-19502: write error on file "/u01/app/oracle/oradata/adg1/test_tbs01.dbf", block number 1337472 (block size=8192)
ORA-27072: File I/O error
Additional information: 4
Additional information: 1337472
Additional information: 577536
File #11 added to control file as ‘UNNAMED00011‘.
Originally created as:
‘/home/oracle/test_tbs01.dbf‘
Recovery was unable to create the file as:
‘/u01/app/oracle/oradata/adg1/test_tbs01.dbf‘
Errors with log /u01/app/oracle/oradata/arch/1_116_994543603.arc
MRP0: Background Media Recovery terminated with error 1274
Errors in file /u01/app/oracle/diag/rdbms/adg1/adg1/trace/adg1_pr00_7496.trc:
ORA-01274: cannot add datafile ‘/home/oracle/test_tbs01.dbf‘ - fFri Feb 22 23:29:54 2019
MRP0: Background Media Recovery process shutdown (adg1)
MRP進程自動shutdown：
嘗試啟動Mrp進程
recover managed standby database disconnect from session;
Fri Feb 22 23:32:52 2019
ALTER DATABASE RECOVER managed standby database disconnect from session
Attempt to start background Managed Standby Recovery process (adg1)
Fri Feb 22 23:32:52 2019
MRP0 started with pid=20, OS id=8449
MRP0: Background Managed Standby Recovery process started (adg1)
started logmerger process
Fri Feb 22 23:32:57 2019
Managed Standby Recovery not using Real Time Apply
Fri Feb 22 23:32:57 2019
Errors in file /u01/app/oracle/diag/rdbms/adg1/adg1/trace/adg1_dbw0_7052.trc:
ORA-01186: file 11 failed verification tests
ORA-01157: cannot identify/lock data file 11 - see DBWR trace file
ORA-01111: name for data file 11 is unknown - rename to correct file
ORA-01110: data file 11: ‘/u01/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00011‘
File 11 not verified due to error ORA-01157
MRP0: Background Media Recovery terminated with error 1111
Errors in file /u01/app/oracle/diag/rdbms/adg1/adg1/trace/adg1_pr00_8453.trc:
ORA-01111: name for data file 11 is unknown - rename to correct file
ORA-01110: data file 11: ‘/u01/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00011‘
ORA-01157: cannot identify/lock data file 11 - see DBWR trace file
ORA-01111: name for data file 11 is unknown - rename to correct file
ORA-01110: data file 11: ‘/u01/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00011‘
Completed: ALTER DATABASE RECOVER managed standby database disconnect from session
Recovery Slave PR00 previously exited with exception 1111
MRP0: Background Media Recovery process shutdown (adg1)
2.問題處理
問題處理方法論，1.立即對空間擴容；
2.找到存在空間空間的路徑（磁盤組）先存放一陣子；
3.nfs等文件系統臨時掛載存放
本次模擬，使用第二種，找到存在的空閑空間
[root@11gtest software]# rm p13390677_112040_Linux-x86-64_* rm：是否刪除 一般文件 “p13390677_112040_Linux-x86-64_1of7.zip”? y rm：是否刪除 一般文件 “p13390677_112040_Linux-x86-64_2of7.zip”? y [root@11gtest software]# df -h 文件系統 容量 已用 可用 已用% 掛載點 /dev/mapper/VolGroup00-Lvroot 34G 20G 13G 61% /
SQL> select file#,name,status,bytes/1024/1024/1024 g from v$datafile;
FILE# NAME STATUS G
---------- ----------------------------------------------------------------- ------- ----------
1 /u01/app/oracle/oradata/adg1/system01.dbf SYSTEM 2.02148438
2 /u01/app/oracle/oradata/adg1/sysaux01.dbf ONLINE .60546875
3 /u01/app/oracle/oradata/adg1/undotbs01.dbf ONLINE .25390625
4 /u01/app/oracle/oradata/adg1/users01.dbf ONLINE .337646484
5 /u01/app/oracle/oradata/adg1/test01.dbf ONLINE .004150391
6 /u01/app/oracle/oradata/adg1/user02.dbf ONLINE .009765625
7 /u01/app/oracle/oradata/adg1/ogg.dbf ONLINE .009765625
8 /u01/app/oracle/oradata/adg1/test1.dbf ONLINE .059570313
9 /u01/app/oracle/oradata/adg1/test2.dbf ONLINE .010742188
10 /u01/app/oracle/oradata/adg1/test001.dbf ONLINE .000976563
11 /u01/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00011 RECOVER 0
11 rows selected.
新的數據文件在dba_data_files都不存在。
SQL> select file_name,file_id,tablespace_name,bytes/1024/1024/1024 g,status from dba_data_files;
FILE_NAME FILE_ID TABLESPACE_NAME G STATUS
--------------------------------------------- ---------- ------------------------------ ---------- ---------
/u01/app/oracle/oradata/adg1/users01.dbf 4 USERS .337646484 AVAILABLE
/u01/app/oracle/oradata/adg1/undotbs01.dbf 3 UNDOTBS1 .25390625 AVAILABLE
/u01/app/oracle/oradata/adg1/sysaux01.dbf 2 SYSAUX .60546875 AVAILABLE
/u01/app/oracle/oradata/adg1/system01.dbf 1 SYSTEM 2.02148438 AVAILABLE
/u01/app/oracle/oradata/adg1/test01.dbf 5 AUDITING .004150391 AVAILABLE
/u01/app/oracle/oradata/adg1/user02.dbf 6 USERS .009765625 AVAILABLE
/u01/app/oracle/oradata/adg1/ogg.dbf 7 OGG .009765625 AVAILABLE
/u01/app/oracle/oradata/adg1/test1.dbf 8 IMAGE_APP_TBS .059570313 AVAILABLE
/u01/app/oracle/oradata/adg1/test2.dbf 9 IMAGE_APP_IDX_TBS .010742188 AVAILABLE
/u01/app/oracle/oradata/adg1/test001.dbf 10 TEST001 .000976563 AVAILABLE
10 rows selected.
SQL> alter database create datafile ‘/u01/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00011‘ as ‘/home/oracle/test_temp00028‘;
alter database create datafile ‘/u01/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00011‘ as ‘/home/oracle/test_temp00028‘
*
ERROR at line 1:
ORA-01275: Operation CREATE DATAFILE is not allowed if standby file management is automatic.
SQL> alter system set standby_file_management=MANUAL;
SQL> alter database create datafile ‘/u01/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00011‘ as ‘/home/oracle/test_temp00028‘;
Database altered.
Fri Feb 22 23:50:35 2019
ALTER SYSTEM SET standby_file_management=‘MANUAL‘ SCOPE=BOTH;
Fri Feb 22 23:50:46 2019
alter database create datafile ‘/u01/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00011‘ as ‘/home/oracle/test_temp00028‘
Fri Feb 22 23:51:23 2019
Completed: alter database create datafile ‘/u01/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00011‘ as ‘/home/oracle/test_temp00028‘
備庫開啟MRP進程
SQL> recover managed standby database disconnect from session;
3.後續處理
移動文件後，原有空間足夠需要轉移，DG數據文件遷移
正確流程操作： 數據庫啟動到Mount階段 rman進行backup as copy方式拷貝數據文件 swich 修改控制文件數據文件目錄 open數據庫 開啟Mrp進程，恢復dg應用 rman刪除copy備份信息
SQL> startup force mount;
RMAN> backup as copy datafile 11 format ‘/u01/app/oracle/oradata/adg1/test_tbs01.dbf‘;
RMAN> switch datafile 11 to copy;
SQL> alter database open;alter database recover managed standby database cancel; recover managed standby database using current logfile disconnect;
recover managed standby database disconnect from session;
RMAN> list copy of database; RMAN> delete copy of datafile 11;
***************************************
SQL> startup force mount; RMAN> backup as copy datafile 11 format ‘/u01/app/oracle/oradata/adg1/test_tbs01.dbf‘; Starting backup at 22-FEB-19 using target database control file instead of recovery catalog allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=20 device type=DISK channel ORA_DISK_1: starting datafile copy input datafile file number=00010 name=/u01/app/oracle/oradata/adg1/test001.dbf output file name=/u01/app/oracle/oradata/adg1/test_tbs01.dbf tag=TAG20190222T235621 RECID=11 STAMP=1000943781 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01 Finished backup at 22-FEB-19 RMAN-00571: =========================================================== RMAN-03009: failure of backup command on ORA_DISK_1 channel at 02/23/2019 00:08:45 ORA-19502: write error on file "/u01/app/oracle/oradata/adg1/test_tbs01.dbf", block number 81536 (block size=8192) ORA-27072: File I/O error Additional information: 4 Additional information: 81536 Additional information: 53248 ORA-19502: write error on file "/u01/app/oracle/oradata/adg1/test_tbs01.dbf", block number 81536 (block size=8192) 空間不足，由於是測試環境，因此對datafile 11進行resize回收空間， SQL> select file_id,bytes/1024/1024/1024 from dba_data_files where file_id=11; FILE_ID BYTES/1024/1024/1024 ---------- -------------------- 11 12 Elapsed: 00:00:00.00 SQL> alter database datafile 11 resize 5g; Database altered. Elapsed: 00:00:01.40 SQL> select file_id,bytes/1024/1024/1024 from dba_data_files where file_id=11; FILE_ID BYTES/1024/1024/1024 ---------- -------------------- 11 5 SQL> alter system switch logfile ; 關庫，dg環境啟動到Mount狀態 SQL> recover managed standby database disconnect from session; Media recovery complete. SQL> select file_id,bytes/1024/1024/1024 from dba_data_files where file_id=11; FILE_ID BYTES/1024/1024/1024 ---------- -------------------- 11 5 繼續RMAN RMAN> backup as copy datafile 11 format ‘/u01/app/oracle/oradata/adg1/test_tbs01.dbf‘; Starting backup at 23-FEB-19 using target database control file instead of recovery catalog allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=20 device type=DISK channel ORA_DISK_1: starting datafile copy input datafile file number=00011 name=/home/oracle/test_temp00028 output file name=/u01/app/oracle/oradata/adg1/test_tbs01.dbf tag=TAG20190223T001911 RECID=12 STAMP=1000945177 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:35 Finished backup at 23-FEB-19 RMAN> switch datafile 11 to copy; RMAN-06572: database is open and datafile 11 is not offline
switch datafile 修改數據文件目錄操作，需要數據文件offline,數據庫需要啟動到mount階段。 RMAN> switch datafile 11 to copy; using target database control file instead of recovery catalog datafile 11 switched to datafile copy "/u01/app/oracle/oradata/adg1/test_tbs01.dbf" RMAN> list copy of database; RMAN> delete copy of datafile 11; SQL> alter database open; Database altered. SQL> r 1* select file_id,file_name,tablespace_name,bytes/1024/1024/1024 from dba_data_files FILE_ID FILE_NAME TABLESPACE_NAME BYTES/1024/1024/1024 ------- ------------------------------------------------------- ------------------------------ -------------------- 4 /u01/app/oracle/oradata/adg1/users01.dbf USERS .337646484 3 /u01/app/oracle/oradata/adg1/undotbs01.dbf UNDOTBS1 .25390625 2 /u01/app/oracle/oradata/adg1/sysaux01.dbf SYSAUX .60546875 1 /u01/app/oracle/oradata/adg1/system01.dbf SYSTEM 2.02148438 5 /u01/app/oracle/oradata/adg1/test01.dbf AUDITING .004150391 6 /u01/app/oracle/oradata/adg1/user02.dbf USERS .009765625 7 /u01/app/oracle/oradata/adg1/ogg.dbf OGG .009765625 8 /u01/app/oracle/oradata/adg1/test1.dbf IMAGE_APP_TBS .059570313 9 /u01/app/oracle/oradata/adg1/test2.dbf IMAGE_APP_IDX_TBS .010742188 10 /u01/app/oracle/oradata/adg1/test001.dbf TEST001 11 /u01/app/oracle/oradata/adg1/test_tbs01.dbf TEST_TBS 5 11 rows selected. alter tablespace TEST_TBS rename datafile ‘/home/oracle/test_temp00028‘ to ‘/u01/app/oracle/oradata/adg1/test001.dbf‘ * ERROR at line 1: ORA-16000: database open for read-only access SQL> alter database create datafile ‘/home/oracle/test_temp00028‘ as ‘/u01/app/oracle/oradata/adg1/test001.dbf‘; alter database create datafile ‘/home/oracle/test_temp00028‘ as ‘/u01/app/oracle/oradata/adg1/test001.dbf‘ * ERROR at line 1: ORA-01524: cannot create data file as ‘/u01/app/oracle/oradata/adg1/test001.dbf‘ - file already part of database
模擬主庫創建數據文件，dg備庫空間不足時問題處理