環境：Linux + Oracle 11.2.0.1 ADG
現象：發現備庫沒有應用日誌

1. 數據庫查詢備庫目前狀態
發現備庫目前沒有應用日誌，apply lag已經顯示備庫有3天21小時多沒有應用日誌。

SQL> set linesize 1200
SQL> SELECT OPEN_MODE, DATABASE_ROLE, SWITCHOVER_STATUS, FORCE_LOGGING, DATAGUARD_BROKER, GUARD_STATUS FROM V$DATABASE; 

OPEN_MODE            DATABASE_ROLE    SWITCHOVER_STATUS    FOR DATAGUAR GUARD_S
-------------------- ---------------- -------------------- --- -------- -------
READ ONLY            PHYSICAL STANDBY NOT ALLOWED          YES DISABLED NONE

SQL> select * from v$dataguard_stats;

NAME                             VALUE                                                            UNIT                           TIME_COMPUTED                  DATUM_TIME
-------------------------------- ---------------------------------------------------------------- ------------------------------ ------------------------------ ------------------------------
transport lag                    +00 00:00:00                                                     day(2) to second(0) interval   01/17/2017 16:07:12            01/17/2017 16:07:12
apply lag                        +03 21:34:49                                                     day(2) to second(0) interval   01/17/2017 16:07:12            01/17/2017 16:07:12
apply finish time                +00 03:10:34.000                                                 day(2) to second(3) interval   01/17/2017 16:07:12
estimated startup time           15                                                               second                         01/17/2017 16:07:12
 

2. 查詢alert告警日誌
從alert告警日誌中定位到ADG出現問題的時刻，有600報錯信息，進而導致MRP進程終止，詳細日誌如下：

Fri Jan 13 18:32:25 2017
Errors in file /home/oracle/opt/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr03_22555.trc  (incident=67480):
ORA-00600: internal error code, arguments: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /home/oracle/opt/oracle/diag/rdbms/orcl/orcl/incident/incdir_67480/orcl_pr03_22555_i67480.trc
Slave exiting with ORA-600 exception
Errors in file /home/oracle/opt/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr03_22555.trc:
ORA-00600: internal error code, arguments: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Fri Jan 13 18:32:26 2017
Errors in file /home/oracle/opt/oracle/diag/rdbms/orcl/orcl/trace/orcl_mrp0_22547.trc  (incident=67448):
ORA-00600: internal error code, arguments: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /home/oracle/opt/oracle/diag/rdbms/orcl/orcl/incident/incdir_67448/orcl_mrp0_22547_i67448.trc
Fri Jan 13 18:32:26 2017
Trace dumping is performing id=[cdmp_20170113183226]
Recovery Slave PR03 previously exited with exception 600
Fri Jan 13 18:32:27 2017
MRP0: Background Media Recovery terminated with error 448
Errors in file /home/oracle/opt/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr00_22549.trc:
ORA-00448: normal completion of background process
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Fri Jan 13 18:32:27 2017
Sweep [inc][67480]: completed
Sweep [inc][67480]: completed
Recovered data files to a consistent state at change 2010287982
Errors in file /home/oracle/opt/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr00_22549.trc:
ORA-00448: normal completion of background process
Errors in file /home/oracle/opt/oracle/diag/rdbms/orcl/orcl/trace/orcl_mrp0_22547.trc:
ORA-00600: internal error code, arguments: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
MRP0: Background Media Recovery process shutdown (orcl)
Sweep [inc][67448]: completed
Sweep [inc2][67480]: completed
Sweep [inc2][67448]: completed
Trace dumping is performing id=[cdmp_20170113183227]
Fri Jan 13 18:33:04 2017
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
 

3. 嘗試手工啟動備庫MRP恢復進程
發現手工啟動備庫MRP恢復進程，告警日誌中依然會報出相同ORA-600 [kcbr_apply_change_11]錯誤。

4. 嘗試mount狀態啟動MRP恢復進程
發現在mount狀態下，可以正常啟動MRP恢復進程，等恢復完成後，重新開啟ADG實時應用，一切正常。

shutdown immediate
startup mount
alter database recover managed standby database disconnect from session;
此時等待恢復完成...
alter database recover managed standby database cancel;
alter database open;
alter database recover managed standby database using current logfile disconnect from session;
 

查詢備庫狀態確認一切正常：

SQL> SELECT OPEN_MODE, DATABASE_ROLE, SWITCHOVER_STATUS, FORCE_LOGGING, DATAGUARD_BROKER, GUARD_STATUS FROM V$DATABASE; 

OPEN_MODE            DATABASE_ROLE    SWITCHOVER_STATUS    FOR DATAGUAR GUARD_S
-------------------- ---------------- -------------------- --- -------- -------
READ ONLY WITH APPLY PHYSICAL STANDBY NOT ALLOWED          YES DISABLED NONE

SQL> select * from v$dataguard_stats;

NAME                             VALUE                                                            UNIT                           TIME_COMPUTED                  DATUM_TIME
-------------------------------- ---------------------------------------------------------------- ------------------------------ ------------------------------ ------------------------------
transport lag                    +00 00:00:00                                                     day(2) to second(0) interval   01/17/2017 17:42:26            01/17/2017 17:42:26
apply lag                        +00 00:00:00                                                     day(2) to second(0) interval   01/17/2017 17:42:26            01/17/2017 17:42:26
apply finish time                +00 00:00:00.000                                                 day(2) to second(3) interval   01/17/2017 17:42:26
estimated startup time           18                                                               second                         01/17/2017 17:42:26

5. 查詢MOS，定位根本原因
查詢MOS發現該現象與bug 10419984 相匹配
Bug 10419984 : ACTIVE DATA GUARD STANDBY GIVES ORA-600 [KCBR_APPLY_CHANGE_11]
建議應用該補丁防止該問題再次被觸發。

Oracle 11.2.0.1 ADG環境MRP進程遭遇ORA