Oracle RAC是Oracle Real Application Cluster的簡寫，官方中文文件一般翻譯為“真正應用叢集”，它一般有兩臺或者兩臺以上同構計算機及共享儲存裝置構成，可提供強大的資料庫處理能力，現在是Oracle 10g Grid應用的重要組成部分。RAC（Race Game）指競速類遊戲。--百度。

一些圖片在很久之前就截了下載，最近在虛擬機器裡面跑一下測試，現在再整理一下，釋出上來，圖片很多。在整理的過程中發現，自己單獨做一篇都是簡單，但是記錄成文件，再發布上來真的就不容易了，感覺像一下子做了十次一樣，每個錯誤都想記錄下來，但是又很多細節無法一一詳記。希望大家能知道老夫的不容易啊，呵呵。有一些記錄的不好的話大家多多見諒，互相學習啊。

部分測試環境

硬體：兩個虛擬機器記憶體2g，swap，結果節點一出現宕機，兩個節點記憶體差不多佔滿，建議記憶體3g+。

系統：rhel5.5,oracle linux5U8,centos6.6，oracle linux 6.7，結果，除了redhat5.x以及oracle linux5 比較好之外，其他版本核心和模組原因不能繼續。

...

...

網路規劃
rac1
eth0 10.1.2.140 rac1
10.1.2.141 rac1-vip
eth1 192.168.0.140 rac1-priv

rac2
eth0 10.1.2.142 rac2
10.1.2.143 rac2-vip
eth1 192.168.0.142 rac2-priv

/etc/hosts
#eth0 public
10.1.2.140 rac1.kong.com rac1
10.1.2.142 rac2.kong.com rac2
#virtual
10.1.2.141 rac1-vip.kong.com rac1-vip
10.1.2.143 rac2-vip.kong.com rac2-vip
#eth1 private
192.168.0.140 rac1-priv.kong.com rac1-priv
192.168.0.142 rac2-priv.kong.com rac2-priv
#scan
10.1.2.144 rac-scan.kong.com rac-scan


儲存
10g-->分三分割槽,i

勾選立即分配，單個檔案儲存提高效能


高階修改->scsi通道改為1：X，獨立永久。


節點2新增現有磁碟 修改和節點1相同配置

修改虛擬機器檔案“虛擬機器名稱.vmx”檔案
#shared disks configure
disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.dataCacheMinReadAheadSize = "0"
diskLib.maxUnsyncedWrites = "0"
disk.EnableUUID = "TRUE"  #否則UUID出不來
scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1.sharedBus = "VIRTUAL"
scsi1:0.present = "TRUE"
scsi1:0.mode = "independent-persistent"
scsi1:0.fileName = "E:\share\ocr_vote.vmdk"
scsi1:0.deviceType = "disk"
scsi1:0.redo = ""

－－－－－－－－－－－－－
scsi1.virtualDev = "lsilogic"
scsi1.present = "TRUE"
scsi1.sharedBus = "VIRTUAL"
scsi1:0.present = "TRUE"
scsi1:0.fileName = "D:\Program Files (x86)\VMware\Shared Virtual Mchine\ShareStore\vmShareStore.vmdk"
scsi1:0.mode = "independent-persistent"
scsi1:0.deviceType = "disk"
scsi0:1.present = "FALSE"
floppy0.present = "FALSE"

disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.dataCacheMinReadAheadSize = "0"
diskLib.maxUnsyncedWrites = "0"
disk.EnableUUID = "TRUE"

－－－－－－－－－－－－－
現在rac1新增共享儲存，再手動rac2新增共享儲存，再新增上面的程式碼。

依賴關係
使用指令碼批量安裝
vi yumget.sh
#!bin/bash
#start install required packages
echo "start installing required packages"
yum -y install binutils-*
yum -y install compat-libstdc++-*
yum -y install compat-libstdc++-*
yum -y install elfutils-libelf-*
yum -y install elfutils-libelf-devel-*
yum -y install gcc-*
yum -y install gcc-c++-*
yum -y install glibc-*
yum -y install glibc-*
yum -y install glibc-common-*
yum -y install glibc-devel-*
yum -y install glibc-devel-*
yum -y install glibc-headers-*
yum -y install ksh-*
yum -y install libaio-*
yum -y install libaio-*
yum -y install libaio-devel-*
yum -y install libaio-devel-*
yum -y install libgcc-*
yum -y install libgcc-*
yum -y install libstdc++-*
yum -y install libstdc++-*
yum -y install libstdc++-devel*
yum -y install make-*
yum -y install sysstat-*
yum -y install unixODBC-*
執行sh yumget.sh

勾選客機時間與主機同步相當於ntp，前提

安裝工作（兩個節點同時進行）
1.修改主機名
vi /etc/sysconfig/network
HOSTNAME=rac1.xl.com
2.建立使用者及相關
使用者規劃
grid：
ORACLE_HOME =/oracle/app/grid_home
ORACLE_BASE=/oracle/app/grid

oracle：
ORACLE_BASE=/oracle/app/oracle
ORACLE_HOME=/oracle/app/oracle/db
使用者建立
/usr/sbin/groupadd -g 505 oinstall
/usr/sbin/groupadd -g 502 dba
/usr/sbin/groupadd -g 503 oper
/usr/sbin/groupadd -g 504 asmadmin
/usr/sbin/groupadd -g 506 asmdba
/usr/sbin/groupadd -g 507 asmoper
/usr/sbin/useradd -u 505 -g oinstall -G asmadmin,dba,asmdba,asmoper grid
/usr/sbin/useradd -u 506 -g oinstall -G dba,asmdba,oper oracle
使用者密碼
echo -n 123456|passwd --stdin grid
echo -n 123456|passwd --stdin oracle
本地檔案
mkdir -p /oracle/app/grid
chown grid.oinstall /oracle/app/grid
chmod 775 /oracle/app/grid

mkdir -p /oracle/app/grid_home
chown grid.oinstall /oracle/app/grid_home
chmod 775 /oracle/app/grid_home

mkdir -p /oracle/app/oracle
chown oracle.oinstall /oracle/app/oracle
chmod 775 /oracle/app/oracle

mkdir -p /oracle/app/oracle/db
chown oracle.oinstall /oracle/app/oracle/db
chmod 775 /oracle/app/oracle/db

chown grid.oinstall /oracle/app
chmod 775 /oracle/app

3.修改系統，使用者引數
cp /etc/sysctl.conf /etc/sysctl.conf_bak
vi /etc/sysctl.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 4294967295
kernel.shmall = 268435456
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586

sysctl -p

vi /etc/security/limits.conf
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536

4.配置使用者環境變數
4.1 grid使用者環境變數，節點不同ORACLE_SID->+ASM1+1
ORACLE_BASE=/oracle/app/grid; export ORACLE_BASE
ORACLE_HOME=/oracle/app/grid_home; export ORACLE_HOME
ORACLE_SID=+ASM1; export ORACLE_SID
PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin

export PATH

4.2 oracle使用者環境變數，節點不同ORACLE_SID->rac1+1
ORACLE_BASE=/oracle/app/oracle; export ORACLE_BASE
ORACLE_HOME=/oracle/app/oracle/db; export ORACLE_HOME
ORACLE_SID=rac1; export ORACLE_SID
PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin

export PATH

5. 使用者等效性
略過，後面會自動完成

6.關閉防火牆senlinux
臨時關閉
service iptables stop;
setenforce 0;
永久關閉
chkconfig iptables off;
vi /etc/selinux/config
SELINUX=disabled

7.安裝ASM驅動
安裝網站“http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html”
如果是redhat5的需要下載三個檔案 區分32與64位
• oracleasm-2.6.18-194.el5xen-2.0.5-1.el5.i686.rpm
• oracleasm-2.6.18-194.el5debug-2.0.5-1.el5.i686.rpm
• oracleasm-2.6.18-194.el5PAE-2.0.5-1.el5.i686.rpm
• oracleasm-2.6.18-194.el5-debuginfo-2.0.5-1.el5.i686.rpm
• oracleasm-2.6.18-194.el5-2.0.5-1.el5.i686.rpm
如果是redhat6的只有64位的lib下載（或許可以下載oracle linux6 的32位）
• oracleasm-support-2.1.8-1.el6.x86_64.rpm
• oracleasmlib-2.0.4-1.el6.x86_64.rpm
如果是oracle linux6的有32與64位
• oracleasm-support-2.1.8-1.el6.x86_64.rpm
• oracleasmlib-2.0.4-1.el6.i686.rpm
如果是oracle linux7（redhat7以上只有64位系統）
• oracleasm-support-2.1.8-1.el6.x86_64.rpm
• oracleasmlib-2.0.8-2.el7.x86_64.rpm
[[email protected] note]# uname -a
Linux rac1.kong.com 2.6.18-194.el5PAE #1 SMP Tue Mar 16 22:00:21 EDT 2010 i686 i686 i386 GNU/Linux
系統是有2.6.18-194.el5PAE 有PAE的所以要注意
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release (如果報驗證問題時執行）
[[email protected] note]# rpm -ivh oracleasm-support-2.1.8-1.el5.i386.rpm
warning: oracleasm-support-2.1.8-1.el5.i386.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing... ########################################### [100%]
1:oracleasm-support ########################################### [100%]
[[email protected] note]# rpm -ivh oracleasm-2.6.18-194.el5PAE-2.0.5-1.el5.i686.rpm
warning: oracleasm-2.6.18-194.el5PAE-2.0.5-1.el5.i686.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing... ########################################### [100%]
1:oracleasm-2.6.18-194.el########################################### [100%]
[[email protected] note]# rpm -ivh oracleasmlib-2.0.4-1.el5.i386.rpm
warning: oracleasmlib-2.0.4-1.el5.i386.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing... ########################################### [100%]
1:oracleasmlib ########################################### [100%]



7.1配置asm
A
[[email protected] note]# oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
B
[[email protected] note]# oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Configuring "oracleasm" to use device physical block size
Mounting ASMlib driver filesystem: /dev/oracleasm

8.分割槽（在節點1做）
[[email protected] note]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xadbadc79.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').

Command (m for help): p

Disk /dev/sdb: 16.1 GB, 16106127360 bytes
255 heads, 63 sectors/track, 1958 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xadbadc79

Device Boot Start End Blocks Id System

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1958, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1958, default 1958): +3G

Command (m for help): p

Disk /dev/sdb: 16.1 GB, 16106127360 bytes
255 heads, 63 sectors/track, 1958 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xadbadc79

Device Boot Start End Blocks Id System
/dev/sdb1 1 393 3156741 83 Linux

Command (m for help): n
Command action
e extended
p primary partition (1-4)
2
Invalid partition number for type `2'
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (394-1958, default 394):
Using default value 394
Last cylinder, +cylinders or +size{K,M,G} (394-1958, default 1958): +7G

Command (m for help): p

Disk /dev/sdb: 16.1 GB, 16106127360 bytes
255 heads, 63 sectors/track, 1958 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xadbadc79

Device Boot Start End Blocks Id System
/dev/sdb1 1 393 3156741 83 Linux
/dev/sdb2 394 1308 7349737+ 83 Linux

Command (m for help): w
基本是3g，7g兩個分割槽

8.1節點一建立並發現ASM分割槽
[[email protected] note]# oracleasm createdisk OCR_VOTE /dev/sdb1
Writing disk header: done
Instantiating disk: done
[[email protected] note]# oracleasm createdisk DATA /dev/sdb2
Writing disk header: done
Instantiating disk: done
[[email protected] note]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[[email protected] note]# oracleasm listdisks
DATA
OCR_VOTE
[[email protected] note]#

節點二隻需要發現ASM分割槽
[[email protected] note]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "OCR_VOTE"
Instantiating disk "DATA"
[[email protected] note]# oracleasm listdisks
DATA
OCR_VOTE
[[email protected] note]# sync

9安裝grid（單節點執行）
切換grid使用者
執行runInstall

修改SCAN NAME為你hosts檔案裡面的對應的scan名

新增節點，大家可以看到hostname和virtual ip name在hosts檔案是對應啊

其他的能預設的都預設








解決依賴關係


執行yumget.sh之後還有不能安裝的包就可以忽略有些64位不能識別

安裝ing

在65%的時候，rac2上面會自動出現一個grid使用者的ractrans 傳輸程序，如果沒有就檢查iptables和selinux有沒有關閉。


在這裡就用root使用者在兩個節點上面執行這兩個指令碼，執行完就點ok。
最後回報一個“[INS-20802] Oracle Cluster Vertificate utility...."的錯，不用管點選ok即可，點next，完成安裝
安裝完成後使用crs_stat -t去檢視資源狀態，發現gsd/oc4j 都是在offline狀態！ 這個是11.2.0.1的預設現象
gsd用來支援oem等互動式工具，oc4j是oracle應用服務的一些元件，如biee
[[email protected] ~]$ /oracle/app/grid_home/bin/crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type ONLINE ONLINE rac2
ora.asm ora.asm.type ONLINE ONLINE rac1
ora.eons ora.eons.type ONLINE ONLINE rac1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type OFFLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type ONLINE ONLINE rac1
ora....SM1.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application OFFLINE OFFLINE
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type ONLINE ONLINE rac2
ora....ry.acfs ora....fs.type ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type ONLINE ONLINE rac2
ping 節點的vip看看能不能ping通

這裡vm可以先打個點做個快照，但是由於vm的共享儲存叢集是用獨立\立即\寫入的，所以不受快照影響，寫快照要想關機。
關機要先關crs
關閉crs
[[email protected] ~]# /oracle/app/grid_home/bin/crs_stop -all 或
[[email protected] ~]# /oracle/app/grid_home/bin/crsctl stop crs
開啟crs
[[email protected] ~]# /oracle/app/grid_home/bin/crs_start -all 或
[[email protected] ~]# /oracle/app/grid_home/bin/crsctl start crs

oracle rac 預設會開機自啟動，如需維護時可使用以下命令：
關閉：
crsctl stop cluster 停止本節點叢集服務
crsctl stop cluster –all 停止所有節點服務
開啟
crsctl start cluster 開啟本節點叢集服務
crsctl stop cluster –all 開啟所有節點服務
注：以上命令需以 root 使用者執行
9.2 錯誤
執行root.sh報錯，是在紅帽6的一個bug
Adding daemon to inittab
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
ohasd failed to start: Inappropriate ioctl for device
ohasd failed to start at /oracle/app/grid_home/crs/install/rootcrs.pl line 443.
解除安裝配置
[[email protected] Desktop]# /oracle/app/grid_home/crs/install/roothas.pl -delete -force
2015-08-04 01:23:49: Checking for super user privileges
2015-08-04 01:23:49: User has super user privileges
2015-08-04 01:23:49: Parsing the host name
Using configuration parameter file: /oracle/app/grid_home/crs/install/crsconfig_params
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Delete failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
/oracle/app/grid_home/bin/acfsdriverstate: line 51: /lib/acfstoolsdriver.sh: No such file or directory
/oracle/app/grid_home/bin/acfsdriverstate: line 51: exec: /lib/acfstoolsdriver.sh: cannot execute: No such file or directory
Successfully deconfigured Oracle Restart stack

chmod a+wr /var/tmp/.oracle/npohasd
[[email protected] install]# /bin/dd if=/var/tmp/.oracle/npohasd of=/dev/null bs=1 count=1024
/oracle/app/grid_home/crs/install/roothas.pl -deconfig -force -verbose 


ADVM/ACFS is not supported on redhat-release-server-6Server-6.7.0.3.0.1.el6.x86_64

[[email protected] install]# more /tmp/.linux_release
redhat-release-server-6Server-6.7.0.3.0.1.el6.x86_64
[[email protected] install]# echo "redhat-release-5Server-5" > /tmp/.linux_release
[[email protected] install]# more /tmp/.linux_release
redhat-release-5Server-5
[[email protected] install]# /usr/bin/chattr +i /tmp/.linux_release
[[email protected] install]#

acfsroot: ACFS-9301: ADVM/ACFS installation can not proceed:

acfsroot: ACFS-9302: No installation files found at /oracle/app/grid_home/install/usm/EL5/x86_64/2.6.18-8/2.6.18-8.x86_64-x86_64/bin.

[email protected] bin]# gedit /oracle/app/grid_home/lib/osds_acfslib.pm
1. [[email protected] lib]# vi osds_acfslib.pm  
2.   if ((defined($release)) &&                     # Redhat or OEL if defined 
3.       (($release =~ /^redhat-release/) ||        # straight RH 
4.        ($release =~ /^enterprise-release/) ||    # Oracle Enterprise Linux 
5.        ($release =~ /^oraclelinux-release/)))    # Oracle Linux 
6.   { 
7.  
8. 將上面的程式碼片段修改如下: 
9.  if ((defined($release)) &&                     # Redhat or OEL if defined 
10.       (($release =~ /^redhat-release/) ||        # straight RH 
11.        ($release =~ /^enterprise-release/) ||    # Oracle Enterprise Linux 
12.        ($release =~ /^centos-release/) ||        # CentOS hack 
13.        ($release =~ /^oraclelinux-release/)))    # Oracle Linux 
14.   { 




Oracle 11g R2+RAC+ASM+redhat安裝詳解2
解除安裝Oracle 11g R2 RAC