ORACLE GOLDEN GATE oracle同步數據至kafka
一.服務器信息
| ip | 軟件版本 | ogg版本 | 軟件包 | 操作系統版本 | OGG安裝路徑 | |
| 10.1.50.52 | 源 | oracle11.2.0.4 | 12.2.0.1.1 | V100692-01.zip | centos6.9 | /data/ogg |
| 10.0.71.102 | 目標 | kafka0.10 | 12.2.0.1.160823 | 123010_ggs_Adapters_Linux_x64.zip | centos7.3 | /data/ogg |
二.源端安裝OGG
1.解壓V100692-01.zip
unzip V100692-01.zip
這個時候會看到fbo_ggs_Linux_x64_shiphome文件夾
2.我們采用靜默安裝的方式進行安裝（註意要使用oracle用戶來進行安裝，oggcore.rsp配置需要根據實際需要進行修改，也可使用圖形界面進行安裝）
/usr/local/software/fbo_ggs_Linux_x64_shiphome/Disk1/runInstaller -silent -responseFile /usr/local/software/fbo_ggs_Linux_x64_shiphome/Disk1/response/oggcore.rsp
三.源端OGG配置
1.開啟主庫歸檔日誌、補充日誌及force logging
alter database add supplemental log data;
alter database force logging;
alter system set enable_goldengate_replication=true scope=both;
2.創建ogg的管理用戶
create user ogg identified by password account unlock;
grant connect,resource to ogg;
grant select any dictionary to ogg;
grant select any table to ogg;
grant execute on utl_file to ogg;
grant restricted session to ogg;
grant create table,create sequence to ogg;
alter table DCSDBA.YH_ITF_INVENTORY_TRANSACTION add supplemental log data (all) columns;--如果kafka需要接收到完整的修改信息就需要開啟補全日誌
3.登錄ogg並創建目錄
/data/ogg/ggsci
create subdirs
4.配置源端manager進程
edit params mgr
port 1357
autostart er *
autorestart er *
5.啟動manager並確認狀態
start mgr
info all 查看MANAGER進程為running狀態
6.配置加密
ENCRYPT PASSWORD password ENCRYPTKEY DEFAULT
7.配置數據抽取進程
①加一個extract，名字隨便取
add extract hnwmsprd,tranlog,begin now
②為這個ext指定他把遠程文件放到遠程機的哪個目錄
ADD RMTTRAIL /data/ogg/dirdat/tt, EXTRACT hnwmsprd
③編輯配置文件
edit params hnwmsprd
extract hnwmsprd
setenv (NLS_LANG="AMERICAN_AMERICA.UTF8")
setenv (ORACLE_SID="hnwmsprd")
userid ogg,password AACAAAAAAAAAAAKALILFJIXIECMGKFAHRBJHGDGBNJLBOAUC,ENCRYPTKEY DEFAULT
rmthost 10.0.71.102, mgrport 1357
rmttrail /data/ogg/dirdat/tt
discardfile /data/ogg/dirrpt/trail.dsc,append,megabytes 100
gettruncates
GETUPDATEBEFORES
NOCOMPRESSDELETES
NOCOMPRESSUPDATES
TRANLOGOPTIONS MINEFROMACTIVEDG
table DCSDBA.YH_ITF_INVENTORY_TRANSACTION;
④啟動進程
start hnwmsprd
四.OGG目標端搭建
目標端搭建只需解壓軟件即可
unzip 123010_ggs_Adapters_Linux_x64.zip
解壓出來ggs_Adapters_Linux_x64.tar文件
將文件移動到/data/ogg目錄下解壓
mv ggs_Adapters_Linux_x64.tar /data/ogg tar -xvf ggs_Adapters_Linux_x64.tar
五.OGG目標端配置
1.登錄ogg並創建目錄
/data/ogg/ggsci
create subdirs
2.配置源端manager進程
edit params mgr
port 1357
autostart er *
autorestart er *
3.啟動manager並確認狀態
start mgr
info all 查看MANAGER進程為running狀態
4.修改復制進程配置
edit params rkafka
REPLICAT rkafka
TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.props
REPORTCOUNT EVERY 1 MINUTES, RATE
GROUPTRANSOPS 10000
MAP DCSDBA.YH_ITF_INVENTORY_TRANSACTION, TARGET DCSDBA.YH_ITF_INVENTORY_TRANSACTION;
5.修改kafka.props（註意：kafka.props要放在/data/ogg/dirprm路徑下）
gg.handlerlist = kafkahandler
gg.handler.kafkahandler.type = kafka
gg.handler.kafkahandler.KafkaProducerConfigFile=custom_kafka_producer.properties
gg.handler.kafkahandler.TopicName =hnwmsprd
gg.handler.kafkahandler.format =json
gg.handler.kafkahandler.SchemaTopicName=ogg
gg.handler.kafkahandler.BlockingSend =true
gg.handler.kafkahandler.includeTokens=false
gg.handler.kafkahandler.mode =tx
goldengate.userexit.timestamp=utc
goldengate.userexit.writers=javawriter
javawriter.stats.display=TRUE
javawriter.stats.full=TRUE
gg.log=log4j
gg.log.level=INFO
gg.report.time=30sec
gg.classpath=dirprm/:/usr/local/kafka/libs/*:
javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar
以上配置文件根據實際配置進行修改
6.編輯custom_kafka_producer.properties文件（註意：custom_kafka_producer.properties要放在/data/ogg/dirprm路徑下）
#bootstrap.servers=host:port
bootstrap.servers=10.0.71.102:9092
acks=1
#compression.type=gzip
reconnect.backoff.ms=1000
value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
# 100KB per partition
batch.size=102400
linger.ms=10000
7.啟動復制進程
start rkafka
六.驗證
目標端執行
./kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic hnwmsprd
源端更新DCSDBA.YH_ITF_INVENTORY_TRANSACTION表。
ORACLE GOLDEN GATE oracle同步數據至kafka