(Step 10) Install Dependencies (This is Required to Create Replicated in Parquet)
https://docs.oracle.com/en/middleware/goldengate/big-data/23/gadbd/qs-realtimeparquet-ingestion-azure-data-lake-storage-goldengate-daa.html#GUID-75E81FD0-0DF4418A-9643-8F63243DB74C
[oracle@******* DependencyDownloader]$ which java
/usr/bin/java
[oracle@******* DependencyDownloader]$ pwd
/u01/app/oracle/product/GG/23_az/opt/DependencyDownloader
[oracle@******* DependencyDownloader]$
./parquet.sh 1.15.0 || https://search.maven.org/artifact/org.apache.parquet/parquethadoop
./hadoop.sh 3.4.1 || https://search.maven.org/artifact/org.apache.hadoop/hadoop-azure ./azure_blob_storage.sh 12.29.0 || https://search.maven.org/artifact/com.azure/azurestorage-blob
Copy the downloaded files from above to /u01/app/oracle/product/GG/oggjar/
(Step 11) Configure Replicate Process to Push Data to Azure Blob Storage
Properties files:
#The File Writer Handler – no need to change
gg.handlerlist=filewriter
gg.handler.filewriter.type=filewriter
gg.handler.filewriter.mode=op
gg.handler.filewriter.pathMappingTemplate=./dirout
gg.handler.filewriter.stateFileDirectory=./dirsta
gg.handler.filewriter.fileRollInterval=7m
gg.handler.filewriter.inactivityRollInterval=5s
gg.handler.filewriter.fileWriteActiveSuffix=.tmp
gg.handler.filewriter.finalizeAction=delete
### Avro OCF – no need to change
gg.handler.filewriter.format=avro_row_ocf
gg.handler.filewriter.fileNameMappingTemplate=${groupName}_${fullyQualifiedTableName}_${currentTimestamp}.avro
gg.handler.filewriter.format.pkUpdateHandling=delete-insert
gg.handler.filewriter.format.metaColumnsTemplate=${optype},${position}
gg.handler.filewriter.format.iso8601Format=false
gg.handler.filewriter.partitionByTable=true
gg.handler.filewriter.rollOnShutdown=true
#The Parquet Event Handler – no need to change
gg.handler.filewriter.eventHandler=parquet
gg.eventhandler.parquet.type=parquet
gg.eventhandler.parquet.pathMappingTemplate=./dirparquet
gg.eventhandler.parquet.fileNameMappingTemplate=${groupName}_${fullyQualifiedTableName}_${currentTimestamp}.parquet
gg.eventhandler.parquet.writeToHDFS=false
gg.eventhandler.parquet.finalizeAction=delete
#TODO Select ABS Event Handler – no need to change
gg.eventhandler.parquet.eventHandler=abs
#TODO Set ABS Event Handler - please update as needed
gg.eventhandler.abs.type=abs
gg.eventhandler.abs.bucketMappingTemplate=******
gg.eventhandler.abs.pathMappingTemplate=ogg/data/fullload/${fullyQualifiedTableName}
gg.eventhandler.abs.accountName=*********
#TODO: Edit the Azure storage account key if access key is used
gg.eventhandler.abs.accountKey=************
#TODO: Edit the Azure shared access signature (SAS) to if SAS is used.
#gg.eventhandler.abs.sasToken=<sas-token>
#TODO: Edit the tenant ID, Client ID and Secret of the application if LDAP is used.
#gg.eventhandler.abs.tenantId=<azure-tenant-id>
#gg.eventhandler.abs.clientId=<azure-client-id>
#gg.eventhandler.abs.clientSecret=<azure-client-secret>
#TODO Set the classpath to the paths you noted in step1
gg.classpath=/u01/app/oracle/product/GG/oggjar/*
jvm.bootoptions=-Xmx512m -Xms32m
Want to connect with the author or explore other database resources? Visit Datavail.com for more.