Real Time Replication / Migrations from MongoDB to Oracle Autonomous Database - Part 4

No Downtime Migrations from MongoDB to Autonomous JSON Database using Oracle GoldenGate

Disclaimer:

The information provided in this article is for general informational and educational purposes only. All content, including code snippets, configurations, and technical recommendations, is shared in good faith and reflects the author's personal understanding and experience at the time of writing.

  All views expressed are the author’s personal opinions and it is not affiliated with, endorsed by Oracle Corporation or any other entity. My current employer or previous employer nowhere responsible for this content. Readers are advised to perform their own research and testing before applying any of the concepts discussed.

Preface:

I had an opportunity to migrate collections from MongoDB (On-premises) to Autonomous Database using Oracle GoldenGate Microservices.

This technical article details the steps I took to migrate collections from MongoDB On-Prem to Oracle ADB (JSON) for your reference.

Autonomous JSON Database

• Is designed for JSON-centric development at low cost. Developers can use both the API for MongoDB and Oracle SQL for accessing the same document data, avoiding the need to move data to a separate database for analytics, machine learning, spatial analysis, and more.

MongoDB

• Is open-source document database, also known as a NoSQL database. It's designed to store and manage data in a flexible, document-oriented format rather than traditional tables. This means data is stored as JSON-like documents within collections.

  
  

Oracle GoldenGate

• Is a comprehensive software package for real-time data integration and replication. GoldenGate for Big Data can be configured for no down-time migrations from MongoDB to Autonomous JSON Database

• GoldenGate license for Database Migration available as Free for 183 days from day of deployment ..

Binaries / Softwares:

• OGG MicroService v21c from Oracle Market Place

• MongoDB running on Oracle Compute VM

• Oracle Autonomous JSON Database v19

1. Deployment – OGG MicroService (OGGMS)

GoldenGate license for Database Migration available as Free for 183 days from day of deployment.

Refer:  Oracle GoldenGate from Oracle Cloud Marketplace document for assistance

Create Stack

Follow the Stack creation wizard.

Once OGGMS is deployed successfully,  it will be showing as compute instance under your compartment.

2. Pre-requisite

   
   

   2.1 Ingress: Enable port 443 @ OCI

   • To access OGG microservice console, enable port 443 in ingress rule @ OCI

     
     

   2.2 Install MongoDB and ADB Dependency jar files in OGG MS Server

   • GoldenGate for Big Data uses client dependency jars for the various supported Oracle GoldenGate for Big Data integrations.

   • To download dependency jars, we can use “dependency downloader” which is a set of utility scripts that have been added to download 3rd party dependency libraries for various integrations.

   • Dependency downloader is located

     $OGG_HOME/opt/DependencyDownloader.

   • We need to execute mongodb_capture.sh for MongoDB capture dependencies and mongodb.sh for Autonomous JSON Database dependencies.

     
     

     mongodb.sh will download MongoDB Java drivers needed for Autonomous JSON replication. We can refer to MongoDB Java Driver document to see the list of driver versions. Oracle GoldenGate requires that we use the

     3.12.8 MongoDB Java Driver or higher.

     
     

     mongodb_capture.sh will download the drivers needed for capturing from MongoDB. We can refer to repository for versions that we can use.

     
     

In my environment, DependencyDownloader existed in following folder

cd /u01/app/ogg/opt/DependencyDownloader

For this use-case, I used MongoDB driver version for 4.4.1

Ref: https://mongodb.github.io/mongo-java-driver/

As opc user, I have installed 4.4.1 MongoDB drivers

-bash-4.2$

./mongodb.sh 4.4.1 <<< Provide MongoDB driver version

./mongodb_capture.sh 4.4.1

After installation i found the dependencies under below folder

/u01/app/ogg/opt/DependencyDownloader/dependencies

-bash-4.2$ ls -lrt

total 8

drwxrwxr-x. 2 opc opc 4096 Nov 16 06:51 mongodb_4.4.1

drwxrwxr-x. 2 opc opc 4096 Nov 16 06:55 mongodb_capture_4.4.1

-bash-4.2$

Make note of this Path, that required to be used to provided in properties file of OGG Replicat

/u01/app/ogg/opt/DependencyDownloader/dependencies/mongodb_4.4.1

/u01/app/ogg/opt/DependencyDownloader/dependencies/mongodb_capture_4.4.1

3 Deployment : Autonomous ( JSON ) Database [ AJD ]

I am not going to discuss about AJD installation. For AJD deployment steps, refer https://blogs.oracle.com/database/post/mongodb-api

Assuming AJD is deployed and existing in OCI

GoldenGate for Big Data uses “Oracle Database API for MongoDB” to connect to Autonomous JSON Database.

The Database API for MongoDB will only be available if we define "Secure access from allowed Ips and VCNs only" in “Choose Network Access” step of Autonomous JSON Database service creation. We can see the details in the above shared link.

Once Autonomous JSON Database is created, we can check the Oracle Database API for MongoDB from Service Console/ Development.

4 Deployment: MongoDB

• Assuming MongoDB is already installed, it is up and running

• Assuming MongoDB is configured with admin / application user

  
  

  • Example

    • db.createUser({user:"adminusr",pwd:"admin123",roles:["clusterAdmin","readWriteAnyDatabase","dbAdminAnyDatabase","userAdminAnyDatabase"]})

      
      

      admin> db.grantRolesToUser('adminusr', [{ role: 'root', db: 'admin' }])

      
      

      Under crskdb7 database

      db.createUser({user:"crsk",pwd:"crsk123",roles:[{role: "dbOwner",db:"crskdb7"}]})

      
      

    • MongoDB should be running in Replica Mode

      Start the MongoDB in replicate mode if it runs in standalone mode

      
      

      AS ROOT

      Start mongod Deamon as replicate

      [root@cvm-pgl bin]#

      /usr/bin/mongod --bind_ip 0.0.0.0 --port 27017 --replSet rs0 --dbpath /var/lib/mongo

      
      

• [ Optional ] Do Connection check: from OGGMS_MDB server

  
  

  • To ensure there is NO connection issue, optionally we can deploy mongoshell in GoldenGate MDB server

    ( OGGMS_MDB) and connect to MongoDB server

    
    

    cvm_ogg: -bash-4.2$

    mongosh "mongodb://140.xxx.xxx.xx1:27017/crskdb7" --username=crsk

    rs0 [direct: primary] crskdb7>

    
    

    Once we able to connect, same connection string can be used later in OGG Microservice console

    
    

    mongodb://140.xxx.xxx.xx1:27017/crskdb7

    mongodb://140.xxx.xxx.xx1:27017/crskdb7?replicaSet=rs0

5 Configure OGG MicroService

We need to configure OGG by providing SOURCE  and TARGET database credentials

5.1 Configure Source Database Connection

• Login to OGG console via browser

• Login to Administration Service via credentials

• Under configuration , Add MongoDB connection string, click submit

User ID is the MongoDB connection URI that we use to connect to our MongoDB deployment. ReplicatSet can also be provided in User ID.

mongodb://140.xxx.xxx.xx1:27017/crskdb7

By default I am connecting as ‘admin’ user to connect database “crskdb7” via port “27017” which runs at host of “140.xxx.xxx.xx1”

If required we can provide any other user who has admin privilege as below example with replicaSet 

User ID: mongodb://UserAdmin@localhost:27017/admin?replicaSet=rs0

In this example, “UserAdmin” is the user name to connect to MongoDB, “localhost” is the hostname where MongoDB is running, “admin”, is the user name (but not mandatory) and “replicaSet” is the Mongo DB replica set name.

5.2 Globals

Once connected, provide below values to GLOBALS configuration as below

-        JVMCLASSPATH and dependencies jar file location

-        JVMBOOTOPTIONS for Java heap size

Example:

# GLOBALS

OGGSOURCE MONGODB

JVMCLASSPATH /u01/app/ogg/opt/DependencyDownloader/dependencies/mongodb_capture_4.4.1/*:/u01/app/ogg/ggjava/resources/lib/*:/u01/app/ogg/ggjava/resources/lib/optional/*

JVMBOOTOPTIONS -Xms512m -Xmx512m -Xss32m -Dgg.log=log4j 
-Dgg.log.level=INFO

5.3 Configure Extract: CDC

Note:

For No-Down time Migration / Replication, the first step should be, to create and start the CDC Extract from the source MongoDB to capture ongoing changes before the initial load

Create and Start CDC Extract from Source MongoDB

To create a Change Data Capture extract, go to Administration Service/ Extracts/ Add Extract.

Select “Change Data Capture Extract” and click “Next”.

In my setup,

• I’m capturing from a database named “crskdb7” and a collection called “sales”. 

• I’ve also added “No_ABEND_ON_DDL” property as Mongo DB capture currently does not support Create/ Rename/ Drop Collection operations. 

Snapshots:

Creation of CDC Extract Process

Edit Parameter File and click “Create & Run”.

-- Parameter file for MongoDB extract.

EXTRACT CDCEX

EXTTRAIL ex

SOURCEDB USERIDALIAS OGG_MongoDB DOMAIN OGG_MongoDB

no_abend_on_ddl

TABLE crskdb7.movies;

If your Extract starts successfully, it will be shown in the “Running” status as seen below.

If Extract fails, we can check the error message in Action/ Details/ Report. Fix the issue.

5.4 Configure Extract: Initial Load

After creating Change Data Capture Extract, we will configure “Initial Load Extract”. “Initial Load Extract” can be used to extract source table data to Oracle GoldenGate trail files. It will create a file that will be used by the replicat.

To add initial load extract,

• Administration Service -> Add Extract -> [ "Initial Load Extract " ] -> Next.

• Provide appropriate database credentials that created in previous step

System generated Parameter File will have exttrail parameter.

We need to change “exttrail” into “extfile” and provide the trail name that we defined in “Extract Options” (in my case it was IN).

EXTFILE parameter specifies an extract file that will be read by the Initial Load Replicat. 

-- Parameter file I used for this use case MongoDB extract.

EXTRACT INEX

EXTFILE in

SOURCEDB USERIDALIAS OGG_MongoDB DOMAIN OGG_MongoDB

no_abend_on_ddl

TABLE crskdb7.movies;

Click action > Start

This action will initiate the load. To see if it is successfully completed,

go to Actions -> Details -> Report.

If we scroll down, we’ll see the number of inserts completed. Below screen shot is from my extract report.

If NO error reported, we can view the number of records loaded in trail file.

We can see trail file being created in OGGMS_MDB  instance under /u02/trails

5.5 Configure Replicat: Initial Load

The "Initial Load Replicat" reads the trail files and loads the data into Autonomous Database, which is generated by the "Initial Load Extract."

To configure the replicat, we need to get the "Oracle Database API for MongoDB" from the Autonomous JSON Database which we created in step 3.

To get it, go to Autonomous Database Details and click ["Database Actions"].

We'll see Oracle Database API for MongoDB in Related Services.

When we copied, it looks like similar to this:

mongodb://[user:password@] EZDOOMZGCKWGUYC-DMAJD.adb.ap-mumbai-1.oraclecloudapps.com:27017/[user]?authMechanism=PLAIN&authSource=$external&ssl=true&retryWrites=false

We need to update it with the appropriate username and password.

So, it would look like this:

mongodb://usr4mig:MyPassWord@EZDOOMZGCKWGUYC-DMAJD.adb.ap-mumbai-1.oraclecloudapps.com:27017/usr4mig?authMechanism=PLAIN&authSource=$external&ssl=true&retryWrites=false

Above, Oracle MongoDB API is used in Replicat properties section.

To configure the initial load replicat,

- Administration Service -> Replicats -> Add Replicat. 

- Choose ["Classic Replicat"]

- Select Target as ‘MongoDB’

5.5.1 Parameter - Replicate Initial Load

-        We can define our source to target mappings.

As shown below, I have updated my source database collection, which has been mapped from MongoDB crskdb7 database to the Oracle TARGET usr4mig schema.

#Parameters

REPLICAT INLD2ATP

MAP crskdb7.movies, TARGET usr4mig.movies;

5.5.2 Properties - Replicate Initial Load

Properties

• We need to update gg.handler.mongodb.clientURI and gg.classpath. 

• gg.handler.mongodb.clientURI is the connection string that we can find in Autonomous JSON that was created in section 3.

• gg.classpath is the path to MongoDB dependency files directory that we created in section 1 by using the “Dependency Downloader Utility”. 

Below is the property file that I used for my replicat:

# Properties file for Replicat INREP

gg.handlerlist=mongodb

gg.handler.mongodb.type=mongodb

gg.handler.mongodb.clientURI=mongodb://usr4mig:Yourxxpwd@EZDOOMZGCKWGUYC-DMAJD.adb.ap-mumbai-1.oraclecloudapps.com:27017/usr4mig?authMechanism=PLAIN&authSource=$external&ssl=true&retryWrites=false&loadBalanced=true

gg.classpath=/u01/app/ogg/opt/DependencyDownloader/dependencies/mongodb_capture_4.4.1/*:/u01/app/ogg/opt/DependencyDownloader/dependencies/mongodb_4.4.1/*

# Java Memory Heap size

jvm.bootoptions=-Xmx512m -Xms32m

Once replicats created, click Action -> Start

If all is good, we’ll see the replicat in running status with a green check as below. 

5.5.3 Log - Replicate Initial Load 

Log file will provide the tables that are mapped and error if any.

2023-12-06 09:55:06  INFO    OGG-25339  Connected to database Genericdb, CPU information not available.

***********************************************************************

                     Run Time Messages                             

***********************************************************************

2023-12-06 09:55:06  INFO    OGG-30067  Opened trail file /u02/trails/in000000, RBA: 0 at 2023-12-06 09:55:06.499665.

2023-12-06 09:55:06  INFO    OGG-03506  The source database character set, as determined from the trail file, is UTF-8.

2023-12-06 09:55:06  INFO    OGG-06505  MAP resolved (entry crskdb7.movies): MAP "crskdb7"."movies", TARGET usr4mig.movies.

2023-12-06 09:55:06  INFO    OGG-02756  The definition for table crskdb7.movies is obtained from the trail file.

2023-12-06 09:55:06  INFO    OGG-15056  The definition for target table usr4mig.movies is derived from the source table crskdb7.movies.

2023-12-06 09:55:06  INFO    OGG-06511  Using following columns in default map by name: id, payload.

2023-12-06 09:55:06  INFO    OGG-06510  Using the following key columns for target table usr4mig.movies: id.

5.5.4 Statistics - Replicate Initial Load 

Statistics will provide , number of records that are inserted in Target Database.

Go to Report tab Under Overview > Statistics information , we’ll  see the number of inserts.

Crosscheck the Record count

Source :

Target:

5.6 Configure Replicat: CDC ( Change Data Capture )

Once initial load is complete, we need to make sure that source MongoDB and target Autonomous JSON databases are in sync continuously. To achieve source and target sync, we need to configure a CDC (Change Data Capture) replicat. CDC replicat will write messages to target Autonomous JSON as captured from the source MongoDB.

To Configure the CDC replicat

• Go to Administration Service/ Replicats/ Add Replicat.

Challenge:

We have configured and initiated the CDC Extract from MongoDB before the Initial Load Extract. This indicates that it has been recording change data from the source MongoDB for some time, and there could be identical data in the trial file of the CDC Extract, potentially causing duplicates between the completion of the Initial Load and the CDC Extract.

To avoid duplicates or to make sure there NO transactions are missing, we need to use HANDLECOLLISIONS parameter. 

5.6.1 Parameter - Replicate CDC

In the parameters section, we can define our source to target mappings with appropriate MAPPINGs.

Add HANDLECOLLISIONS parameter here. 

#Parameters for CDCREP

REPLICAT CDCREP

HANDLECOLLISIONS

MAP crskdb7.movies, TARGET usr4mig.movies;

5.6.2 Properties - Replicate CDC

Next step is to configure properties file.

-        There is an icon next to “Properties File”. If we click that, it will show the properties in the other replicats that were created.

-        We can re-use the property from the initial load that you created in the earlier steps.

My values in properties

gg.handlerlist=mongodb

gg.handler.mongodb.type=mongodb

gg.handler.mongodb.clientURI=mongodb://usr4mig:Welcomexxpwd908@EZDOOMZGCKWGUYC-DMAJD.adb.ap-mumbai-1.oraclecloudapps.com:27017/usr4mig?authMechanism=PLAIN&authSource=$external&ssl=true&retryWrites=false&loadBalanced=true

gg.classpath=/u01/app/ogg/opt/DependencyDownloader/dependencies/mongodb_capture_4.4.1/*:/u01/app/ogg/opt/DependencyDownloader/dependencies/mongodb_4.4.1/*

jvm.bootoptions=-Xmx512m -Xms32m

Once “Properties” are set, we can click & run and start the replicat. 

If all is good, we’ll see the replicat in running status with a green check as below. 

When our source & target is synced, we can stop the CDC replicat, remove HANDLECOLLISIONS and re-start the replicat.

HANDLECOLLISIOS has a negative impact on the performance and that’s why it is recommended to remove it. 

6 Crosscheck - CDC Replicate

Let we do test CDC replicate by inserting few records at source

6.1 Do Some inserts at Source

db.movies.insertOne({plot:"crsk plot1",genres: [ 'Short', 'Drama' ],runtime: 14,rated: 'UNRATED',type: 'crskmovie1'});

db.movies.insertOne({plot:"crsk plot2",genres: [ 'Short', 'Drama' ],runtime: 14,rated: 'UNRATED',type: 'crskmovie2'});

db.movies.insertOne({plot:"crsk plot3",genres: [ 'Short', 'Drama' ],runtime: 14,rated: 'UNRATED',type: 'crskmovie3'});

db.movies.insertOne({plot:"crsk plot4",genres: [ 'Short', 'Drama' ],runtime: 14,rated: 'UNRATED',type: 'crskmovie4'});

6.2 Check CDC Replicate Statistics

Under Statistics tab , we can see number of inserts inserted to TARGET JSON database that read from trial file.

6.2 Check at Target ( AJD )

Login to sqlplus and do select query from AJD

Observation:

Recent inserted records at source were replicated in Target AJD. Here after any time you can plan for switchover window to route our application to AJD.

Conclusion:

GoldenGate for Big Data is an easy to configure solution and easily can be used for no down-time migrations from non-oracle to Oracle Cloud Infrastructure Autonomous JSON Database.

No-Downtime migrations from MongoDB to OCI Autonomous JSON Database can be achieved with GoldenGate for Big Data.

In next article i'll share how to migrate MongoDB-Atlas to ADB .

I believe the this article offered valuable insights about migrating from MongoDB to Oracle Autonomous (JSON) DB.

Thanks for your time

Ref:

https://blogs.oracle.com/database/post/mongodb-api

https://blogs.oracle.com/dataintegration/post/oci-goldengate-mongodb-to-autonomous-json-database

https://www.mongodb.com/docs/drivers/java/sync/current/compatibility/

https://mongodb.github.io/mongo-java-driver/

https://docs.oracle.com/en/middleware/goldengate/core/21.3/oggmp/provisioning-oracle-goldengate-microservices-oci-marketplace.html#GUID-B7BB0551-C5CF-42B0-9E52-01C2384FE2D8