(No-Downtime) Migration from MongoDB(Atlas) to Oracle ADB 23ai - Part 5

With continuation of migration series from MongoDB (Collections) to Oracle (JSON), this article will present the MoP ( Method Of Procedure ) to migrate MongoDB collections from MongoDB Atlas to Oracle Autonomous Database 23ai without down time using GoldenGate v23ai.

Migration from MongoDB (Atlas) to Oracle autonomous Database via Oracle GoldenGate

Disclaimer:

The information provided in this article is for general informational and educational purposes only. All content, including code snippets, configurations, and technical recommendations, is shared in good faith and reflects the author's personal understanding and experience at the time of writing.

  All views expressed are the author’s personal opinions and it is not affiliated with, endorsed by Oracle Corporation or any other entity. My current employer or previous employer nowhere responsible for this content. Readers are advised to perform their own research and testing before applying any of the concepts discussed.

Preface:

I had a requirement to migrate collections from MongoDB (Atlas) to Autonomous Database using Oracle GoldenGate Microservices (DAA) v23ai.

Here is the technical details, I used to migrate Collections from MongoDB Atlas to Oracle ADB 23ai using Oracle GoldenGate Microservices v23ai:

Bill of Material ( BoM)

• Oracle GoldenGate 23ai ( at time of PoC, OGG v23.5.2 is the latest release )

- Free from Oracle marketplace

• Oracle Autonomous Database 23ai Free

• MongoDB Atlas

Oracle GoldenGate 23ai for Distributed Applications and Analytics (GGDAA) is an enhanced version of Oracle GoldenGate that streamlines data replication and integration for distributed environments, particularly for applications and analytics. It focuses on real-time data capture, delivery, and integration with various data stores, including big data systems and cloud platforms. For more details please refer here

Oracle Database 23ai is the next long-term support release of Oracle Database, superseding 19c. This release brings AI to your data with the addition of AI Vector Search to Oracle's converged database. For more details about Oracle Database 23ai Free offers refer here

Since i already covered below sections in Part-4,

1/ Deployment of OGG microservices,

2/ Pre-requisite ( Ports, Dependency jar files )

3/ Deployment of ADB

4/ And under assumption of source MongoDB (Atlas) is pre-configured and

accessible.

I'll skip above sections and focus on other critical factors . Also i have shared demo video for detailed reference.

Dependency Downloader ( Pre-requisite )

For this usecase, i used v5.4 for mongodb. (dependency ) java drivers for Mongodb.sh and mongodb_capture.sh

[opc@ogg23aibigdata]$ cd /u01/app/ogg/opt/DependencyDownloader

[opc@ogg23aibigdata DependencyDownloader]$ ./mongodb.sh 5.4.0

[opc@ogg23aibigdata DependencyDownloader]$ ./mongodb_capture.sh 5.4.0

Atlas Connection String

This is my connection string used in OGG-MS to connect Atlas

mongodb+srv://mymongo:

@mongo-mycluster.1w8og.mongodb.net/?retryWrites=true&w=majority&appName=Mongo-mycrskcluster

Connection check: from ogg23aibigdata (OGGMS_DAA) server

• To ensure there is NO connection issue, optionally we can deploy mongoshell in GoldenGate (OGG23aibigdata) server and connect to MongoDB Atlas

Once we able to connect, same connection string can be used later in OGG Microservice console

5 Configure OGG MicroService

We need to configure OGG by providing SOURCE  and TARGET database credentials

5.1 Configure Source Database Connection

• Login to OGG console via browser

• Login to Administration Service via credentials

• Under configuration , Add MongoDB connection string, click submit

5.2 Globals

Once connected, provide below values to GLOBALS configuration as below

-        JVMCLASSPATH and dependencies jar file location

-        JVMBOOTOPTIONS for Java heap size

# GLOBALS

OGGSOURCE MONGODB	

JVMCLASSPATH /u01/app/ogg/opt/DependencyDownloader/dependencies/mongodb_capture_5.4.0/*:/u01/app/ogg/ggjava/resources/lib/*:/u01/app/ogg/ggjava/resources/lib/optional/*

JVMBOOTOPTIONS -Xms512m -Xmx512m -Xss32m -Dgg.log=log4j -Dgg.log.level=INFO

5.3 Configure Extract: CDC

For No-Down time Migration, the first step should be to create and start the CDC Extract from the source MongoDB to capture ongoing changes before the initial load

To create a Change Data Capture extract, go to Administration Service/ Extracts/ Add Extract.

Select “Change Data Capture Extract” and click “Next”.

In this usecase,

• I’m capturing 1 collections from “sample_airbnb" and 5 collections from sample_analytics database. 

  
  

-- Parameter file for MongoDB extract.
EXTRACT cdcex

EXTTRAIL ex

SOURCEDB USERIDALIAS crskatlas DOMAIN cratlas

JVMOPTIONS CLASSPATH /u01/app/ogg/ggjava/ggjava.jar:/u01/app/ogg/opt/DependencyDownloader/dependencies/mongodb_capture_5.4.0/*:/u01/app/ogg/opt/DependencyDownloader/dependencies/mongodb_5.4.0/*

no_abend_on_ddl

TABLE sample_airbnb.listingsAndReviews;

TABLE sample_analytics.accounts;

TABLE sample_analytics.customers;

TABLE sample_analytics.transactions;

TABLE sample_analytics.mybooks;

TABLE sample_analytics.mycollection;

5.4 Configure Extract: Initial Load

After creating Change Data Capture Extract, we will configure “Initial Load Extract”.

“Initial Load Extract” is used to extract source table data to Oracle GoldenGate trail files.

To add initial load extract,

• Administration Service -> Add Extract -> [ "Initial Load Extract " ] -> Next.

• Provide appropriate database credentials that created in previous step

-- Parameter file for MongoDB Initial Load extract.

EXTRACT inex

EXTFILE in

SOURCEDB USERIDALIAS crskatlas DOMAIN cratlas

JVMOPTIONS CLASSPATH /u01/app/ogg/ggjava/ggjava.jar:/u01/app/ogg/opt/DependencyDownloader/dependencies/mongodb_capture_5.4.0/*

no_abend_on_ddl

TABLE sample_airbnb.listingsAndReviews;

TABLE sample_analytics.accounts;

TABLE sample_analytics.customers;

TABLE sample_analytics.transactions;

TABLE sample_analytics.mybooks;

TABLE sample_analytics.mycollection;

if initial load succesfully executed, we’ll see the number of inserts completed in the report file and we can notice the trail file being created in OGGMS  instance under /u02/trails

i have shared demo video for reference.

5.5 Configure Replicat: Initial Load

The "Initial Load Replicat" reads the trail files and loads the data into Autonomous Database 23ai, which is generated by the "Initial Load Extract."

To configure the replicat, we need to get the "Oracle Database API for MongoDB" from the Autonomous Database 23ai.

To obtain "Oracle Database API for MongoDB" , navigate to Autonomous Database Details and select ["Database Actions"].

We'll see Oracle Database API for MongoDB in Related Services.

We need to update "Oracle Database API for MongoDB" with the appropriate username and password.

Example:

mongodb://[username:userpassword]@LSTSSXUEGQJLDW2-HX7M-ADB23AI.adb.us-ashburn-1.oraclecloudapps.com:27017/[username]?authMechanism=PLAIN&authSource=$external&ssl=true&retryWrites=false&loadBalanced=true

Edited with my values:

mongodb://appjson:Mypwdxxxxx@LSTSSXUEGQJLDW2-HX7M-ADB23AI.adb.us-ashburn-1.oraclecloudapps.com:27017/appjson?authMechanism=PLAIN&authSource=$external&ssl=true&retryWrites=false&loadBalanced=true

Above, Oracle MongoDB API is utilized in the [Replicat] properties section.

To configure the initial load replicat,

- Administration Service -> Replicats -> Add Replicat. 

- Choose ["Classic Replicat"]

- Provide appropriate value of Replicat Trial Name, ( i used " in" for my case) and

rest leave blank to take default values.

- Select Target as ‘MongoDB’.. proceed further click "Create"

5.5.1 Parameter - Replicate Initial Load 

We define our source to target mappings.

As shown below, I have updated my source database collection, which has been mapped from MongoDB databases to the Oracle TARGET appjson schema.

5.5.2 Properties - Replicate Initial Load 

Properties 

• We need to update gg.handler.mongodb.clientURI and gg.classpath. 

• gg.handler.mongodb.clientURI is the connection string that we can find in Oracle Autonomous Database

• gg.classpath is the path to MongoDB dependency files directory that we created using the “Dependency Downloader Utility”. 

Below is the property file that I used for my replicat:

# Properties file for Replicat INREP

# MongoDB Handler Template

# Note: Recommended to only edit the configuration marked as  TODO

gg.handlerlist=mongodb

gg.handler.mongodb.type=mongodb

# connection to ADB

gg.handler.mongodb.clientURI=mongodb://appjson:Mypwdxxxx@LSTSSXUEGQJLDW2-HX7M-ADB23AI.adb.us-ashburn-1.oraclecloudapps.com:27017/appjson?authMechanism=PLAIN&authSource=$external&ssl=true&retryWrites=false&loadBalanced=true

#TODO: Edit the path to MongoDB Java driver.

gg.classpath=/u01/app/ogg/opt/DependencyDownloader/dependencies/mongodb_capture_5.4.0/*:/u01/app/ogg/opt/DependencyDownloader/dependencies/mongodb_5.4.0/*

# addtional parameter for Java memory

jvm.bootoptions=-Xmx512m -Xms32m

Once CDC initial replicat created and parameters & properties are good, click [Start] button

If all is good, we’ll see the Intial Load replicat is in running status.

5.5.3 Log & Statistics - Replicate Initial Load

- Log file will provide the tables that are mapped and error if any.

- Statistics will provide , number of records that are inserted in Target Database.

5.6 Configure Replicat: CDC ( Change Data Capture )

Once initial load is complete, we need to make sure that source MongoDB and target Oracle Autonomous Databases are in sync continuously.

To achieve source and target sync, we need to configure a CDC (Change Data Capture) replicat.

CDC replicat will replicat changes ( Inserts/Updated/Delete) to target Oracle Autonomous as captured from the source MongoDB.

To Configure the CDC replicat

• Go to Administration Service/ Replicats/ Add Replicat.

Challenge:

We set up and initiated the CDC Extract from MongoDB before the Initial Load Extract. This indicates that it has been recording change data from the source MongoDB for some time, and there could be identical data in the trial file of the CDC Extract, potentially causing duplicates between the completion of the Initial Load and the CDC Extract.

To avoid duplicates or to make sure there NO transactions are missing, we need to use HANDLECOLLISIONS parameter. 

Note: Above challenge is fixed in OGG v23.7 using oplogLSN checkpoint. i'll share new blog on this release.

5.6.1 Parameter - Replicate CDC

In the parameters section, we can define our source to target mappings with appropriate MAPPINGs.

REPLICAT cdcrep

HANDLECOLLISIONS

MAP sample_airbnb.listingsAndReviews, TARGET APPJSON.listingsandreviews;

MAP sample_analytics.accounts, TARGET APPJSON.accounts;

MAP sample_analytics.customers, TARGET APPJSON.customers;

MAP sample_analytics.transactions, TARGET APPJSON.transactions;

MAP sample_analytics.mybooks, TARGET APPJSON.mybooks;

MAP sample_analytics.mycollection, TARGET APPJSON.mycollection;

5.6.2 Properties - Replicate CDC

Next step is to configure properties file.

-        We can re-use the property from the initial load Replicat that we created in the earlier steps.

Now we are good to "Create" Replicat - CDC ( Generally i choose option to "Create" rather than "Create and Run")

if parameters and properties are good, we can Start the Replicat - CDC

and If all is good, we’ll see the replicat in running status

When our source & target is synced, we can stop the CDC replicat, remove HANDLECOLLISIONS and re-start the replicat.

6 Crosscheck - CDC Replicate

Test CDC replicate by inserting few records at source

6.1 Do Some inserts at Source

6.2 Check CDC Replicate Statistics

Under Statistics tab , we can see number of inserts inserted to TARGET JSON database that read from trial file.

As the sections above are thoroughly covered in part-4, I have omitted them here to prevent redundancy. Please view the video (demo) that provides the full steps for migrating from MongoDB Atlas to Oracle Autonomous Database.

Conclusion

Oracle GoldenGate 23ai for Distributed Applications and Analytics (GGDAA) ensures data consistency, minimal or NO downtime, seamless business continuity and plays a pivotal role in enabling real-time replication and ensuring transactional integrity throughout the process.

In next article i'll share how to migrate MongoDB to ADB using oplogLSN checkpoint. ( without using HANDLECOLLISIONS ).

I believe the this article offered valuable insights about migrating from MongoDB to Oracle Autonomous Database 23ai without Downtime.

Thanks for your time.

Other References:

Real Time Replication / Migrations from MongoDB to Oracle Autonomous Database - Part 4

Migration of MongoDB collections To Oracle JSON using SODA API - Part 3

Migration of MongoDB collections To Oracle JSON(ADB) — Part 2

Migration of Mongodb collections To Oracle Json — Part 1