{"id":11905,"date":"2017-11-22T08:46:10","date_gmt":"2017-11-22T13:46:10","guid":{"rendered":"http:\/\/www.iri.com\/devblog\/?p=11905"},"modified":"2022-08-18T15:06:59","modified_gmt":"2022-08-18T19:06:59","slug":"db-changes-oracle-mongo","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/","title":{"rendered":"Detecting Incremental Database Changes (Oracle to MongoDB ETL)"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">Detecting additions and updates to database tables for data replication, ETL, PII masking, and other incremental data movement and manipulation <a href=\"https:\/\/www.iri.com\/solutions\/data-integration\/implement\">activities<\/a> can be automated in <\/span><a href=\"http:\/\/www.iri.com\/products\/voracity\"><span style=\"font-weight: 400;\">IRI Voracity<\/span><\/a><span style=\"font-weight: 400;\"> workflows designed and run in <\/span><a href=\"http:\/\/www.iri.com\/products\/workbench\"><span style=\"font-weight: 400;\">IRI Workbench<\/span><\/a><span style=\"font-weight: 400;\"> (WB). This article explains how to regularly check for changes in Oracle source tables to decide when to move data into a MongoDB target. <span id='easy-footnote-1-11905' class='easy-footnote-margin-adjust'><\/span><span class='easy-footnote'><a href='https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#easy-footnote-bottom-1-11905' title='This approach differs from third-party log-based &lt;a href=&quot;http:\/\/www.iri.com\/solutions\/business-intelligence\/embedded-bi\/change-data-capture&quot;&gt;change data capture&lt;\/a&gt; solutions, which typically have performance bottlenecks, and are limited to specific databases, and do not enable simultaneous data transformation, PII data masking, cleansing, and reporting.'><sup>1<\/sup><\/a><\/span> Note however that in 2022, IRI also added the &#8220;<a href=\"https:\/\/www.iri.com\/blog\/vldb-operations\/getting-started-with-iri-ripcurrent\">Ripcurrent<\/a>&#8221; module for automatic (real-time), incremental data <a href=\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/real-time-database-data-replication\/\">replication<\/a> and classified PII <a href=\"https:\/\/www.iri.com\/blog\/data-protection\/real-time-incremental-data-masking\/\">masking<\/a> for selected relational databases.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In the example this article details, changed data is loaded into different databases or files using a task-scheduled batch file or shell script. This can be done using a timestamp and specific fields in the source table. Error checking is included and can also be reacted upon.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">This example will be created and run on a Windows machine; however, it can be easily modified to work on a Linux or Unix like platform.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Creating the batch file is easy using a Voracity Flow diagram in WB. In this example, the source table contains columns named <\/span><i><span style=\"font-weight: 400;\">CREATION_DATE<\/span><\/i><span style=\"font-weight: 400;\"> and <\/span><i><span style=\"font-weight: 400;\">UPDATE_DATE<\/span><\/i><span style=\"font-weight: 400;\"> that are important in this job. <\/span><\/p>\n<p><span style=\"font-weight: 400;\">The image below shows the steps that are contained in the batch file. To summarize: <\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">the job is run in a specific directory<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">an environment variable is set using the timestamp of the last job run<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">the current timestamp is recorded<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">current changes are captured<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">the error level is checked and acted upon if successful or not<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">the current timestamp overwrites the last run timestamp<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">the changed data is converted to CSV<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">a stall occurs to wait for the last file to exist<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">the CSV file is imported into MongoDB<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">the error level is checked, the current file is truncated<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">the changes file is deleted<\/span><\/li>\n<\/ul>\n<p><a href=\"http:\/\/www.iri.com\/devblog\/wp-content\/uploads\/2017\/11\/workflow-diagram-mongo.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-11909\" src=\"http:\/\/www.iri.com\/devblog\/wp-content\/uploads\/2017\/11\/workflow-diagram-mongo.png\" alt=\"workflow diagram mongodb\" width=\"600\" height=\"333\" srcset=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/workflow-diagram-mongo.png 1127w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/workflow-diagram-mongo-300x166.png 300w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/workflow-diagram-mongo-768x426.png 768w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/workflow-diagram-mongo-1024x568.png 1024w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/a><br \/>\n<span style=\"font-weight: 400;\">Note that it is also possible to add data cleansing and data masking functions to the target fields within the same job. Each task block in the workflow is explained below. For the how-to on building Voracity workflows from the palette, see <\/span><a href=\"http:\/\/www.iri.com\/blog\/data-protection\/creating-a-voracity-flow-from-the-palette-part-1-of-2\/\"><span style=\"font-weight: 400;\">this article<\/span><\/a><span style=\"font-weight: 400;\">.<\/span><\/p>\n<h3><b>Change Directory<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">This block changes the current working directory to the one specified.<\/span><\/p>\n<h3><b>Set LASTTIME<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">This command line block sets an environment variable called <\/span><i><span style=\"font-weight: 400;\">LASTTIME<\/span><\/i><span style=\"font-weight: 400;\">. The value set to the variable is the contents of the file <\/span><i><span style=\"font-weight: 400;\">LastTime.txt<\/span><\/i><span style=\"font-weight: 400;\">. The timestamp in this file is the timestamp that was recorded during the last run of this job. If this is the first run, this file will have to be made manually with an arbitrary timestamp dated before this job is run.<\/span><\/p>\n<h3><b>Timestamp.scl<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">This transformation block uses the CoSort <\/span><a href=\"http:\/\/www.iri.com\/products\/cosort\/sortcl\"><span style=\"font-weight: 400;\">SortCL<\/span><\/a><span style=\"font-weight: 400;\"> program in Voracity to query the source database for the current time. That timestamp is saved to a file called <\/span><i><span style=\"font-weight: 400;\">LastTimeTemp.txt<\/span><\/i><span style=\"font-weight: 400;\">. The reason it is stored in a temp file is so that both the current and last time stamps can be preserved until error checking occurs. <\/span><\/p>\n<p><span style=\"font-weight: 400;\">It is important that the timestamp comes from the database, and not the local machine. This avoids problems where the database and the execution environment are not synchronized.<\/span><\/p>\n<h3><b>Changes.scl<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">This transformation block does a few things. Displayed below is the Transform Mapping Diagram for this block. Input is the source table and output is the file <\/span><i><span style=\"font-weight: 400;\">current.txt<\/span><\/i><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><a href=\"http:\/\/www.iri.com\/devblog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-11908\" src=\"http:\/\/www.iri.com\/devblog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo.png\" alt=\"transform mapping diagram mongodb\" width=\"600\" height=\"514\" srcset=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo.png 1252w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo-300x257.png 300w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo-768x658.png 768w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo-1024x877.png 1024w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo-350x300.png 350w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/a><\/p>\n<p><span style=\"font-weight: 400;\">In the input <\/span><i><span style=\"font-weight: 400;\">Section Options, <\/span><\/i><span style=\"font-weight: 400;\">a query is submitted to the source table for all records that have a <\/span><i><span style=\"font-weight: 400;\">CREATION_DATE<\/span><\/i><span style=\"font-weight: 400;\"> or <\/span><i><span style=\"font-weight: 400;\">UPDATE_DATE<\/span><\/i><span style=\"font-weight: 400;\"> \u00a0greater than the environment variable <\/span><i><span style=\"font-weight: 400;\">LASTTIME<\/span><\/i><span style=\"font-weight: 400;\">. <\/span><\/p>\n<p><span style=\"font-weight: 400;\">While the output appears to have two targets , the data is actually being appended to the same file using two different conditions. In the first output section, there is an <\/span><i><span style=\"font-weight: 400;\">Include<\/span><\/i><span style=\"font-weight: 400;\"> statement that finds all records that have a <\/span><i><span style=\"font-weight: 400;\">CREATION_DATE<\/span><\/i><span style=\"font-weight: 400;\"> greater than <\/span><i><span style=\"font-weight: 400;\">LASTTIME<\/span><\/i><span style=\"font-weight: 400;\">. There is also an additional output field called <\/span><i><span style=\"font-weight: 400;\">CDC_TYPE<\/span><\/i><span style=\"font-weight: 400;\">. The string \u201cCREATE\u201d is recorded in that new field. <\/span><\/p>\n<p><span style=\"font-weight: 400;\">In the second output section, an <\/span><i><span style=\"font-weight: 400;\">Include<\/span><\/i><span style=\"font-weight: 400;\"> statement finds all records that have a <\/span><i><span style=\"font-weight: 400;\">UPDATE_DATE<\/span><\/i><span style=\"font-weight: 400;\"> greater than <\/span><i><span style=\"font-weight: 400;\">LASTTIME<\/span><\/i><span style=\"font-weight: 400;\"> and where <\/span><i><span style=\"font-weight: 400;\">CREATION_DATE <\/span><\/i><span style=\"font-weight: 400;\">is not equal to <\/span><i><span style=\"font-weight: 400;\">UPDATE_DATE. <\/span><\/i><span style=\"font-weight: 400;\">This ensures that newly created files are not included in this pass. \u00a0The string \u201cUPDATE\u201d is recorded in <\/span><i><span style=\"font-weight: 400;\">CDC_TYPE.<\/span><\/i><\/p>\n<h3><b>Error CoSort<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">This decision block checks the variable <\/span><i><span style=\"font-weight: 400;\">ERRORLEVEL<\/span><\/i><span style=\"font-weight: 400;\"> to make sure it returned 0 (or success) after running the CoSort job above. If it did not, the job continues to the <\/span><i><span style=\"font-weight: 400;\">EXIT<\/span><\/i><span style=\"font-weight: 400;\"> block where the job is terminated. If it returns true, the job continues to the next block.<\/span><\/p>\n<h3><b>Rename LastTimeTemp<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">This command block copies the contents of LastTimeTemp.txt to LastTime.txt. This records the previously captured current timestamp in the file to be used for the next job run.<\/span><\/p>\n<h3><b>Convert.scl<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">This transform block takes <\/span><i><span style=\"font-weight: 400;\">current.txt<\/span><\/i><span style=\"font-weight: 400;\"> and converts it to <\/span><i><span style=\"font-weight: 400;\">changes.csv<\/span><\/i><span style=\"font-weight: 400;\">. The conversion is from the default delimited file type to CSV. Using the CSV process type in CoSort prepends a header row to the output file using the field names. This is the task block where I could apply <\/span><a href=\"http:\/\/www.iri.com\/products\/cosort\/sortcl\/function-matrix\"><span style=\"font-weight: 400;\">other manipulations<\/span><\/a><span style=\"font-weight: 400;\">\u00a0(like data masking) to the data if I so choose.<\/span><\/p>\n<h3><b>Wait Files<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">This wait block stalls the batch file for 3 seconds and then checks for the existence of the <\/span><i><span style=\"font-weight: 400;\">changes.csv<\/span><\/i><span style=\"font-weight: 400;\"> file before proceeding. <\/span><\/p>\n<h3><b>MongoImport<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">This command block executes the mongoimport command using the parameters specified in the properties view as shown below.<\/span><\/p>\n<p><a href=\"http:\/\/www.iri.com\/devblog\/wp-content\/uploads\/2017\/11\/mongo-properties-block.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-11907\" src=\"http:\/\/www.iri.com\/devblog\/wp-content\/uploads\/2017\/11\/mongo-properties-block.png\" alt=\"mongo properties block\" width=\"377\" height=\"197\" srcset=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/mongo-properties-block.png 377w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/mongo-properties-block-300x157.png 300w\" sizes=\"(max-width: 377px) 100vw, 377px\" \/><\/a><\/p>\n<p><span style=\"font-weight: 400;\">The parameters indicate that the MongoDB database called <\/span><i><span style=\"font-weight: 400;\">fnx <\/span><\/i><span style=\"font-weight: 400;\">is to be loaded with the contents of the file <\/span><i><span style=\"font-weight: 400;\">changes.csv<\/span><\/i><span style=\"font-weight: 400;\"> which is of the type <\/span><i><span style=\"font-weight: 400;\">csv<\/span><\/i><span style=\"font-weight: 400;\"> and contains a <\/span><i><span style=\"font-weight: 400;\">headerline<\/span><\/i><span style=\"font-weight: 400;\"> which defines the fields.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Note that Voracity supports other methods of moving and manipulating MongoDB data. See <\/span><a href=\"http:\/\/www.iri.com\/blog\/data-protection\/direct-data-masking-for-mongodb\/\"><span style=\"font-weight: 400;\">this example<\/span><\/a><span style=\"font-weight: 400;\"> of using Progress ODBC drivers for data masking using built-in \u201c<a href=\"https:\/\/www.iri.com\/products\/fieldshield\">FieldShield<\/a>\u201d functions. Voracity can also process BSON data directly via API through \/PROCESS=MongoDB support in CoSort v10, now too.<\/span><\/p>\n<p><b>Error Loading<\/b><\/p>\n<p><span style=\"font-weight: 400;\">This decision block checks the variable <\/span><i><span style=\"font-weight: 400;\">ERRORLEVEL<\/span><\/i><span style=\"font-weight: 400;\"> to make sure it returned 0 (or success) after importing into MongoDB. If it did not, the job continues to the<\/span><i><span style=\"font-weight: 400;\"> Delete-Changes<\/span><\/i><span style=\"font-weight: 400;\"> and <\/span><i><span style=\"font-weight: 400;\">EXIT<\/span><\/i><span style=\"font-weight: 400;\"> blocks where the job is terminated. If it returns true, the job continues to the next block.<\/span><\/p>\n<h3><b>Truncate Current<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">This command block truncates the file <\/span><i><span style=\"font-weight: 400;\">current.txt<\/span><\/i><span style=\"font-weight: 400;\">. This is to clear out the records that were loaded into MongoDB. If the import failed and the block above exited the job, then these changed records are appended to on the next pass. Then, as the job repeated, they would be loaded into MongoDB with the next group of changed records. <\/span><\/p>\n<h3><b>Delete Changes<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">This command block deletes <\/span><i><span style=\"font-weight: 400;\">changes.csv<\/span><\/i><span style=\"font-weight: 400;\"> so that the next pass is started with a newly created file for the pass. <\/span><\/p>\n<h3><b>Batch File<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">The batch file and transformation scripts are created when the Flow diagram is exported. A copy of the batch file is below. Each block adds executable lines to the batch file.<\/span><\/p>\n<p><a href=\"http:\/\/www.iri.com\/devblog\/wp-content\/uploads\/2017\/11\/batch-file-mongo.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-11906\" src=\"http:\/\/www.iri.com\/devblog\/wp-content\/uploads\/2017\/11\/batch-file-mongo.png\" alt=\"batch file mongo\" width=\"600\" height=\"496\" srcset=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/batch-file-mongo.png 679w, https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/batch-file-mongo-300x248.png 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/a><\/p>\n<h3><b>Task Scheduler<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">Using the Windows Task Scheduler, this batch file can be executed repeatedly to capture the changes in the source database.<\/span><\/p>\n<h3><b>Conclusion<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">With a little planning, and the use of command blocks, changes to a database table can be detected automatically using a batch file, and then scheduled to run at selected intervals.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Contact <\/span><a href=\"mailto:voracity@iri.com\"><span style=\"font-weight: 400;\">voracity@iri.com<\/span><\/a><span style=\"font-weight: 400;\"> or your IRI representative for more information or help with your use case<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Detecting additions and updates to database tables for data replication, ETL, PII masking, and other incremental data movement and manipulation activities can be automated in IRI Voracity workflows designed and run in IRI Workbench (WB). This article explains how to regularly check for changes in Oracle source tables to decide when to move data into<\/p>\n<div><a class=\"btn-filled btn\" href=\"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/\" title=\"Detecting Incremental Database Changes (Oracle to MongoDB ETL)\">Read More<\/a><\/div>\n","protected":false},"author":43,"featured_media":11908,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[31,776,90,3],"tags":[77,51,100,789,533,1427,50,1706,24],"class_list":["post-11905","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-migration","category-etl","category-migration","category-vldb-operations","tag-data-migration-2","tag-data-replication","tag-etl","tag-iri-voracity","tag-mongodb","tag-mongodb-database-masking","tag-oracle","tag-oracle-data-replication","tag-vldb"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v23.4 (Yoast SEO v23.4) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Detecting Incremental Database Changes (Oracle to MongoDB ETL) - IRI<\/title>\n<meta name=\"description\" content=\"Detect database changes in Oracle source tables to decide when to move data into a MongoDB target. IRI Voracity automates this process for ETL and more.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Detecting Incremental Database Changes (Oracle to MongoDB ETL)\" \/>\n<meta property=\"og:description\" content=\"Detect database changes in Oracle source tables to decide when to move data into a MongoDB target. IRI Voracity automates this process for ETL and more.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/\" \/>\n<meta property=\"og:site_name\" content=\"IRI\" \/>\n<meta property=\"article:published_time\" content=\"2017-11-22T13:46:10+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-08-18T19:06:59+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1252\" \/>\n\t<meta property=\"og:image:height\" content=\"1072\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Claudia Irvine\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Claudia Irvine\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/\"},\"author\":{\"name\":\"Claudia Irvine\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/72af50bbb317610e193e96392081f9b0\"},\"headline\":\"Detecting Incremental Database Changes (Oracle to MongoDB ETL)\",\"datePublished\":\"2017-11-22T13:46:10+00:00\",\"dateModified\":\"2022-08-18T19:06:59+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/\"},\"wordCount\":1227,\"commentCount\":1,\"publisher\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo.png\",\"keywords\":[\"data migration\",\"data replication\",\"ETL\",\"IRI Voracity\",\"MongoDB\",\"mongodb database masking\",\"Oracle\",\"oracle data replication\",\"VLDB\"],\"articleSection\":[\"Data Migration\",\"ETL\",\"Migration\",\"VLDB\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/\",\"url\":\"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/\",\"name\":\"Detecting Incremental Database Changes (Oracle to MongoDB ETL) - IRI\",\"isPartOf\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo.png\",\"datePublished\":\"2017-11-22T13:46:10+00:00\",\"dateModified\":\"2022-08-18T19:06:59+00:00\",\"description\":\"Detect database changes in Oracle source tables to decide when to move data into a MongoDB target. IRI Voracity automates this process for ETL and more.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#primaryimage\",\"url\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo.png\",\"contentUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo.png\",\"width\":1252,\"height\":1072,\"caption\":\"transform mapping diagram mongodb\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.iri.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Detecting Incremental Database Changes (Oracle to MongoDB ETL)\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.iri.com\/blog\/#website\",\"url\":\"https:\/\/www.iri.com\/blog\/\",\"name\":\"IRI\",\"description\":\"Total Data Management Blog\",\"publisher\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.iri.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\",\"name\":\"IRI\",\"url\":\"https:\/\/www.iri.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"contentUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"width\":750,\"height\":206,\"caption\":\"IRI\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/72af50bbb317610e193e96392081f9b0\",\"name\":\"Claudia Irvine\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/fb33dee44756bd13ac23f1342c2bb7b2?s=96&d=blank&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/fb33dee44756bd13ac23f1342c2bb7b2?s=96&d=blank&r=g\",\"caption\":\"Claudia Irvine\"},\"url\":\"https:\/\/www.iri.com\/blog\/author\/claudiai\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Detecting Incremental Database Changes (Oracle to MongoDB ETL) - IRI","description":"Detect database changes in Oracle source tables to decide when to move data into a MongoDB target. IRI Voracity automates this process for ETL and more.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/","og_locale":"en_US","og_type":"article","og_title":"Detecting Incremental Database Changes (Oracle to MongoDB ETL)","og_description":"Detect database changes in Oracle source tables to decide when to move data into a MongoDB target. IRI Voracity automates this process for ETL and more.","og_url":"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/","og_site_name":"IRI","article_published_time":"2017-11-22T13:46:10+00:00","article_modified_time":"2022-08-18T19:06:59+00:00","og_image":[{"width":1252,"height":1072,"url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo.png","type":"image\/png"}],"author":"Claudia Irvine","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Claudia Irvine","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#article","isPartOf":{"@id":"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/"},"author":{"name":"Claudia Irvine","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/72af50bbb317610e193e96392081f9b0"},"headline":"Detecting Incremental Database Changes (Oracle to MongoDB ETL)","datePublished":"2017-11-22T13:46:10+00:00","dateModified":"2022-08-18T19:06:59+00:00","mainEntityOfPage":{"@id":"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/"},"wordCount":1227,"commentCount":1,"publisher":{"@id":"https:\/\/www.iri.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#primaryimage"},"thumbnailUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo.png","keywords":["data migration","data replication","ETL","IRI Voracity","MongoDB","mongodb database masking","Oracle","oracle data replication","VLDB"],"articleSection":["Data Migration","ETL","Migration","VLDB"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/","url":"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/","name":"Detecting Incremental Database Changes (Oracle to MongoDB ETL) - IRI","isPartOf":{"@id":"https:\/\/www.iri.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#primaryimage"},"image":{"@id":"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#primaryimage"},"thumbnailUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo.png","datePublished":"2017-11-22T13:46:10+00:00","dateModified":"2022-08-18T19:06:59+00:00","description":"Detect database changes in Oracle source tables to decide when to move data into a MongoDB target. IRI Voracity automates this process for ETL and more.","breadcrumb":{"@id":"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#primaryimage","url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo.png","contentUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo.png","width":1252,"height":1072,"caption":"transform mapping diagram mongodb"},{"@type":"BreadcrumbList","@id":"https:\/\/www.iri.com\/blog\/etl\/db-changes-oracle-mongo\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.iri.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Detecting Incremental Database Changes (Oracle to MongoDB ETL)"}]},{"@type":"WebSite","@id":"https:\/\/www.iri.com\/blog\/#website","url":"https:\/\/www.iri.com\/blog\/","name":"IRI","description":"Total Data Management Blog","publisher":{"@id":"https:\/\/www.iri.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.iri.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.iri.com\/blog\/#organization","name":"IRI","url":"https:\/\/www.iri.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","contentUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","width":750,"height":206,"caption":"IRI"},"image":{"@id":"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/72af50bbb317610e193e96392081f9b0","name":"Claudia Irvine","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/fb33dee44756bd13ac23f1342c2bb7b2?s=96&d=blank&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/fb33dee44756bd13ac23f1342c2bb7b2?s=96&d=blank&r=g","caption":"Claudia Irvine"},"url":"https:\/\/www.iri.com\/blog\/author\/claudiai\/"}]}},"jetpack_featured_media_url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/11\/transform-mapping-diagram-mongo.png","_links":{"self":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/11905"}],"collection":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/users\/43"}],"replies":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/comments?post=11905"}],"version-history":[{"count":7,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/11905\/revisions"}],"predecessor-version":[{"id":16271,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/11905\/revisions\/16271"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/media\/11908"}],"wp:attachment":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/media?parent=11905"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/categories?post=11905"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/tags?post=11905"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}