{"id":9868,"date":"2016-06-20T13:12:25","date_gmt":"2016-06-20T17:12:25","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=9868"},"modified":"2024-10-29T16:20:23","modified_gmt":"2024-10-29T20:20:23","slug":"using-marklogic-data-in-iri-voracity","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/","title":{"rendered":"Managing MarkLogic Data in IRI Voracity"},"content":{"rendered":"<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">The <\/span><a style=\"color: #1155cc; font-size: 10pt; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/products\/voracity\">IRI Voracity<\/a><span style=\"font-size: 10pt;\">\u00a0data management platform now supports the MarkLogic NoSQL database as a source for structured data\u00a0<em>discovery<\/em> (classification, profiling, and search), <em>integration<\/em> (ETL, CDC, SCD), <em>migration\u00a0<\/em>(conversion and replication , <em>governance<\/em> (data cleansing and masking), and <em>analytic<\/em> (reporting and wrangling) jobs. In this article, I explain how to set up the MarkLogic server for SQL operations, and configure Voracity to source MarkLogic data via ODBC in <\/span><a style=\"color: #1155cc; font-size: 10pt; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/products\/workbench\/voracity-gui\">IRI Workbench<\/a><span style=\"font-size: 10pt;\">.<br \/>\n<\/span><\/span><\/p>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">MarkLogic Server is the Enterprise NoSQL Database that combines database internals, search-style indexing, and application server behaviors. It uses XML documents as its data model, and stores the documents within a fully ACID compliant transactional repository. It indexes the words and values from each of the loaded documents, as well as the document structure. And, because of its unique Universal Index, MarkLogic doesn&#8217;t require advance knowledge of the document structure (its &#8220;schema&#8221;), nor complete adherence to a particular schema. Through its application server capabilities, it is programmable and extensible. <\/span><\/span><\/p>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">To set up the MarkLogic server for ODBC access, I need to create:<\/span><\/span><\/p>\n<ul>\n<li><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">a SQL database<\/span><\/span><\/li>\n<li><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">range indexes for the database<\/span><\/span><\/li>\n<li><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">data fields<\/span><\/span><\/li>\n<li><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">an ODBC apps server<\/span><\/span><\/li>\n<\/ul>\n<h3 style=\"font-weight: bold;\"><span style=\"font-weight: bold;\"><span style=\"font-weight: bold;\">Set up MarkLogic Server<\/span><\/span><\/h3>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">Install MarkLogic on your network and reach it through the browser in IRI Workbench for convenience. Select Windows=&gt;Show View=&gt;Other=&gt;Internal Web browser and navigate to http:\/\/hostname:8001:<\/span><\/span><\/p>\n<p style=\"font-style: italic; font-weight: bold;\"><a title=\"Using MarkLogic Data in IRI Voracity - Chaitail Mitra 0.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-0.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 624px; height: 390px;\" title=\"MarkLogic in the IRI Workbench\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-0.jpg\" alt=\"MarkLogic in the IRI Workbench\" width=\"624\" height=\"390\" \/><\/a><\/p>\n<h3 style=\"font-weight: bold;\"><span style=\"font-weight: bold;\"><span style=\"font-weight: bold;\">Configure ODBC Server &amp; Create the Database<\/span><\/span><\/h3>\n<p><span style=\"font-size: 10pt;\">To create a SQL-ready database<\/span> in MarkLogic, <span style=\"font-size: 10pt;\">the first Configure tab step is to create a \u201cforest\u201d and attach it to the database, which I named SQLData1. I then created an <\/span><span style=\"font-size: 10pt; font-weight: bold; font-style: italic;\">ODBC Server<\/span><span style=\"font-size: 10pt;\"> (shown below, via Groups, Apps Server) named SQL with <\/span><span style=\"font-size: 10pt; font-weight: bold; font-style: italic;\">Port<\/span><span style=\"font-size: 10pt;\"> number 5432. In the <\/span><span style=\"font-size: 10pt; font-weight: bold; font-style: italic;\">modules <\/span><span style=\"font-size: 10pt;\">field, select (file system) to store MarkLogic documents, and in the <\/span><span style=\"font-size: 10pt; font-weight: bold; font-style: italic;\">database <\/span><span style=\"font-size: 10pt;\">field, select the SQLData1 database we created.<\/span><\/p>\n<p style=\"font-size: 10pt;\" align=\"center\"><a title=\"Using MarkLogic Data in IRI Voracity - Chaitail Mitra 1.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-1.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 533px; height: 325px;\" title=\"MarkLogic Server\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-1.jpg\" alt=\"MarkLogic Server\" width=\"533\" height=\"325\" \/><\/a><\/p>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">Click OK to save these settings. Next, click to expand Databases in the explorer pane, and under SQLData1, create<\/span><span style=\"font-size: 10pt; font-weight: bold; font-style: italic;\"> range element indexes <\/span><span style=\"font-size: 10pt;\">to define each column name and data type for use in multiple tables within a schema we will later call \u201cmain\u201d:<\/span><\/span><\/p>\n<h3 style=\"font-weight: bold;\"><span style=\"font-weight: bold;\"><span style=\"font-weight: bold;\">Creating Tables (Views) in MarkLogic<\/span><\/span><\/h3>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">Given that we have previously defined columns for use, we can assign them to a new schema which will have a series of defined tables or views. To create the schema, use a Curl command like this:<\/span><\/span><\/p>\n<p style=\"font-size: 8pt; font-weight: bold; font-family: Courier New;\" align=\"center\"><span style=\"font-size: 8pt; font-weight: bold; font-family: Courier New;\"><span style=\"font-family: Courier New; font-size: 8pt; font-weight: bold;\">curl -X POST &#8211;anyauth &#8211;user admin:admin &#8211;header &#8220;Content-Type:application\/json&#8221; -d &#8216;{&#8220;view-schema-name&#8221;: &#8220;main&#8221;}&#8217; http:\/\/localhost:8002\/manage\/v2\/databases\/SQLData1\/view-schemas\/?format=json<\/span><\/span><\/p>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">Once I create the schema \u2018main\u2019 I will create a view called \u2018emps\u2019, which contain some of the previously defined range element index IDs (or columns); e.g., \u2018firstname\u2019, \u2018lastname\u2019, and \u2018employeeid\u2019 range indexes.<\/span><span style=\"font-size: 10pt; font-style: italic;\"> Employeeid <\/span><span style=\"font-size: 10pt;\">uses the integer data type, while <\/span><span style=\"font-size: 10pt; font-style: italic;\">FirstName, LastName<\/span><span style=\"font-size: 10pt;\"> use a string.<\/span><\/span><\/p>\n<p style=\"font-size: 10pt; font-style: italic; font-weight: bold;\"><span style=\"font-size: 10pt; font-style: italic; font-weight: bold;\"><span style=\"font-size: 10pt; font-weight: bold; font-style: italic;\">Curl Code in Cygwin prompt<\/span><\/span><\/p>\n<p style=\"font-size: 10pt;\"><a title=\"Using MarkLogic Data in IRI Voracity - Chaitail Mitra 3.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-3.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 624px; height: 384px;\" title=\"Curl Code in Cygwin prompt\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-3.jpg\" alt=\"Curl Code in Cygwin prompt\" width=\"624\" height=\"384\" \/><\/a><\/p>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">Through these views, SQL inserts and queries via ODBC will work in MarkLogic\u2019s Query Console (below), and thus, operations on this data in IRI Voracity as well. For more detailed instructions in this area, refer to\u00a0<\/span><a style=\"color: #1155cc; font-size: 10pt; text-decoration: underline;\" href=\"https:\/\/docs.marklogic.com\/guide\/sql\/setup\" target=\"_blank\" rel=\"noopener\">https:\/\/docs.marklogic.com\/guide\/sql\/setup<\/a><span style=\"font-size: 10pt; font-weight: bold; font-style: italic;\">.<\/span><\/span><\/p>\n<h3 style=\"font-weight: bold;\"><span style=\"font-weight: bold;\"><span style=\"font-weight: bold;\">Loading &amp; displaying data in MarkLogic Query Console <\/span><\/span><\/h3>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">In the IRI Workbench internal browser, I can access the MarkLogic Query Console to do ad hoc queries<\/span><span style=\"color: #00796b; font-size: 10pt;\">, <\/span><span style=\"font-size: 10pt;\">insert XML or JSON documents, or RDF Triples. In this case, I will use it to enter (load \/ insert) the actual data elements into my now SQL\/ODBC-ready view, emps, via JavaScript. Each row is stored as a JSON document in this case, and can be queried with SQL syntax.<\/span><\/span><\/p>\n<p style=\"font-size: 10pt;\"><a title=\"Using MarkLogic Data in IRI Voracity - Chaitail Mitra 4.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-4.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 624px; height: 392px;\" title=\"Loading &amp; displaying data in MarkLogic Query Console\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-4.jpg\" alt=\"Loading &amp; displaying data in MarkLogic Query Console\" width=\"624\" height=\"392\" \/><\/a><\/p>\n<h3 style=\"font-weight: bold;\"><span style=\"font-weight: bold;\"><span style=\"font-weight: bold;\">ODBC connection in IRI Workbench<br \/>\n<\/span><\/span><\/h3>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">Once the backend DB is configured, we must configure its ODBC driver for use with Voracity. From the IRI Workbench, I click on the toolbar\u2019s IRI icon, and select Data Connection Registry. From there, click add:<\/span><\/span><\/p>\n<p style=\"font-size: 10pt;\"><a title=\"Using MarkLogic Data in IRI Voracity - Chaitail Mitra 5.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-5.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 624px; height: 373px;\" title=\"Data Connection Registry\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-5.jpg\" alt=\"Data Connection Registry\" width=\"624\" height=\"373\" \/><\/a><\/p>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">From the ODBC Data Source Administrator window, use the System DSN tab and Configure \u2026 to enter the connection parameters to MarkLogic. In the MarkLogic SQL ODBC Driver Setup window, enter the database name we created (in this case SQLData1). The server name is localhost, and username and password match what\u2019s in use with the MarkLogic server and port (5432). Test and save the connection.<\/span><\/span><\/p>\n<p style=\"font-size: 10pt;\"><a title=\"Using MarkLogic Data in IRI Voracity - Chaitail Mitra 6.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-6.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 624px; height: 380px;\" title=\"ODBC Data Source Administrator\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-6.jpg\" alt=\"ODBC Data Source Administrator\" width=\"624\" height=\"380\" \/><\/a><\/p>\n<h3 style=\"font-weight: bold;\"><span style=\"font-weight: bold;\"><span style=\"font-weight: bold;\">Retrieve data from MarkLogic (View) &amp; Load in Oracle<br \/>\n<\/span><\/span><\/h3>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">I next need to create an IRI data definition file (DDF) to make use of the MarkLogic data in each view. To do this in the IRI Workbench GUI for Voracity (or other IRI products using DDF), I will use the <\/span><span style=\"font-size: 10pt; font-style: italic;\">Import Metadata Wizard. <\/span><span style=\"font-size: 10pt;\"> First, I create a New IRI Project in the Workbench Project Explorer to hold my work:<br \/>\n<\/span><\/span><\/p>\n<p style=\"font-size: 10pt; font-weight: bold;\" align=\"center\"><a title=\"Using MarkLogic Data in IRI Voracity - Chaitail Mitra 7.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-7.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 300px; height: 285px;\" title=\"Create an IRI Workbench Project\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-7.jpg\" alt=\"Create an IRI Workbench Project\" width=\"300\" height=\"285\" \/><\/a><\/p>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">Next, from the IRI Menu=&gt; Select Import Table metadata. Select the Data Source Name (DSN) \u201cMarkLogicSQL\u201d and the table \u201cmain.emps\u201d:<\/span><\/span><\/p>\n<p style=\"font-size: 10pt; font-style: italic; font-weight: bold;\" align=\"center\"><a title=\"Using MarkLogic Data in IRI Voracity - Chaitail Mitra 8.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-8.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 476px; height: 400px;\" title=\"Import Table metadata\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-8.jpg\" alt=\"Import Table metadata\" width=\"476\" height=\"400\" \/><\/a><\/p>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">The resulting DDF file is shown below; note that my connection to MarkLogic must remain open while I\u2019m interfacing with it:<\/span><\/span><\/p>\n<p style=\"font-size: 10pt;\" align=\"center\"><a title=\"Using MarkLogic Data in IRI Voracity - Chaitail Mitra 9.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-9.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 465px; height: 392px;\" title=\"Resulting DDF file \" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-9.jpg\" alt=\"Resulting DDF file \" width=\"465\" height=\"392\" \/><\/a><\/p>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">This DDF is now available for use in any IRI job script sourcing this table. I will use it in my sort and mask application below.<\/span><\/span><\/p>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">From the CoSort toolbar menu (stopwatch icon), select <\/span><span style=\"font-size: 10pt; font-style: italic;\">New Sort Job<\/span><span style=\"font-size: 10pt;\">. After naming my job script, I am taken to the data source specification. I locate my ODBC source for the MarkLogic table, and then select<\/span><span style=\"font-size: 10pt; font-style: italic;\"> Add Existing Metadata <\/span><span style=\"font-size: 10pt;\">to provide the necessary field layouts for the CoSort program. Voracity uses SortCL to manipulate, mask, and report on MarkLogic and other ODBC and file-based <\/span><a style=\"color: #1155cc; font-size: 10pt; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/products\/workbench\/data-sources\">data sources<\/a><span style=\"font-size: 10pt;\">.<\/span><\/span><\/p>\n<p style=\"font-size: 10pt;\" align=\"center\"><a title=\"Using MarkLogic Data in IRI Voracity - Chaitail Mitra 10.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-10.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 274px; height: 310px;\" title=\"New Sort Job\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-10.jpg\" alt=\"New Sort Job\" width=\"274\" height=\"310\" \/><\/a><\/p>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">I can then specify one or more sort keys:<br \/>\n<\/span><\/span><\/p>\n<p style=\"font-size: 10pt;\" align=\"center\"><a title=\"Using MarkLogic Data in IRI Voracity - Chaitail Mitra 11.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-11.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 285px; height: 321px;\" title=\"New Sort Job - more sort keys\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-11.jpg\" alt=\"New Sort Job - more sort keys\" width=\"285\" height=\"321\" \/><\/a><\/p>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">In the next screen I define and format my target(s), where I also specified a redaction rule to mask sensitive portions of the column values on output:<\/span><\/span><\/p>\n<p style=\"font-size: 10pt;\" align=\"center\"><a title=\"Using MarkLogic Data in IRI Voracity - Chaitail Mitra 12.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-12.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 421px; height: 317px;\" title=\"Define and format target(s)\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-12.jpg\" alt=\"Define and format target(s)\" width=\"421\" height=\"317\" \/><\/a><\/p>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">I also redacted the <\/span><span style=\"font-family: Consolas; font-size: 10pt;\">FIRSTNAME <\/span><span style=\"font-size: 10pt;\">and <\/span><span style=\"font-family: Consolas; font-size: 10pt;\">LASTNAME <\/span><span style=\"font-size: 10pt;\">column values with the <\/span><span style=\"font-family: Consolas; font-size: 10pt;\">replace_chars(FIRSTNAME, <\/span><span style=\"color: #2a00ff; font-family: Consolas; font-size: 10pt;\">&#8220;*&#8221;<\/span><span style=\"font-size: 10pt;\">). Protection rule. See <\/span><a style=\"color: #1155cc; font-size: 10pt; text-decoration: underline;\" href=\"http:\/\/www.iri.com\/blog\/vlog\/basic-data-masking-and-encryption-demonstration\/\">this video<\/a><span style=\"font-size: 10pt;\"> on how to use IRI Workbench dialogs (or wizards) to redact data and otherwise mask sensitive data in your target fields.<\/span><\/span><\/p>\n<h3 style=\"font-weight: bold;\"><span style=\"font-weight: bold;\"><span style=\"font-weight: bold;\">The code and the output produced in IRI Workbench<\/span><\/span><\/h3>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">The job produced in the wizard connects to MarkLogic, sorts and masks the data in the main.emps view, and sends the output to both an Oracle DB and flat-file (standard output) target, both shown below:<\/span><\/span><\/p>\n<p style=\"font-size: 10pt;\"><a title=\"Using MarkLogic Data in IRI Voracity - Chaitail Mitra 13.jpg\" href=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-13.jpg\" rel=\"prettyPhoto\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" style=\"width: 624px; height: 389px;\" title=\"The code and the output produced in IRI Workbench\" src=\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/t_Using-MarkLogic-Data-in-IRI-Voracity-Chaitail-Mitra-13.jpg\" alt=\"The code and the output produced in IRI Workbench\" width=\"624\" height=\"389\" \/><\/a><\/p>\n<p style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\"><span style=\"font-size: 10pt;\">See <a href=\"https:\/\/www.iri.com\/blog\/test-data\/building-loading-big-test-data-into-marklogic\/\">this article<\/a> if you are interested in loading bulk (test) data files generated by Voracity (or IRI RowGen) into MarkLogic. If your data in MarkLogic is unstructured and you need to search, extract or mask its XML (or JSON) exports, you&#8217;d use DarkShield (also in Voracity) per <a href=\"https:\/\/www.iri.com\/blog\/data-protection\/finding-and-masking-pii-in-files-with-the-darkshield-files-wizard\/\">this article<\/a>.\u00a0If you have any questions or need help using MarkLogic as a Voracity data source, contact <a href=\"mailto:voracity@iri.com\">voracity@iri.com<\/a>. <\/span><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>The IRI Voracity\u00a0data management platform now supports the MarkLogic NoSQL database as a source for structured data\u00a0discovery (classification, profiling, and search), integration (ETL, CDC, SCD), migration\u00a0(conversion and replication , governance (data cleansing and masking), and analytic (reporting and wrangling) jobs. In this article, I explain how to set up the MarkLogic server for SQL operations,<\/p>\n<div><a class=\"btn-filled btn\" href=\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/\" title=\"Managing MarkLogic Data in IRI Voracity\">Read More<\/a><\/div>\n","protected":false},"author":53,"featured_media":18083,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[31,776,91],"tags":[1102,1100,561,1099,546,789,850,1106,1104,1361,1097,1103,75,1105,1101,1098,550,1955],"class_list":["post-9868","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-migration","category-etl","category-iri-workbench","tag-application-server","tag-database-internals","tag-ddf","tag-enterprise-nosql-database","tag-iri-cosort","tag-iri-voracity","tag-iri-workbench","tag-javascript","tag-json","tag-json-etl","tag-marklogic","tag-marklogic-query-console","tag-odbc","tag-rdf-triples","tag-search-style-indexing","tag-sql-operations","tag-xml","tag-xml-data-integration"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v23.4 (Yoast SEO v23.4) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Managing MarkLogic Data in IRI Voracity - IRI<\/title>\n<meta name=\"description\" content=\"Learn how to integrate, migrate, cleanse, mask, and report on data in the MarkLogic NoSQL DB using the Voracity data management platform.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Managing MarkLogic Data in IRI Voracity\" \/>\n<meta property=\"og:description\" content=\"Learn how to integrate, migrate, cleanse, mask, and report on data in the MarkLogic NoSQL DB using the Voracity data management platform.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/\" \/>\n<meta property=\"og:site_name\" content=\"IRI\" \/>\n<meta property=\"article:published_time\" content=\"2016-06-20T17:12:25+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-10-29T20:20:23+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-voracity-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1110\" \/>\n\t<meta property=\"og:image:height\" content=\"532\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Chaitali Mitra\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Chaitali Mitra\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/\"},\"author\":{\"name\":\"Chaitali Mitra\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/9bae14a309616863b027c2d56f532caf\"},\"headline\":\"Managing MarkLogic Data in IRI Voracity\",\"datePublished\":\"2016-06-20T17:12:25+00:00\",\"dateModified\":\"2024-10-29T20:20:23+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/\"},\"wordCount\":1083,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-voracity-1.png\",\"keywords\":[\"application server\",\"database internals\",\"DDF\",\"Enterprise NoSQL Database\",\"IRI CoSort\",\"IRI Voracity\",\"IRI Workbench\",\"JavaScript\",\"JSON\",\"json etl\",\"MarkLogic\",\"MarkLogic Query Console\",\"ODBC\",\"RDF Triples\",\"search-style indexing\",\"SQL operations\",\"xml\",\"XML data integration\"],\"articleSection\":[\"Data Migration\",\"ETL\",\"IRI Workbench\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/\",\"url\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/\",\"name\":\"Managing MarkLogic Data in IRI Voracity - IRI\",\"isPartOf\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-voracity-1.png\",\"datePublished\":\"2016-06-20T17:12:25+00:00\",\"dateModified\":\"2024-10-29T20:20:23+00:00\",\"description\":\"Learn how to integrate, migrate, cleanse, mask, and report on data in the MarkLogic NoSQL DB using the Voracity data management platform.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/#primaryimage\",\"url\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-voracity-1.png\",\"contentUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-voracity-1.png\",\"width\":1110,\"height\":532},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.iri.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Managing MarkLogic Data in IRI Voracity\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.iri.com\/blog\/#website\",\"url\":\"https:\/\/www.iri.com\/blog\/\",\"name\":\"IRI\",\"description\":\"Total Data Management Blog\",\"publisher\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.iri.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\",\"name\":\"IRI\",\"url\":\"https:\/\/www.iri.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"contentUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"width\":750,\"height\":206,\"caption\":\"IRI\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/9bae14a309616863b027c2d56f532caf\",\"name\":\"Chaitali Mitra\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/95a11f3d0b709c00df3262bab0152f3a?s=96&d=blank&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/95a11f3d0b709c00df3262bab0152f3a?s=96&d=blank&r=g\",\"caption\":\"Chaitali Mitra\"},\"sameAs\":[\"http:\/\/www.iri.com\"],\"url\":\"https:\/\/www.iri.com\/blog\/author\/chaitalim\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Managing MarkLogic Data in IRI Voracity - IRI","description":"Learn how to integrate, migrate, cleanse, mask, and report on data in the MarkLogic NoSQL DB using the Voracity data management platform.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/","og_locale":"en_US","og_type":"article","og_title":"Managing MarkLogic Data in IRI Voracity","og_description":"Learn how to integrate, migrate, cleanse, mask, and report on data in the MarkLogic NoSQL DB using the Voracity data management platform.","og_url":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/","og_site_name":"IRI","article_published_time":"2016-06-20T17:12:25+00:00","article_modified_time":"2024-10-29T20:20:23+00:00","og_image":[{"width":1110,"height":532,"url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-voracity-1.png","type":"image\/png"}],"author":"Chaitali Mitra","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Chaitali Mitra","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/#article","isPartOf":{"@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/"},"author":{"name":"Chaitali Mitra","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/9bae14a309616863b027c2d56f532caf"},"headline":"Managing MarkLogic Data in IRI Voracity","datePublished":"2016-06-20T17:12:25+00:00","dateModified":"2024-10-29T20:20:23+00:00","mainEntityOfPage":{"@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/"},"wordCount":1083,"commentCount":0,"publisher":{"@id":"https:\/\/www.iri.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/#primaryimage"},"thumbnailUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-voracity-1.png","keywords":["application server","database internals","DDF","Enterprise NoSQL Database","IRI CoSort","IRI Voracity","IRI Workbench","JavaScript","JSON","json etl","MarkLogic","MarkLogic Query Console","ODBC","RDF Triples","search-style indexing","SQL operations","xml","XML data integration"],"articleSection":["Data Migration","ETL","IRI Workbench"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/","url":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/","name":"Managing MarkLogic Data in IRI Voracity - IRI","isPartOf":{"@id":"https:\/\/www.iri.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/#primaryimage"},"image":{"@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/#primaryimage"},"thumbnailUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-voracity-1.png","datePublished":"2016-06-20T17:12:25+00:00","dateModified":"2024-10-29T20:20:23+00:00","description":"Learn how to integrate, migrate, cleanse, mask, and report on data in the MarkLogic NoSQL DB using the Voracity data management platform.","breadcrumb":{"@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/#primaryimage","url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-voracity-1.png","contentUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-voracity-1.png","width":1110,"height":532},{"@type":"BreadcrumbList","@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/using-marklogic-data-in-iri-voracity\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.iri.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Managing MarkLogic Data in IRI Voracity"}]},{"@type":"WebSite","@id":"https:\/\/www.iri.com\/blog\/#website","url":"https:\/\/www.iri.com\/blog\/","name":"IRI","description":"Total Data Management Blog","publisher":{"@id":"https:\/\/www.iri.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.iri.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.iri.com\/blog\/#organization","name":"IRI","url":"https:\/\/www.iri.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","contentUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","width":750,"height":206,"caption":"IRI"},"image":{"@id":"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/9bae14a309616863b027c2d56f532caf","name":"Chaitali Mitra","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/95a11f3d0b709c00df3262bab0152f3a?s=96&d=blank&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/95a11f3d0b709c00df3262bab0152f3a?s=96&d=blank&r=g","caption":"Chaitali Mitra"},"sameAs":["http:\/\/www.iri.com"],"url":"https:\/\/www.iri.com\/blog\/author\/chaitalim\/"}]}},"jetpack_featured_media_url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2016\/06\/marklogic-voracity-1.png","_links":{"self":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/9868"}],"collection":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/users\/53"}],"replies":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/comments?post=9868"}],"version-history":[{"count":17,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/9868\/revisions"}],"predecessor-version":[{"id":18061,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/9868\/revisions\/18061"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/media\/18083"}],"wp:attachment":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/media?parent=9868"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/categories?post=9868"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/tags?post=9868"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}