{"id":13061,"date":"2019-08-08T17:51:48","date_gmt":"2019-08-08T21:51:48","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=13061"},"modified":"2026-02-23T17:08:37","modified_gmt":"2026-02-23T22:08:37","slug":"mongodb-cassandra-darkshield","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/data-protection\/mongodb-cassandra-darkshield\/","title":{"rendered":"Masking PII in MongoDB, Cassandra, and Elasticsearch with DarkShield: 4th IRI Method"},"content":{"rendered":"

Editor’s Note:<\/strong><\/em><\/p>\n

As of 2024 and DarkShield V5, the search and masking methods for these NoSQL DBs has been upgraded; please refer to the discussion and demonstration in this updated article<\/a> instead!<\/em><\/p>\n

This article demonstrates the use of <\/span>IRI DarkShield<\/span><\/a> to identify and remediate (<\/span>mask<\/span><\/a>) personally identifiable information (PII) and other sensitive data in MongoDB, Cassandra, and Elasticsearch databases. Although these steps mainly focus on finding and shielding data in MongoDB collections, the same steps can be used for data in Cassandra tables as well. See this article<\/a> on Elasticsearch, too.<\/span><\/p>\n

Note that this article represents the <\/span>fouth method<\/span><\/a> IRI supports for masking data in MongoDB, and the <\/span>second method<\/span><\/a> for Cassandra. Those previous\u00a0 and still supported methods rely on structured data discovery and de-identification via\u00a0<\/span>IRI FieldShield<\/span><\/a>, while the DarkShield method supports textual data in either structured or unstructured collections.<\/span>1<\/sup><\/a><\/span> Though DarkShield and FieldShield are standalone IRI data masking products, both are included in the IRI Voracity<\/a> data management platform.<\/span><\/p>\n

The latest approach uses some elements of <\/span>Data Classification<\/span><\/i>, an integrated data cataloging paradigm for defining the search methods used for finding PII independently from the source of the data. While this article provides a small introduction to Data Classification during Step 1, you may find it useful to read up on how Data Classification fits into our unified approach to conducting searches. For more information on Data Classification in the <\/span>IRI Workbench<\/span><\/a> front end for DarkShield et al<\/span>2<\/sup><\/a><\/span><\/span>, please read <\/span>this article<\/span><\/a> before proceeding.<\/span><\/p>\n

The identification and remediation of PII using IRI DarkShield involves 4 general steps:<\/span><\/p>\n

(Optional) Step – Register Your Data Source(s)<\/b><\/p>\n

In this (optional) step, data sources for a Mongo database, Cassandra keyspace, or Elasticsearch cluster are registered. This allows data sources to be reusable. As a result, this step is optional if the desired data source already exists in the registry, or if you plan on defining them from the wizard.<\/span><\/p>\n

Step 1 – Specify Search Parameters<\/b><\/p>\n

Here all aspects of a search are chosen. First, a source and target collection\/table are set up based on the data connection specified in the registry or created in the wizard. Then, you can specify the search and remediation criteria for your data using Search Matchers the kinds of information to look for and how that information should be remediated. The result of this step is a .<\/span>search<\/span><\/i> file.<\/span><\/p>\n

Step 2 – Conduct a Search<\/b><\/p>\n

A search can be run from a <\/span>.search<\/span><\/i> file. The result is a <\/span>.darkdata<\/span><\/i> file annotating any identified PII.<\/span><\/p>\n

Step 3 – Remediation (Masking)<\/b><\/p>\n

Remediation can be done from a <\/span>.darkdata<\/span><\/i> file. Any identified PII will be remediated in the manner specified during search creation.<\/span><\/p>\n

(Optional) Step – Register Your Data Sources<\/h2>\n

As a prerequisite step, you will need to register the connections to your online data sources (and targets) in the URL Connection Registry, which is located in the <\/span>Preferences > IRI > URL Connection Registry <\/span><\/i>dialog in IRI Workbench. <\/span><\/p>\n

\"\"<\/a><\/p>\n

All URL Connections, including URI connection strings for MongoDB, Cassandra, and Elasticsearch can be saved. This allows for the URLs, authentication credentials, and any additional parameters to be saved and stored by IRI Workbench for future use.<\/span>3<\/sup><\/a><\/span><\/span><\/p>\n

Step 1 – Specify Search Parameters (Create .Search File)<\/h2>\n

In the <\/span>IRI Workbench<\/span><\/a> IDE for DarkShield, Select New Database Discovery Job from the DarkShield Menu.\u00a0<\/span>Select a project folder and enter a name for the job:<\/span><\/p>\n

\"\"<\/a><\/p>\n

Specifying a Source and Target<\/span><\/h4>\n

Any Mongo, Cassandra or ElasticsearchURLs that were previously created and saved in the registry can be accessed from the <\/span>URI <\/span><\/i>dropdown for both the Source and Target selectors. The name of the corresponding MongoDB collection, Cassandra table, or Elasticsearch index will also have to be entered:<\/span><\/p>\n

\"\"<\/a><\/p>\n

A new URI can also be created by pressing the <\/span>New <\/span><\/i>button. This will open the URL Connection Details dialog. Enter a name for the connection, select the desired scheme, enter the host, and enter the database. If no port is present the default port for the scheme will be assumed.\u00a0<\/span><\/p>\n

A username and password can also be supplied if the database requires authorization. Any new URL connections will be saved in the URL Connection Registry, and can be re-used as a target.\u00a0<\/span><\/p>\n

\"\"<\/a><\/p>\n

After a source is specified, you can continue to the next page to select or create a target URI. The scheme of the target URI will be limited to the selected source URI, so a MongoDB source can only be sent to another MongoDB target, and similarly for Cassandra or Elasticsearch.<\/span><\/p>\n

\"\"<\/a><\/p>\n

When a masking job is run, all rows in the source will be appended to the target, and any rows with matching keys will be overwritten. For Cassandra, ensure that the target table schema is compatible with data from the source table.<\/span><\/p>\n

Adding Search Matchers<\/span><\/h4>\n

After both a source and target are specified, you can go to the next page to add search matchers<\/span>.<\/span>4<\/sup><\/a><\/span> Select a library location containing any Patterns or Rules libraries which you wish to use and click <\/span>Add <\/span><\/i>to add a new Search Matcher.<\/span><\/p>\n

KeyNameMatcher<\/span><\/h4>\n

The first Search Matcher that we will create will be used to match the entire value corresponding to any \u201cname\u201d key located in arbitrarily nested json structures and apply a Format Preserving Encryption algorithm to mask it. We can achieve this by creating a JSON path filter \u201c$..name\u201d. More information about JSON\u00a0 paths can be found <\/span>here<\/span><\/a>.<\/span><\/p>\n

\"\"<\/a><\/p>\n

Since MongoDB collections, Cassandra tables, and Elasticsearch indexes are parsed by DarkShield as json documents, the filter can be applied to both in order to mask any value corresponding to any \u201cname\u201d key.\u00a0<\/span><\/p>\n

To match on the contents of the filtered data, we need to create a new <\/span>Data Class<\/span><\/i>. A Data Class represents PII and the associated matchers used to identify it. These matchers can include any combination of:<\/span><\/p>\n