Introduction to IRI DarkShield
What is DarkShield?
IRI DarkShield is a data masking tool for classifying, searching, and de-identifying Personally Identifiable Information (PII) and other sensitive data in structured, semi-structured, and unstructured sources. You can define and run DarkShield jobs through a GUI, CLI, or API.
DarkShield supports data on-premises or in the cloud. Supported sources include free-floating text, files, and documents, various NoSQL DBs , and any relational database connected through a JDBC driver.
As one of several data masking tools in the IRI Voracity platform and IRI Data Protector Suite , DarkShield supports a wide range of data formats and silos.1
In addition, the data classes and masking functions defined for DarkShield jobs are also used in IRI FieldShield and IRI CellShield EE. This helps secure sensitive data consistently across different data silos and formats.
How Does It Work?
You can configure, run, and share most DarkShield jobs through the Eclipse-based IRI Workbench GUI. For more information about the DarkShield front end, see:
https://www.iri.com/products/workbench/darkshield-gui
Under the hood, DarkShield runs one or more Remote Procedure Call (RPC) API engines, depending on the data source or sources involved. These engines perform the searching and masking operations through a server in your infrastructure called the Plankton Server. The server accepts HTTP(S) requests containing instructions for finding and masking PII.
In practice, you host the DarkShield API server either on-premises or on a cloud server that you manage. IRI does not maintain or manage these services, and PII does not leave your environment.
DarkShield sends its PII search methods to the applicable API in instructional JSON payloads called Search Contexts. In contrast, Mask Contexts provide instructions for masking the PII identified during a search.
Within IRI Workbench, Data Classes containing Data Matchers and Location Matchers define the Search Contexts. In turn, Masking Rules paired with Data Classes create the Mask Contexts. The consistent application of deterministic data masking functions helps preserve data and referential integrity across all target silos.
Additionally, provided alongside the Search and Mask Contexts are additional contexts that provide instructions on where to read data from (FileSearchContext, NoSqlSearchContext, RdbSearchContext) and where to write masked data to (FileMaskContext, NoSqlMaskContext, RdbMaskContext).
Pairing search methods with masking functions provides more granular control over how sensitive data is protected. For example, you can configure a DarkShield job to redact detected credit card numbers while applying a consistent pseudonym replacement to names wherever they are found.
Getting Started
You can license DarkShield as a standalone product, or as part of a bundle with IRI FieldShield or IRI CellShield EE in the IRI Voracity data management platform. Licensing and pricing options are on this page.
After licensing or evaluation begins, IRI will send you download and installation instructions. At installation time, you will need to register a provided version of the IRI CoSort engine, sortcl, to enable masking jobs. 
At runtime, the DarkShield API runs on a web server called Plankton, which listens for requests. These HTTP(S) payloads can contain instructions for finding and masking PII in the form of Search Contexts and Mask Contexts. Additionally, they can also contain instructions specifying where to access and read data from and where to write the masked results.
You can install DarkShield either as the backend engine (API) only or together with IRI Workbench (GUI). Before running any jobs, you must first start the DarkShield server. 2
You can start the DarkShield server through the:
- Shell or batch startup script in the DarkShield API distribution bin directory;
- Workbench Window > Preferences > IRI > DarkShield properties dialog; or
- DarkShield API Status view in your workspace
Connecting to Different Sources
Whether called directly or used through IRI Workbench, DarkShield uses different API plugins to access and manipulate different source types. These include streaming text (strings), standalone or embedded files, documents and images, NoSQL databases, and relational databases.
For file sources, DarkShield supports several formats, including:
- .doc, .docx, .xls, .xlsx, .ppt
- .csv, .tsv, and fixed-length files
- JSON and XML
- DICOM, JPG, PNG, GIF, and TIF
- Parquet
- HL7v2 and X12
For local deployments, DarkShield can access files stored on the local machine or LAN. In addition, from IRI Workbench, DarkShield can access files stored in cloud sources such as Azure Blob Storage, Google Cloud Storage, S3 Buckets, OneDrive, and SharePoint Online.
Read the articles on the DarkShield-Files plug-in (RPC API for files) or the New File Search/Masking Job … wizard in IRI Workbench to learn how to find and mask PII across many different file formats and silos at once.
For NoSQL databases, the DarkShield NoSQL wizard currently supports these sources:
- Cassandra
- Elasticsearch
- MongoDB
For more details, read the blog articles on the NoSQL plugin or the NoSQL wizard for DarkShield to learn more about setting up connections to those sources. You can also use the DarkShield API with glue code to search and mask PII in at least seven additional NoSQL databases. Sample projects are available here.
For relational database sources, DarkShield supports any instance that you can access through a JDBC connection. 3 See the JDBC sections of these articles for database-specific connection guidance. You can also read about the DarkShield RDB plugin or RDB wizard to learn more about working with relational database sources.
Classifying PII
Data classification is the process of defining and labeling specific types of data, such as email addresses, ID numbers, and last names, into unique abstract categories called Data Classes or Data Class Groups. These classifications use certain location attributes or characteristics of the data itself.
To begin, DarkShield requires you to classify data in IRI Workbench, usually as a one-time setup, so it can find and mask the data you are looking for. The Data Classification process begins when you create a project and choose to configure a Data Class and Rule Library (.dcrlib file).
Once configured, this library stores the Data Classes you define, along with the search methods and masking functions assigned to them, for use in search and masking jobs. The library can also be shared where appropriate.
To learn more about the Data Classification process, please refer to this article.
Applying Masking Functions
For masking, DarkShield applies masking functions through Masking Rules. You create and store these rules in the Data Class and Rule Library, which resides in an IRI Project folder. As a result, you can reuse and modify the rules as needed.
When defining a DarkShield job in any of the DarkShield masking wizards, you can pair Masking Rules with Data Classes. These rules can support masking PII using various techniques such as:
- NSA Suite B and FIPS-compliant encryption and decryption algorithms, including format-preserving encryption
- MD5, SHA-1 and SHA-2 hashing
- Deletion/removal
- Full or partial string editing
- Pseudonymization
Auditing Jobs
After running a search-only, search and mask, or mask-only job, DarkShield produces a number of artifacts for PII discovery, compliance auditing, and operational monitoring.
For example, search and masking logs are produced in machine-readable delimited 4 and JSON formats. These outputs are suitable for reporting or export to analytic tools like Excel , Datadog , Splunk ES , and Phantom playbooks.
In addition, DarkShield also produces interactive, aggregate dashboard charts that can be used to visualize results and take action without requiring a SIEM tool, such as the “heat map” shown below.
See this article for details on the HTML5 charts you can customize from IRI Workbench.
Advanced Topics
For more information about DarkShield, see these additional resources:
- Video: PDF configuration options
- Named Entity Recognition (NER)
- DarkShield Wizards
- DarkShield RPC API calls
In general, you can find additional DarkShield resources here: https://www.iri.com/services/training/courseware#iri-darkshield
Frequently Asked Questions
What types of data sources does IRI DarkShield support?
IRI DarkShield can find and mask sensitive data in structured, semi-structured, and unstructured sources. Supported sources include free-floating text, files and documents, NoSQL databases, and relational databases accessible through a JDBC connection. DarkShield can work with data stored on-premises or in supported cloud environments.
How does DarkShield find and mask PII?
DarkShield uses Search Contexts to define how PII and other sensitive data should be found. In IRI Workbench, Search Contexts are derived from Data Classes containing Data Matchers and Location Matchers. Mask Contexts specify how discovered data should be protected using Masking Rules paired with those Data Classes.
What data masking techniques does DarkShield support?
DarkShield Masking Rules support techniques including NSA Suite B and FIPS-compliant encryption and decryption algorithms, format-preserving encryption, MD5, SHA-1 and SHA-2 hashing, deletion or removal, full or partial string editing, and pseudonymization.
Can DarkShield protect data without sending PII outside the organization?
Yes. The DarkShield API server can be hosted on-premises or on a cloud server managed by the customer. IRI does not maintain or manage those services, so PII does not need to leave the customer’s environment.
If you have questions about IRI DarkShield or other IRI data masking tools, or if you are interested in a demonstration or free trial, please email info@iri.com.
- Consider your use case. IRI FieldShield or RowGen may be a better fit for your relational database test data requirements. IRI CellShield EE may be easier for your Excel users to operate. See the tool comparison matrix.
- If you are running DarkShield from Workbench, open the DarkShield API configuration dialog in preferences and paste the local or remote folder location of the DarkShield API distribution (e.g., C:\IRI\DarkShield\API\plankton-1.5.1). Click the Start Server button, and then Apply (and Close). You should see startup messages for the API server in the Workbench console window confirming the DarkShield API is configured correctly for use and management in Workbench.
- The JDBC driver(s) must be located in the API distribution /lib subfolder, which happens automatically when running a DarkShield job.
- The delimited log file is for file-related search results only. JSON logs cover all sources plus masking.
