{"id":14208,"date":"2021-01-11T17:58:51","date_gmt":"2021-01-11T22:58:51","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=14208"},"modified":"2026-02-23T17:59:10","modified_gmt":"2026-02-23T22:59:10","slug":"darkshield-rpc-api","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/data-protection\/darkshield-rpc-api\/","title":{"rendered":"IRI DarkShield RPC API"},"content":{"rendered":"

Editors Note<\/strong>: The article below was written prior to the release of V5 of the IRI DarkShield PII data masking tool for structured, semi-structured, and unstructured data. Version 5<\/a>, released in Q1, 2024, introduced separate RPC APIs based on the one below — for files, relational databases and NoSQL databases. All four APIs are now also used to run search and mask jobs configured in the DarkShield GUI<\/a>. DarkShield Version 6, released in Q3, 2025 now also offers a stateless, REST API<\/a> option.<\/em><\/p>\n

IRI DarkShield<\/i><\/a> Version 4 features a Remote Procedure Call (RPC) Application Programming Interface (API) for searching and masking unstructured text. The API allows for the use of DarkShield\u2019s search matchers and masking rules outside of the context of files.<\/p>\n

This \u201cBase API\u201d is also the underlying technology used by the DarkShield Files API<\/i><\/a> for performing search and masking operations specifically in free-form text, CSV\/TSV, JSON\/XML, PDF, Parquet, and\/or image files residing on premise, or in cloud storage like Azure, GCP, S3 or OneDrive\/SharePoint.<\/p>\n

The DarkShield API is built as a plugin on top of the IRI Web Services Platform (codenamed Plankton<\/i>), allowing the user to pick which services they will require while utilizing the same hosting, configuration, and logging capabilities provided through the platform.<\/p>\n

\"\"<\/a><\/p>\n

This article covers the API\u2019s core search and masking capabilities, and provides a few examples.<\/span>1<\/sup><\/a><\/span><\/p>\n

\"\"<\/a><\/p>\n

The DarkShield RPC API is documented through an OpenAPI document<\/i>, a JSON\/YAML based declarative specification of the different endpoints supported by the API, the schemas of the request payloads and the expected responses, as well as examples that can be run against each endpoint.<\/p>\n

The screenshots used throughout this article are HTML-rendered views of DarkShield\u2019s OpenAPI<\/i><\/a> document. By default, all Plankton services use Swagger-UI<\/i><\/a> to render the OpenAPI documents under the \/docs<\/i> endpoint, which provides a useful interface for executing examples.<\/p>\n

In some cases we will also be showing Redoc<\/i><\/a>-rendered versions for added clarity (certain OpenAPI V3 concepts like discriminators <\/i>are not properly modeled in Swagger-UI<\/i>).<\/p>\n

\"This<\/p>\n

 <\/p>\n

 <\/p>\n

For file-based search and masking, we also provide an additional plugin called the IRI DarkShield Files API<\/i>. The files plugin utilizes the same search and matcher contexts that are defined in the base API discussed in this article, so we recommend that you read this article first before continuing on to the files article.<\/p>\n

\"\"<\/a><\/p>\n

Running the API<\/b><\/h5>\n

To follow along the examples given in this article, we recommend that you look at our Plankton setup guide (packaged with the trial) for how to install and run the API. Once you launch Plankton, navigate to the location where the OpenAPI docs are hosted in your browser, which by default is on http:\/\/localhost:8959\/docs<\/i>.<\/p>\n

\"Startup<\/p>\n

Creating Search Contexts<\/b><\/h5>\n

Before we can begin using the API to annotate text, we must first define the search methodology that DarkShield will use. For that, we need to host a Search Context which defines a set of Search Matchers that we will use to match on the data, as well as a unique name so that we can send search requests to it.<\/p>\n

If you select the \/api\/darkshield\/searchContext.create <\/i>endpoint, you will be shown a description of the endpoint along with a useful example for how to create one. You can also select other examples from the dropdown:<\/p>\n

\"\"<\/a><\/p>\n

To execute an example, click on the Try it out <\/i>button, which adds an Execute <\/i>button on the bottom and allows you to modify the currently selected example in case you want to tweak it or create your own:<\/p>\n

\"\"<\/a><\/p>\n

Click Execute <\/i>to create the example Search Context which you have selected. On the bottom, you will see the curl<\/i> command that can be used to execute the same command through the command line, as well as the response (which should return a 200 status code with an empty body for a successful execution):<\/p>\n

\"\"<\/a><\/p>\n

You will note that the name of the Search Context has to be unique, and duplicate Search Contexts are prohibited. If you try to execute the same exact example again, you will get a 400 Bad Request response with an error message stating that the Search Context you were trying to create already exists:<\/p>\n

\"\"<\/a><\/p>\n

If you wish to change a Search Context, you must first explicitly destroy it by calling the\u00a0\/api\/darkshield\/searchContext.destroy <\/i>endpoint. All Search and Mask Contexts will also be destroyed automatically when the server stops, and have to be re-created when Plankton<\/i> is restarted.<\/p>\n

Search Matchers<\/b><\/h5>\n

The OpenAPI<\/i> document not only provides examples for the searchContext.create <\/i>endpoint, but also the schema for how a request payload should look like in order for a Search Context to be created. If you are still in example mode, click Cancel <\/i>to return to the original view, and the click on Schema <\/i>next to the Examples <\/i>tab under the Request Body <\/i>to view the schema for the endpoint:<\/p>\n

\"\"<\/a><\/p>\n

From this view, you can see that a Search Context requires a name and a list of Search Matchers (with at least one Search Matcher defined). The name of the Search Matcher will be added to an annotation to indicate which matcher was used to annotate it.<\/p>\n

Note that, unlike the Search Context name, the Search Matcher name does not have to be unique. For example, there are multiple ways to match on names, and each matcher can be named \u201cNamesMatcher.\u201d<\/p>\n

Looking at the screenshot above, the schema for a Search Matcher does not give a lot of clues regarding what the required fields are, or what the type <\/i>should be. This is a known limitation for Swagger UI<\/i> since it cannot display all subclasses of the Search Matcher schema based on the discriminator value (in this case, the type <\/i>field).<\/p>\n

You can scroll down to the bottom of the page to view the Schemas<\/i> tab that contains the various different subtypes of the Search Matcher. Or if you are using Redoc<\/i>, you will be able to see more information:<\/p>\n

\"\"<\/a><\/p>\n

On Redoc<\/i>, you can toggle between the 5 different Search Matcher types, each of which have their own parameters and return their own annotations:<\/p>\n

    \n
  1. Fuzzy Lookup (fuzzy): Searches for exact and non-exact matches within text using Levenshtein distance. Fuzzy Matchers can specify various matching criteria, like maximum Levenshtein Distance or the number of permitted deletions\/insertions. Returns annotations that contain the Levenshtein distance between the match and the keyword that the matcher thought it matched.<\/li>\n
  2. Named Entity Recognition (ner): Searches for a specific category of data that the OpenNLP-compatible model was trained on based on the sentence context in which that data could appear in (e.g., , names, addresses, dates, etc.). Can specify optional sentence detectors and tokenizer models, otherwise standard whitespace detectors will be used to split the text into sentences and tokenize them. Return annotations containing a statistical probability that the named entity was correctly identified.<\/li>\n
  3. Pattern (pattern): A regular expression pattern with an optional validation JavaScript file for performing computational validation on matches (for example, calculating the checksum for credit cards). A pattern match can also be split into named groups, which is useful in cases where only a part of the pattern is of interest. Returns standard annotation which can optionally include the group that was matched from the pattern if named groups were specified (both in the pattern and the groups<\/i> matcher parameter).<\/li>\n
  4. Set File Lookup (set): direct lookup of entries from a set file. Can choose whether to match whole words and whether to ignore case. Return standard annotations.<\/li>\n
  5. Transformers Named Entity Recognition (transformers): Searches for text that an NER model (of either the PyTorch or TensorFlow frameworks) has annotated. Either a name of a model on Huggingface Model Hub<\/a> or the folder location of the model can be specified for both the NER model and tokenizer model. What labels to include in annotations can be specified by the entityLabels<\/em> parameter. For example, an NER model may label different types of named entities (location names, peoples’ names,\u00a0 organization names, etc.) differently. By default, all entities labeled by the model are included in annotations.\u00a0 How entities are grouped can be tuned via the aggregationStrategy<\/em> parameter. Returns annotations containing a statistical probability that the named entity was correctly identified.<\/li>\n<\/ol>\n
    Searching Text<\/b><\/h5>\n

    Once we have hosted our Search Context, we want to perform an actual search against it. Scroll down to the \/api\/darkshield\/searchContext.search<\/i> endpoint and select the example that matches the Search Context that you have created previously:<\/p>\n

    \"\"<\/a><\/p>\n

    The request contains the name of the Search Context that we want to use and the text that we want to search. If we click on the Try it out <\/i>button and then click on the Execute<\/i>, we can see the resulting annotations file that is returned from the API:<\/p>\n

    \"\"<\/a><\/p>\n

    A list of annotations are returned, with each annotation containing the match, the name of the matcher that matched it, the start and end character offsets of the match, and the type of the annotation (which can help determine what additional fields could be defined for the match).<\/p>\n

    These annotations are in JSON and downloadable to a file. It is therefore possible to export your search results to third-party tools; for example, SIEM portals like Datadog<\/a> or Splunk<\/a>, or a data catalog like Alation.<\/p>\n

    If you are using multiple Search Matchers in a Search Context, there is a chance that overlapping annotations are produced (for example, if a set lookup and an NER model matches on the same name). In those cases, DarkShield only keeps one of the overlapping matches, preferring the left-most and longest match.<\/p>\n

    In the future, there may be additional strategies which can be specified inside of a Search Context for dealing with overlapping annotations.<\/p>\n

    If you pick an example for a Search Context which has not yet been hosted, you will see the following error, which states that the Context in question does not exist:<\/p>\n

    \"\"<\/a><\/p>\n

    In a case like this, you can either select the correct example or create the Search Context which you are trying to use from the previous section.<\/p>\n

    Creating Mask Contexts<\/b><\/h5>\n

    As a direct analog to a Search Context, we need to create a corresponding Mask Context which can be used to mask annotated text. A Mask Context defines a list of masking rules that should be applied, as well as a list of Rule Matchers which match annotations to their corresponding data masking rule (function<\/a>).<\/p>\n

    Navigate to the \/api\/darkshield\/maskContext.create<\/i> endpoint:<\/p>\n

    \"\"<\/a><\/p>\n

    You will note that the names of the examples correspond to the Search Context examples. However, it would be incorrect to say that there is a one-to-one matching between a Mask Context and a Search Context. That is because a Mask Context can be used to mask annotations produced from any number of Search Contexts, provided that the annotations match the criteria of the Rule Matchers.<\/p>\n

    To illustrate this point, we will first look at the definition of the Rule Matchers in Redoc<\/i>.<\/p>\n

    Rule Matchers<\/b><\/h5>\n

    A Rule Matcher is used to map an annotation to a corresponding Masking Rule. Currently, there is only one type of Rule Matcher, a Name Rule Matcher<\/i>, which matches on the name of the Search Matcher that created the annotation using a regular expression pattern.<\/p>\n

    \"\"<\/a><\/p>\n

    A Name Rule Matcher can therefore match on multiple different Search Matcher names, including all possible Search Matchers if an inclusive pattern is specified (for example, \u201c.+\u201d which matches on all possible sequences of ASCII characters).<\/p>\n

    Rule Matchers are applied in order, so a common strategy for making sure that all annotations are masked is to define the specific Rule Matchers first, and then more general Rule Matchers which can catch all non-matched annotations.<\/p>\n

    If no Rule Matcher matches on an annotation, that annotation will not be masked, and will be placed inside of the failedResults<\/i> array along with the reason why it failed (because it could not be matched).<\/p>\n

    Masking Rules<\/b><\/h5>\n

    Each Rule Matcher corresponds to exactly one Masking rule, defined in the rules <\/i>array:<\/p>\n

    \"\"<\/a><\/p>\n

    Currently, there is only one type of masking rule, a CoSort <\/i>rule, which is defined using an IRI Domain-Specific Language (DSL) which is used in FieldShield. The DarkShield API delegates masking tasks to the underlying SortCL engine by generating the associated script.<\/p>\n

    Some examples of common rule expressions include:<\/p>\n