IRI Blog Articles

Diving Deeper into Data Management



Which Data Masking Function Should I Use?

gold costume mask

by David Friedland

According to Simson L. Garfinkel at the NIST Information Access Division’s Information Technology Laboratory,

De-identification is not a single technique, but a collection of approaches, algorithms, and tools that can be applied to different kinds of data with differing levels of effectiveness. In general, privacy protection improves as more aggressive de-identification techniques are employed, but less utility remains in the resulting dataset.

-De-Identification of Personal Information, NISTIR 8053

Static data masking (SDM) is the industry-recognized term for these various means of de-identifying data elements at rest. The elements are typically database column or flat-file field values that are considered sensitive; in the healthcare industry, they are referred to as key identifiers. Specifically at risk are personally identifiable information (PII), protected health information (PHI), primary account numbers (PAN), trade secrets, or other sensitive values.

The “startpoint” data-centric security product IRI FieldShield — or the IRI CoSort product and IRI Voracity platform that include the same capabilities — provide multiple data discovery and SDM functions for multiple data sources. Available per-field/column masking functions include:

  1. multiple, NSA Suite B and FIPS-compliant encryption (and decryption) algorithms, including format-preserving encryption
  2. SHA-1 and SHA-2 hashing
  3. ASCII de-ID
  4. binary encoding
  5. randomization
  6. redaction (character obfuscation)
  7. reversible and non-reversible pseudonymization
  8. custom expression (calculation / shuffle) logic
  9. conditional / partial filtering (omission)
  10. custom value replacement
  11. byte shifting and string functions
  12. tokenization (for PCI)

You can also “roll your own” external data masking function. This allows you to call a custom-written, field-level routine at runtime instead of a built-in.

The question remains, which masking function should I use (on each item)? That depends on your business needs and rules, as well as applicable data privacy law(s). At a technical level, that usually means deciding how the resulting ciphertext (masked data) needs to appear, if it needs to be reversible or unique, how secure it is, and possibly, what kind of compute resources and time are available for the process. Let’s look at these common decision criteria in detail:

Appearance (Realism)

a hand drawing an eye

Should the newly masked data look more or less like the original data? What about its size and format? Pseudonymization and format-preserving encryption are the two most common ways to 

retain the look and feel of proper nouns and alpha-digit account or phone numbers, respectively. But substring masking (a/k/a partial field redaction, e.g., XXX-XX-1234) may be just fine for things like SSNs. Think about the persistence and display of the data for analytics, etc.

Related to this, appearance and realism of the ciphertext can also determine the usability of the results. Application and database table (load utility) targets may require that the format of the data not only comply with pre-defined structures, but continue working in queries or other operational contexts downstream.

In other words, if masked data that’s pretty and/or functional data is required, don’t go for full redaction, randomization, hashing, or straight encryption (which widens and obfuscates the results). You may be able to get away with smaller tweaks like ageing and sub-string manipulation, but consider the impact of these choices on your other decision criteria …

Reversibility (Re-Identification)


u-turn signNeed the original data restored? The answer to that may depend on whether you’re leaving the source data alone, as you would in dynamic data masking, or when you’re writing the masked data out to new targets. In those cases, the answer is no.

If the answer is no, you may still need realism, in which cases non-reversible pseudonymization may be your best bet. If it’s no and appearance doesn’t matter, go with character redaction. And if neither is true, consider outright deletion of the source column from the target.

When the answer is yes, IRI data masking functions like encryption, reversible pseudonymization or tokenization, or ASCII re-ID are indicated. In more advanced uses, you may also need differential reversal; i.e., when different recipients of the same target are authorized to see different things in the same data set. In such cases, private encryption keys, user-specific revelation job scripts or even executables can be employed.


Uniqueness (Consistency)

field of flowers

Does the same original value always need to be replaced by the same, but different, replacement value? Is the data going to be joined on, or grouped by, the replacement values? If so, then the replacement algorithm chosen must produce results which are unique and repeatable in order to preserve referential integrity despite the masking that’s occurred.

This can be achieved through encryption when the same algorithm and passphrase (key) are used against the same plaintext. The data classification and cross-table protection wizards in the IRI Workbench GUI for FieldShield (or Voracity) facilitate this through cross-table (or more global) application of the matched masking rule. In this way, the same plaintext value always gets the same ciphertext result regardless of its location.

another field of flowersPseudonymization is trickier here, however, because of a shortage of unique replacement names, duplicate original names, and changes (inserts, updates, or deletes) to the original values in source tables or files. IRI addressed the issue of consistent cross-table pseudonymization in this Voracity workflow example.


Strength (Security)

A look at the algorithms inside each function can help you determine their relative “crackability”, and assess that against the other ciphertext considerations like appearance and speed. For example, IRI’s AES256 function is stronger than its AES128 option, SHA2 is stronger than SHA1, and all best base64 encode/decode and ASCII de-ID/re-ID functions.

By definition, reversible functions are typically weaker than those that cannot be reversed. For example, IRI’s irreversible (foreign lookup set) pseudonymization method is more secure than its reversible (scrambled original set) pseudonymization method. That said, IRI’s AES-256 encryption algorithm can be very hard to crack when the key has been lost.

data lockEven stronger security is of course omission, followed by character obfuscation (redaction), which are irreversible. But the downside is lack of usability. In the HIPAA safe harbour context, the removal of key identifiers complies. If you need to use any part of the source data for analysis, research, marketing, or demonstration, however, you will need a masking function instead, and an expert to determine (and certify) that your technique(s) carry a low statistical likelihood of re-identification.

While we’re on the subject of HIPAA de-identification, remember there may also be risk associated with so-called quasi identifiers (like zip code and age). Those values can be used in conjunction with other data sets to establish a re-identification trail, and are thus also worth masking in many cases; the whether and how are subject to these same considerations.

Computation (Performance)

One of the nice things about the data masking approach — even when compute-intensive encryption algorithms are involved — is that its overhead relative to broad-brush encryption (of an entire network, database, file/system, disk drive) is far lower. Only those data elements (column values) that you designate for protection need to be ingested into, processed by, and returned from, the masking function.

cipher encryption chartIn general, the more complex (and stronger) the algorithm, the longer it will take to apply. Data masking speeds will also depend on the number of functions applied, the number of DB columns and rows, the number of lookup constraints to respect in the process (for referential integrity), network bandwidth, RAM, I/O, concurrent processes, and so on.

The following non-scientific chart breaks down most of the attributes described above for convenient reference, and in generally relative terms only. Needless to say, IRI disclaims any warranty of fitness or liability for this chart!

IRI Data Masking Functions (in FieldShield & Voracity)

data encryption table

data hashing table

data redaction table

data masking encoding and decoding tabledata masking pseudonymization table

data masking randomization table

data masking manipulation table

Whether you use these built-in or custom functions you define, the idea is to apply functions (possibly based on business rules / conditions) to specific rows or columns and/or across tables through protection rules. You can can define, store, and re-use. It is also possible to apply these functions against auto-classified data as rules, or dynamically via an API call.

FieldShield (or Voracity) users can create, run, and manage data masking jobs in the free IRI Workbench GUI, built on Eclipse.™ Or, they can edit and run compatible, self-documenting 4GL scripts defining their source/target data and protections in the command line environment.

For more information, see, or contact an IRI representative.


Print Friendly

{ 0 comments… add one now }

Leave a Comment

Previous post: