The masking approach covered in this post is based on a common pseudonymization method, but does not guarantee data can be restored to their original values. This may seem a little confusing, but the reasoning will become clearer later on.<\/p>\n
While the approach described below can be leveraged for a variety of data fields, this post focuses on the masking of last names.<\/p>\n
Challenges to Address<\/b><\/h5>\n
Pseudonymizing data that results in realistic values brings with it the challenges of masking new values introduced into production, and the risk of being able to derive the original value from the masked value (re-identification). Before we discuss a method for addressing these issues, let\u2019s look at a common approach for performing data pseudonymization.<\/p>\n
Basic Pseudonymization<\/b><\/h5>\n
One of the easiest ways to pseudonymize data is to use a substitution value contained in a lookup file or a database table with two columns like the maskedLastMap.txt<\/em> file shown below.\u00a0 Masking names in this case involves finding a matching value in the first column, and using the second column as the output value.<\/p>\n The ability to do this type of search and replace is built into IRI tooling, which will be covered in more detail later in this post.<\/p>\n With basic pseudonymization, what happens if there is a new input value from production that is not in the file above?\u00a0 A maintenance process could be built to update the file, but that will not work in a dynamic environment where masking capabilities are needed on-demand.<\/p>\n What about the risk of determining the original value from a masked value? The file above provides a direct one-to-one mapping back to the production value. You could secure the file, but properly securing this type of file may be easier said than done.<\/p>\n So, how can we reduce re-identification risk? One possibility is to use a file like the LastName.set file below instead of a file that has direct mapping like the one illustrated in Basic Pseudonymization<\/i> above.<\/p>\n The first thing you will notice is that the first column in the file appears to be an arbitrary string of characters, but is in fact a hash value derived from actual names.\u00a0 Since the hash value is not reversible, mapping masked dataset values back to their production values is not possible.<\/p>\n With this file, the input name must be hashed prior to searching the first column for a match.<\/p>\n Introducing hashing into the process may seem to make things much more complicated, but don\u2019t worry, the process is pretty simple with IRI FieldShield<\/a> and is covered in Pseudonymization with FieldShield<\/i> below. And in case you are wondering, the process for setting up the LastName.set file is covered at the end of the post.<\/p>\n Over a period of time, new values will likely be introduced into production and will need to be masked.\u00a0 As we saw earlier, a standard search and replace approach will not work if the input value doesn\u2019t exist in the file.<\/p>\n One method for resolving this issue, using the LastName.set file from above, is to use the first row containing a value less than the hash value of the input name.<\/p>\n For example, the hash value of Wonderkin is not contained in the LastName.set file and using the first row with a lesser value would result in CAMPER being the masked output value.<\/p>\n As illustrated above, using the less than method can result in multiple input names mapping to the same output name and will therefore not allow the data to be restored to its original value.<\/p>\n Generally, multiple input names mapping to a single name for non-production testing should not be an issue.\u00a0 The frequency of this occurring can be minimized by having a file with a large number of names.<\/p>\n IRI FieldShield<\/a> is a software product for masking personally identifiable information (PII) using many different methods<\/a>, including pseudonymization. Instructing FieldShield how to read and protect structured RDBs and flat files is specified via a scripting language called the FieldShield Control Language (FCL), which is based on the antecedent, broader Sort Control Language program, aka SortCL<\/a>; SortCL can thus run .scl or .fcl job scripts.<\/p>\n Among other methods of pseudonymization, FieldShield supports the implementation of find and replace using what is referred to as a Set file and SEARCH function. The LastName.set file discussed earlier is an example of a Set file and will be used to demonstrate pseudonymization of last names with FieldShield.<\/p>\n Using the Set file for search and replace is as simple as defining the name of a Set file and a search operation in a \/FIELD statement.<\/p>\n Masking the last name is a three step process:<\/p>\n Step 1:<\/b> The input name is converted to upper case to ensure any difference in a name\u2019s case will result in the same search result.\u00a0 For example, Smith and SMITH will be converted to SMITH.<\/p>\n Step 2:<\/b> The uppercase name is hashed using a built-in SHA-256 hashing algorithm.<\/p>\n Step 3:<\/b> The Set file is searched for a value less than or equal to (LE) the value of the hashed name.\u00a0 Note: If the name\u2019s hash value is less than the first record of the Set file the masked last name will be assigned a value of \u201cNONAME\u201d.<\/p>\n As was mentioned at the beginning of this post, new values and risk of re-identification are two issues that need to be addressed when pseudonymizing names (or other data) for non-production uses.<\/p>\n The use of LE for the \/FIELD statement search option addresses any new values that may not exist in the Set file.\u00a0 As mentioned earlier, any less than conditions could result in multiple input names being masked to the same name, but generally this shouldn\u2019t impact its usage.<\/p>\n Using hash values in the masking processes greatly reduces the risk of re-identification.\u00a0 For someone to map a masked name back to its original production value, they would need execute authority for the FieldShield software and access to the encryption passphrase used to hash the names.<\/p>\n Below is a complete example of the FieldShield job script that demonstrates the concepts discussed in this post to address new values and reduce the risk of re-identification. The script defines a source file (\/INFILE statement), performs the specialized pseudonymization work in the \/INREC section, and creates the target file with pseudonymized last names file (\/OUTFILE statement).<\/p>\n To minimize the many to one mappings that can occur using the method described earlier, it is important when creating the LastName.set file that it contains a large number of names.<\/p>\n For purposes of this post, a Set file of names provided as part of the IRI Workbench install and a 2010 US Census file of last names were used.\u00a0 Some suggested additional sources of last names would be your company\u2019s master person data repository as well as numerous resources that can be found on the internet.<\/p>\n The LastName.set file discussed earlier was created by a three step process depicted in the flowchart below.\u00a0 A detailed description follows.<\/p>\n The first step in creating the LastName.set file is to combine the IRI example last names Set file and a download of last names from the 2010 US Census.\u00a0 The names are converted to uppercase and sorted to order the records by last name and ensure any duplicates are removed.<\/p>\n Step 1 Script:<\/p>\n Step 1 Input Sample:<\/p>\n Step 1 Output Sample:<\/p>\n In this step the hashed last names are sorted in sequence by hash value and any duplicates are removed.\u00a0 While it may be possible for duplicates to occur, this was not the case in the testing performed using the combined US Census and IRI files with more than 173,000 records.<\/p>\n Step 2 Output Sample<\/p>\n In the final step of creating the LastName.set file, the unique last names file from step 1 and the sorted hash file from step 2 are joined by sequence number. The resulting Set file contains a hash value in the first column, and a last name in the second column and is now ready to be used in a script.<\/p>\n Step 3 Output Sample:<\/p>\n In this post, we discussed the challenges of new values being added to production and reducing the risk of re-identification when masking production data through pseudonymization for non-production usage. We conceptually addressed these issues using hashed name values stored in a .set file and demonstrated the solution using IRI FieldShield<\/a>. Following this article and interest in the subject, IRI developed a fit-for-purpose hash-based pseudonymization rule<\/a>, and hashed set file creation wizard<\/a>, both now available in IRI Workbench for producing consistent, self-updating pseudonyms.<\/p>\n","protected":false},"excerpt":{"rendered":" Abstract: It is common practice to mask sensitive production data for non-production purposes. Creating realistic values presents the challenge of handling new values introduced into production data sources and the risk of re-identification. In this post, we will look at a method available to IRI FieldShield, DarkShield, and CellShield users to address both issues at<\/p>\n![\"\"](\"\/blog\/wp-content\/uploads\/2021\/06\/pseudonymize-and-re-id-1.png\")
<\/a><\/p>\nReducing Re-identification Risk<\/b><\/h5>\n
![\"\"](\"\/blog\/wp-content\/uploads\/2021\/06\/pseudonymize-and-re-id-2.png\")
<\/a><\/p>\nHandling New Values<\/b>
\n<\/b><\/h5>\n![\"\"](\"\/blog\/wp-content\/uploads\/2021\/06\/pseudonymize-and-re-id-3.png\")
<\/a><\/p>\nPseudonymization with FieldShield<\/b>
\n<\/b><\/h5>\nUsing a Set File<\/b><\/h5>\n
![\"\"](\"\/blog\/wp-content\/uploads\/2021\/06\/pseudonymize-and-re-id-4.png\")
<\/a><\/p>\nAddressing the Challenges<\/b><\/h5>\n
Complete Pseudonymization Script<\/b><\/h5>\n
![\"\"](\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/06\/pseudonymize-and-re-id-5.png\")
<\/a><\/p>\nNote again that the input sources and\/or output targets can also be RDB tables as well as files.<\/h6>\n
Creating the LastName.set File<\/b><\/h5>\n
![\"\"](\"\/blog\/wp-content\/uploads\/2021\/06\/pseudonymize-and-re-id-6.png\")
<\/a><\/p>\nStep 1<\/b><\/h5>\n
![\"\"](\"\/blog\/wp-content\/uploads\/2021\/06\/pseudonymize-and-re-id-7.png\")
<\/a><\/p>\n![\"\"](\"http:\/\/www.iri.com\/blog\/wp-content\/uploads\/2021\/06\/pseudonymize-and-re-id-8.png\")
<\/a><\/p>\n![\"\"](\"\/blog\/wp-content\/uploads\/2021\/06\/pseudonymize-and-re-id-9.png\")
<\/a><\/p>\nStep 2<\/b><\/h5>\n
![\"\"](\"\/blog\/wp-content\/uploads\/2021\/06\/pseudonymize-and-re-id-10.png\")
<\/a><\/p>\n![\"\"](\"\/blog\/wp-content\/uploads\/2021\/06\/pseudonymize-and-re-id-11.png\")
<\/a><\/p>\nStep 3<\/b><\/h5>\n
![\"\"](\"\/blog\/wp-content\/uploads\/2021\/06\/pseudonymize-and-re-id-12.png\")
<\/a><\/p>\n![\"\"](\"\/blog\/wp-content\/uploads\/2021\/06\/pseudonymize-and-re-id-13.png\")
<\/a><\/p>\nSummary<\/b><\/h5>\n