Business Intelligence (BI)
This article is a continuation of the previous article, which introduced IRI support for XLS and XLSX file formats in the SortCL program used for moving and manipulating spreadsheet data in both the IRI Voracity data management (ETL, data quality, etc.)
Test Data
Automating where you can, when you can, is the basis of DevOps automation. Allowing IRI job script executions which mask or synthesize test data to be handled in the CI/CD pipeline supports this pursuit.
Data Masking/Protection
One of the data masking requirements for IRI FieldShield that we see in PHI anonymization use cases involves the blurring of dates at the row level, instead of the column level while providing a configurable option to retain the interval between those dates.
Data Masking/Protection
Usually static data masking is performed on production data at rest so it is stored safely, or when replicated to non-production environments for testing or development purposes.
Big Data
An operational data store (or “ODS”) is another paradigm for integrating enterprise data that is relatively simpler than a data warehouse (DW).
Data Masking/Protection
There are times when it is necessary to test with or share data that has elements of personally identifiable information (PII). To comply with data privacy laws and prevent a data breach, you may need to provide data that reflects, and sometimes imparts, critical information, but still protects the PII.
Business Intelligence (BI)
IRI Workbench not only has several ways to create jobs, but also several ways to execute them.
This article focuses on IRI Workbench execution options for scripts based on the SortCL program language, which covers IRI Voracity ETL, CDC, SDC, pivoting and subsetting jobs, as well as its constituent product jobs; i.e.,
ETL
Just as production data processing tools like IRI CoSort must handle big data in NoSQL DB environments, so too must a big test data generation tool like IRI RowGen.
Test Data
Unlike the New DB Test Data Job wizard for creating multiple, related test tables, the New Test Data Job wizard in the IRI Workbench GUI for RowGen generates individual test sets.
Big Data
Note: This example demonstrates a more direct method of using IRI FieldShield or IRI Voracity to statically mask PII within structured MongoDB collections. Our older how-to-article on indirect data masking of MongoDB through export/mask/import from 2015 is here, and a newer method through MongoDB’s native driver support in CoSort v10 from 2018 is here.