{"id":6011,"date":"2024-07-10T17:10:56","date_gmt":"2024-07-10T21:10:56","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=6011"},"modified":"2024-09-19T16:41:16","modified_gmt":"2024-09-19T20:41:16","slug":"textual-etl","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/textual-etl\/","title":{"rendered":"Textual ETL: Unlocking Unstructured Data"},"content":{"rendered":"

Synopsis<\/b><\/h4>\n

Corporations and government agencies store a lot of useful information in non-transactional semi-structured and unstructured data sources. Finding that data \u2013 in documents, logs, and images \u2013 is important not only for data masking, but also for textual ETL.\u00a0\u00a0<\/span><\/p>\n

Textual ETL lends structure and meaning to data that\u2019s hidden in text and prepares it for us in standard structural repositories like flat files, relational databases and Excel. When combined or joined to matching values in other structured sources, more information can be discovered and mined for the benefit of operational monitoring, marketing, trend analytics, law enforcement, etc.\u00a0<\/span><\/p>\n

Beyond enhancing data integration, these values can also be used in some test data scenarios as well. This article talks about producing structure from values discovered during file LAN or cloud <\/span>1<\/sup><\/a><\/span> <\/span>file searches using the <\/span>IRI DarkShield<\/span><\/a> data masking tool. And if you license the broader Voracity data management <\/span>platform<\/span><\/a> which includes DarkShield, you have the ETL and reporting facilities built-in.<\/span><\/p>\n

How this Works<\/b><\/h4>\n

As you know, most data in unstructured sources is difficult to parse, and the values need context and structure to be leveraged in data integration and reporting contexts. Fortunately, the <\/span>New File Search\/Mask Job<\/span><\/i> wizard<\/span><\/a> in the <\/span>IRI Workbench<\/span><\/a> GUI will extract \u2018dark data\u2019 values (and log their metadata) into a flat file, which is what will ultimate enable textual ETL, data fabrics, and other forms or analytics in the <\/span>IRI Voracity<\/span><\/a> ecosystem, powered by <\/span>SortCL<\/span><\/a>.<\/span><\/p>\n

To add value to the delimited log file, DarkShield also generates its field layouts automatically into a separate metadata file in IRI-standard <\/span>data definition format (.DDF)<\/span><\/a>. The results file and its metadata repository are easily used and re-used by IRI software to integrate, transform, migrate, mask, and report on that data, and feed it in other formats or to external applications as needed.<\/span><\/p>\n

Note again that in Voracity, the CoSort <\/span>SortCL<\/span> ETL engine can query and join over flat files directly, or facilitate the creation and population of tables with DBA-defined primary-foreign keys. In this way, dark data extracts can acquire form and relationships (structure) that can make it a lot more useful.<\/span>
\n<\/span><\/p>\n

Creating the Search Job<\/b><\/h4>\n

A DarkShield Files Search Job will search every supported file type in every directory below the root network drive you specify. The scan through your dark data employs search methods you can align to your sensitive type types (e.g., names, email, IP and street addresses, credit card and NID numbers, medical conditions, etc.), or <\/span>Data Classes<\/span><\/i><\/a>.\u00a0<\/span><\/p>\n

You can specify search matchers based on the <\/span>content<\/span><\/a> of the data, fixed metadata (<\/span>location<\/span><\/a>), or both depending on the situation.When looking across all file types, you might want to specify a combination of regular expression patterns, lookup set files, and Named Entity Recognition (NER) models in addition to (metadata) location matchers for data in structured files.<\/span><\/p>\n

Here is a list of file sources containing strings that the wizard can search, extract, and structure:<\/span><\/p>\n