{"id":8374,"date":"2015-10-29T09:50:16","date_gmt":"2015-10-29T13:50:16","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=8374"},"modified":"2025-01-02T13:28:14","modified_gmt":"2025-01-02T18:28:14","slug":"flat-file-profiling","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/iri\/iri-workbench\/flat-file-profiling\/","title":{"rendered":"Flat-File Profiling"},"content":{"rendered":"

Editor’s Note<\/strong>: In addition to the flat-file profiling wizard in the data discovery tools<\/a>\u00a0menu group in IRI Workbench described below, IRI has introduced\u00a0robust data classification tools<\/a> that enable the application of data masking and transformation rules<\/a> through data class libraries. \u00a0<\/strong>IRI also provides a\u00a0 flat-file (directory) data class masking<\/a> wizard <\/em>for IRI FieldShield, described here<\/a>, and a file masking wizard for DarkShield that handles structured, semi-structured, and unstructured file discovery and masking in DarkShield, described here<\/a>.<\/em><\/p>\n

Data profiling is the essential data discovery process that helps you analyze, classify, cleanse, integrate, mask, and report on data in your repositories. With the information profile processes produce, you can make decisions about data integration and analysis, migration and masking, etc.<\/p>\n

As with relational and modern data sources, awareness of the content and character of structured files helps data architects assess and improve the usability, quality, and security of these less sexy, but still mission-critical collections.<\/p>\n

IRI previously released dark (document) data<\/a> and relational database profiling<\/a>\u00a0modules in the IRI Workbench<\/a>\u00a0GUI for all IRI software, built on Eclipse\u2122. \u00a0Similar profiling of\u00a0data in flat files is now also in this GUI\u00a0—\u00a0which is free to all IRI software product users\u00a0— and included in\u00a0the free Lite<\/em><\/a> edition of the IRI NextForm data migration product. In other words, all IRI data profiling tools are free<\/a>.<\/p>\n

Using a graphical wizard similar to the database profiler, it is now possible to scan files and forensically assess their characteristics. Key statistics — such as data counts, aggregates, field lengths, minimum and maximum values — plus fuzzy string matches and patterns, are now readily available and covered in the how-to introduction that follows.<\/p>\n

 <\/p>\n

Inputs<\/strong><\/p>\n

The first part of the wizard is the input selection page, which is comprised of a few different elements:<\/p>\n

    \n
  1. The display where the selected file to profile will be shown.<\/li>\n
  2. The Add Data Source and Edit Source Option buttons. These buttons allow you to select the file to profile.<\/li>\n
  3. The Add Existing, Discover, and Convert Metadata buttons. These buttons allow you to select the metadata for the input file.<\/li>\n<\/ol>\n

    \"datasources\"<\/a><\/p>\n

    To choose the input file, click\u00a0Add Data Source<\/strong>, and select the file to be profiled. Then click the appropriate button to specify metadata for the input file. You can select an existing data definition file (DDF) or auto-create one via metadata discovery or conversion. Once your input file and its metadata are displayed, click Next<\/strong> \u00a0to move on.<\/p>\n

     <\/p>\n

    Options<\/strong><\/p>\n

    The second part of the wizard is the field selection page, which displays a table of profiling statistical output to produce, followed by the fields in the file on which you choose to generate the values.<\/p>\n

    \"fieldselection\"<\/a><\/p>\n

    The options matrix will automatically change to represent the fields of the selected file. There are several ways to handle the viewing options:<\/p>\n