{"id":2869,"date":"2012-11-05T15:56:51","date_gmt":"2012-11-05T20:56:51","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=2869"},"modified":"2019-11-05T15:58:35","modified_gmt":"2019-11-05T20:58:35","slug":"data-validation-with-cosort-in-iri-workbench","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/data-validation-with-cosort-in-iri-workbench\/","title":{"rendered":"Data Validation in IRI Software"},"content":{"rendered":"

Data validation is a process that ensures a program operates with clean, correct and useful data. It uses routines known as validation rules that systematically check for correctness and meaningfulness of data that are entered into the system. The rules are usually automated by either commands, data validation logic, or a library of applied definitions like a data dictionary. Data validation<\/a> is important because users can experience security vulnerability or data corruption without it.<\/p>\n

IRI products offer several methods to check for and discard, or otherwise flag, data that do not meet validation criteria. Meeting or failing data validation tests can affect how individual fields are displayed, or whether entire records are included or omitted in the target. The following are examples of the application of data validation functionality within those IRI’s products that can format data for hand-offs and reports, i.e.,\u00a0CoSort<\/a>\u00a0for data transformation and reporting, NextForm<\/a>\u00a0for data (and DB) migration,\u00a0FieldShield\u00a0<\/a>for data masking, and Voracity<\/a>\u00a0for data discovery, integration, migration, governance and analytics.<\/p>\n

IF THEN ELSE Validation<\/strong><\/h4>\n

1. Use the expression builder in the IRI Workbench to populate an IF THEN expression that creates the following CoSort Sort Control Language (SortCL) job script entry that flags and omits individual field values:<\/p>\n

\/FIELD=(VALID_AGE, ASCII, POSITION=1, SEPARATOR=”,”, IF AGE < 18 THEN “Underage” ELSE AGE)<\/em>
\nThe AGE source field is valid when the value is 18 or higher. Ages will be flagged as “Underage” for minors, otherwise the original source data values will be displayed.<\/span><\/p>\n

2. In the expression builder, you can also generate scripts that looks for exact string matches, or determine whether a field simply contains a certain string:<\/p>\n

\/FIELD=(ID_NO_FLAG, ASCII, POSITION=2, SEPARATOR=”,”, IF ID_NO CT “999” THEN “Invalid” Else “Valid”)
\n<\/em>The ID_NO field is valid only when it does not contain three 9s in a row anywhere throughout the field. The above script entry flags fields as “valid” or “invalid” based on this string matching criterion.<\/p>\n

3. You can incorporate one of several native functions within IF THEN ELSE logic, too. For example,<\/p>\n

\/FIELD=(VALID_CREDIT_CARD_LENGTH, ASCII, POSITION=3, SEPARATOR=”,”, IF length(CREDIT_CARD) NE 16 THEN “Error” ELSE “”)<\/em>
\n<\/span>The above uses the length() function for data validation to ensure whether the credit card value has a size of 16. If the field length does not equal (NE) 16, then the field is flagged with “Error”. \u00c2\u00a0Otherwise the credit card value field is considered valid and, in this example, is not displayed (ELSE “”).<\/p>\n

In addition to the length() function, there are several other native functions that can be used to evaluate data fields for data validation purposes, all of which are accessible via the expressions builder shown below.<\/p>\n

\"\"<\/a><\/p>\n

The following ‘iscompare’ functions are available for CoSort and FieldShield to build useful expressions for data validation:<\/p>\n

C-Style Validation<\/strong><\/h4>\n