{"id":7557,"date":"2015-09-24T10:54:18","date_gmt":"2015-09-24T14:54:18","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=7557"},"modified":"2017-11-06T12:23:16","modified_gmt":"2017-11-06T17:23:16","slug":"data-quality-and-fuzzy-searching","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/vldb-operations\/data-quality-and-fuzzy-searching\/","title":{"rendered":"Data Quality and Fuzzy Searching"},"content":{"rendered":"

IRI is now also delivering fuzzy search functions, both in its free database and flat-file profiling tools, and as available field-function libraries in\u00a0IRI CoSort, FieldShield, and Voracity<\/a> to augment data quality, security, and MDM capabilities. \u00a0This is the first in a series of articles on IRI fuzzy search solutions covering\u00a0their application to data quality improvement.<\/em><\/p>\n

Introduction<\/strong><\/p>\n

The veracity or reliability of data of one of big ‘V’ words (along with volume, variety, velocity, and value) that IRI et al talk\u00a0about in\u00a0the context of data and enterprise information management. Generally, IRI defines data in doubt as having one or more of these attributes:<\/p>\n

    \n
  1. Low quality, because it is inconsistent, inaccurate, or incomplete<\/li>\n
  2. Ambiguous (think MDM), imprecise (unstructured), or deceptive (social media)<\/li>\n
  3. Biased (survey question), noisy (superfluous or contaminated), or abnormal (outliers)<\/li>\n
  4. Invalid for any other reason (is the data correct and accurate for its intended use?)<\/li>\n
  5. Unsafe \u2013 does it contain PII or secrets, and is that properly masked, reversible, etc.?<\/li>\n<\/ol>\n

    This article focuses only on new fuzzy search solutions to the first problem, data quality. Other articles in this blog discuss how IRI software addresses the other four veracity problems;\u00a0ask for help finding them if you can’t.<\/p>\n

    About Fuzzy Searching<\/strong><\/p>\n

    Fuzzy searches find\u00a0words or phrases (values) that are\u00a0similar, but not necessarily identical,\u00a0to other words or phrases (values). This type of search has many uses, such as finding sequence errors, spelling errors, transposed characters, and others we’ll cover later.<\/p>\n

    Performing a fuzzy search for approximate words or phrases\u00a0can help find data that may be a duplicates of previously stored data. However,\u00a0user input or auto correction may have altered the data in some way to make the records seem independent.<\/p>\n

    The rest of the article will\u00a0cover\u00a0four fuzzy search functions which IRI now supports,\u00a0how to use them to scour your\u00a0data, and return those records approximating the search value.
    \n
    \n1. Levenshtein<\/strong><\/p>\n

    The Levenshtein algorithm works by taking two words or phrases, and counting how many edit steps it will\u00a0take to turn one word or phrase\u00a0into the other. The less steps it will take, the more likely the word or phrase is a match. The steps the Levenshtein function can take are:<\/p>\n

      \n
    1. Insertion\u00a0of\u00a0a character into the\u00a0word or phrase<\/li>\n
    2. Deletion\u00a0of\u00a0a character from the\u00a0word or phrase<\/li>\n
    3. Replacement\u00a0of\u00a0one\u00a0character in a word or phrase with\u00a0another<\/li>\n<\/ol>\n

      The following is a CoSort SortCL<\/a> program (job script) demonstrating how to use\u00a0the Levenshtein fuzzy search function:<\/p>\n

      \/INFILE=LevenshteinSample.dat\r\n \/PROCESS=RECORD\r\n \/FIELD=(ID, TYPE=ASCII, POSITION=1, SEPARATOR=\"\\t\") \r\n \/FIELD=(NAME, TYPE=ASCII, POSITION=2, SEPARATOR=\"\\t\")\r\n\/REPORT\r\n\/OUTFILE=LevenshteinOutput.csv\r\n \/PROCESS=CSV \r\n \/FIELD=(ID, TYPE=ASCII, POSITION=1, SEPARATOR=\",\") \r\n \/FIELD=(NAME, TYPE=ASCII, POSITION=2, SEPARATOR=\",\") \r\n \/FIELD=(FS_RESULT=fs_levenshtein(NAME, \"Barney Oakley\"), POSITION=3, SEPARATOR=\",\")\r\n \/INCLUDE WHERE FS_RESULT GT 50<\/pre>\n
      There are two\u00a0parts that must be used to produce\u00a0the desired output.<\/p>\n
      FS_Result=fs_levenshtein(NAME, \"Barney Oakley\")<\/pre>\n
      This line calls the function fs_levenshtein, and stores the result in the field FS_RESULT. The function takes two input parameters:<\/p>\n