{"id":4760,"date":"2014-02-19T17:22:32","date_gmt":"2014-02-19T22:22:32","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=4760"},"modified":"2017-11-06T13:42:30","modified_gmt":"2017-11-06T18:42:30","slug":"working-data-quality","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/iri\/business\/working-data-quality\/","title":{"rendered":"Working Towards Data Quality"},"content":{"rendered":"

\"Data-Quality<\/a><\/strong><\/p>\n

Introduction<\/strong><\/p>\n

In this article, I suggest ways to move your company’s data towards a higher state of quality. \u00a0The highest quality occurs when the data meets the needs of your company. These needs are expressed in rules regarding legitimate values and relationships. This in turn requires that:<\/p>\n

    \n
  1. legitimate values and relationships are defined, and continually expanded and redefined;<\/li>\n
  2. \u00a0you have the means to verify that the rules are being met; and,<\/li>\n
  3. you correct the incorrect data accordingly.<\/li>\n<\/ol>\n

    This is not as hard as it may appear. First, the highest quality is not perfect data, but only operationally sufficient data. Second, most data is wrong independent of context. That means you need to discover impossible or unlikely values. And fortunately, there are powerful tools to discover data that are possible, but wrong in context.<\/p>\n


    \nOverview<\/strong><\/p>\n

    Data would be of the highest quality if it were an exact, complete, protected, and consistent representation of facts. In practical terms, an achievable goal would be to assure that the corporate data is of acceptably high quality which is “fit for their intended uses in operations, decision making, and planning” (J. M. Juran). This definition relieves you from having to do the impossible. Data does not need to be perfect, only be good enough to get the work done. Again, this is achievable.<\/p>\n

    High quality data is developed through data governance — the cooperation of management, auditors, programmers, and data architects who develop and maintain a system of data standards. The operation referred to as master data management (MDM) develops and enforces those standards. It also includes continuously monitoring and correcting the data against the standards.<\/p>\n

    To assess the return on the investment in data quality<\/a> improvement efforts, you must monetize the costs of achieving and maintaining high quality data against the penalty for having not-so-high quality data. Costs come in the form of\u00a0planning, implementation, discovery, and remediation.<\/p>\n

    In short, garbage in derives garbage out. Unfortunately, the benefits of high quality data are usually not appreciated until something goes wrong because the data has lead to an erroneous result.<\/p>\n

    Data Governance<\/strong><\/p>\n

    High quality data needs definition for your enterprise. This takes the form of rules about the possible values and required relationships for the data. Data governance teams or designated information stewards may make these rules initially. But ultimately data architects, DBAs, and other IT users will have to manage within that framework to adjust the rules as data grows. Slowly changing dimensions<\/a>\u00a0and business rule changes should also be recorded with data lineage and asset control repositories, respectively.<\/p>\n

    Sometimes the best that can be done is to control how things might look. As a meaningful double negative, the data in the base may not be correct but it does not look incorrect. For example, this article was written in February. This may or may not be true, but clearly it was not written in the month of Yraubef. \u00a0This is critical knowledge in maintaining data quality because at least when you know that the data is<\/em>\u00a0definitely wrong, you can correct it.<\/p>\n

    A simple plan begins with knowing the rules and centrally recording the possible values or relationships of each datum in an MDM repository (see Master Data Management below).<\/p>\n

    Some databases have a \u00a0facility for recording these constraints and checking these values on data entry. Not all databases work the same way, however, and flat files do not have this feature at all. We propose taking the metadata off-line and enriching it. The offline records should be database independent.<\/p>\n

    For each database table and file, consider at least creating a spreadsheet containing columns for:<\/p>\n