{"id":5057,"date":"2014-05-30T10:58:29","date_gmt":"2014-05-30T14:58:29","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=5057"},"modified":"2019-04-04T09:08:56","modified_gmt":"2019-04-04T13:08:56","slug":"finding-dark-data-unstructured-sources","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/finding-dark-data-unstructured-sources\/","title":{"rendered":"Finding Dark Data in Unstructured Sources"},"content":{"rendered":"
Update April 2019: Added more unstructured file formats.<\/pre>\n
This is the first of a three-part blog series introducing IRI’s\u00a0new data structuring technology. This article\u00a0defines\u00a0“dark data” and the unstructured sources IRI now supports. The second article shows\u00a0how the Data Restructuring wizard works<\/a>, and the third shows how the restructured data can be used by all IRI software<\/a> products.<\/em><\/p>\n
According to Gartner Analyst Douglas Laney, \u201centerprise dark data\u201d is \u201cun<\/em>utilized or under<\/em>utilized information, collected generally for a single purpose \u2014 then forgotten or archived.\u201d1<\/a><\/sup>\u00a0 Much of the\u00a0dark data<\/a> that corporations have is in unstructured data repositories.<\/p>\n
What is unstructured (vs. structured) data? According to Wikipedia, unstructured data<\/a> is \u201cinformation that either does not have a pre-defined data model or is not organized in a pre-defined manner\u201d. It’s data that are not organized or classified in a way that can be easily grouped by subject; it’s mostly textual, but can also be images, audio, and video.<\/p>\n
And let’s not forget social media. Facebook, Twitter, LinkedIn, Pinterest, just to name a few \u2013 these all contain unstructured and semi-structured data. Valuable data that can be very beneficial to businesses, large and small. However, it really needs to be structured before it becomes useful.<\/p>\n
Structured data is of course the\u00a0opposite of unstructured data. Webopedia defines structured data<\/a> as \u201cdata that resides in a fixed field within a record or file.\u201d It’s organized, and relies on a model that determines how the data is stored, processed, and accessed. Structured Query Language (SQL) is often used for managing structured data in database tables, just as SortCL<\/a> data definition files (DDF) in IRI CoSort<\/a> define the layouts of external, flat files.<\/p>\n
Semi-structured data is a cross between both structured and unstructured data. It has structured data but doesn’t fit into the formal models of relational databases or other sequential sources. Legacy (mainframe index) files are a good example of this hybrid, because they consist of structured elements and proprietary layouts. Many XML files may fall into this category, too, although there are also tons of flat (structured) and unstructured (free-form) XML documents.<\/p>\n
IRI software traditionally handled big data<\/a> only in structured sources<\/a>; i.e. all kinds of flat file formats and relational database tables that are extracted or reached via ODBC. But now<\/em> it can also extract, structure, and process data in several semi- and unstructured data sources, including:<\/p>\n
Unstructured Files<\/strong> (using the Data Structuring wizard in the IRI Workbench<\/a> GUI, built on Eclipse\u2122)<\/p>\n