3 COMMENTS
  • Menna Moataz
    Reply

    Thank you very much for this article! Really helpful.
    Can you please further explain to me these two points from the cons of ELT.
    1)Loss of detailed run-time monitoring statistics and data lineage – especially metadata impact analyses on changes to disparate file, table, or unstructured sources
    2)Loss of modularity due to set based design for performance – and the loss of functionality/flexibility flowing from it

    1. David Friedland
      Reply

      Thank you for your questions.

      1) Stats and event monitoring in the DB world are typically products of complex log analysis rather than fit-for purpose, multi-source-supporting reports and XML audit trials like these. ELT tools also haven’t been great at implementing data lineage capabilities like these, which are a regulatory requirement in the BFSI world, and a must for other businesses — and disciplines like data quality, data masking, test data, analytic data preparation, etc. — where change tracking and recording matter.

      2) In many ELT DWs, complex transformation and other logic is in coded in SQL and procedures; ELT tools wrap and run that code in the DB. One fact load job read from a DB can have 500 lines of code alone. There seems to be an incompatibility between the data-flow-diagram-based-design of most drag and drop data integration tools (ETL, ESB) and ELT jobs run in the DB framework which runs better with set-based movements of data. This can be solved by moving to a pattern-based DW like Data Vault (where there are business rules like trim and type conversion applicable to data ingestion into the vault) using a pattern-based load and ingestion tool like IRI CoSort, or the IRI Voracity platform which can leverage both in an ELT or ETL context.

  • Gary Bindbeutel
    Reply

    Certainly good food for thought.
    Thank you for blogging on what is a complicated topic.
    gab

Leave a Reply to Menna Moataz Cancel reply

Your email address will not be published.

X

Search the Blog