CoSort PlugIns and Flat File Transforms Speed ETL Tool Operations

Home » Solutions » ETL DB Acceleration » ETL Transforms

Speed ETL Operations and Tools

» Solutions

ETL/DB Acceleration

Interested in data-centric security in the data warehouse? Click here to listen to The Data Warehouse Institute (TDWI) interview with IRI VP David Friedland to understand how CoSort protects data at risk at the field level during large scale data integration.

Why Data Warehouse Architects Choose CoSort

Challenges:

Despite their parallel processing and "push down optimization" options, ETL and ELT tools and the database engines they may rely on cannot perform the largest transformations optimally, or without having an adverse impact on concurrent operations or budgets. As a result, large sort, join, and aggregation jobs run too slowly, and subsequent tasks like loading, analytics, or BI displays either take too long or require separate passes, products, even proprietary appliances to finish.

Solutions:

If you already use an ETL tool or ELT technology stack, IRI's CoSort can transform big data faster, typically alongside incumbet platforms in sequential (flat) files. If you do not use those tools, consider what CoSort's SortCL program can do instead - especially in line with IRI's Fast Extract (FACT) utility and your database loader (with direct, pre-sorted files).

You can directly speed sorting (and downstream joins, aggregates, and loads) within Informatica or DataStage with CoSort's unique plug-in sort transformations. CoSort also links with Kalido, ETI, Software AG Natural, SAS, and TeraStream.

Better yet, you can also run CoSort's SortCL tool alongside your ETL tool or ELT platform. Using the metadata you have, you can exploit SortCL's ability to run, speed, and combine data transformations and operations like:

• Sorts
• Joins
• Aggregates
• Lookups
• Perl-Compatible Regular Expressions
• Data-type and file-format conversions
• Field/column encryption and masking
• Detail, delta (CDC) and summary reports
• Row-column pivoting
• Slowly changing dimension reports
• Test data generation

You can call SortCL from the shell (as a batch executable or ETL tool command), via API or Eclipse GUI -- and flow data back and forth through files, pipes, or procedures as needed. In the IRI Workbench GUI environment, you construct ELT or ELT flows by connecting CoSort (and FACT) to your database sources and targets along with flat files at the same time.

If you use Meta Integration Model Bridge (MIMB) software or services from Meta Integration, you can automatically convert file and field-level metadata defined for popular ETL tools (like Informatica's .xml and DataStage .dsx repositories) into the equivalent SortCL data definition file (.ddf) format. This can save valuable time in building CoSort transform jobs.

IRI, The CoSort Company