CoSort PlugIns and Flat File Transforms Speed ETL Tool Operations

Home » Solutions » ETL DB Acceleration » ETL Transforms

Speed ETL Operations and Tools

» Solutions

ETL/DB Acceleration

Interested in data-centric security in the data warehouse? Click here to listen to The Data Warehouse Institute (TDWI) interview with IRI VP David Friedland to understand how CoSort protects data at risk at the field level during large scale data integration.

Why Data Warehouse Architects Choose CoSort

Challenges:

Despite their parallel processing and "push down optimization" options, ETL tools and the database engines they may rely on cannot perform the largest transformations optimally, or without having an adverse impact on concurrent operations. As a result, large sort, join, and aggregation jobs may run slowly, and subsequent tasks like loading, analytics, or BI displays may run slowly or require separate passes or products.

Solutions:

If you already use an ETL tool, IRI's CoSort can help you run large jobs faster. If you do not, and you use flat files primarily, consider what CoSort's SortCL tool can do by itself, and in conjunction with IRI Fast Extract (FACT) and your database loader if you have high volumes of data in Oracle and/or DB2.

You can directly speed sorting (and downstream joins, aggregates, and loads) within Informatica or DataStage with CoSort's unique plug-in sort transformations. CoSort also links with Kalido, ETI, Software AG Natural, SAS, and IDS TeraStream.

You can also run CoSort's SortCL tool alongside your ETL tool, or consider SortCL its own ETL tool for flat files. Using the metadata you have, you can exploit SortCL's ability to perform, combine, and accelerate parallel data transformations and operations like:

• Sorts
• Joins
• Aggregates
• Lookups
• Perl-Compatible Regular Expressions
• Data-type and file-format conversions
• Field/column encryption and masking
• Detail, delta and summary reports
• Test data generation

You can call SortCL from the shell (as a batch executable or ETL tool command), via API or Java GUI -- and flow data back and forth through files, pipes, or procedures as needed.

If you use Meta Integration Model Bridge (MIMB) software or services from Meta Integration, you can automatically convert file and field-level metadata defined for popular ETL tools (like Informatica's .xml and DataStage .dsx repositories) into the equivalent SortCL data definition file (.ddf) format. This can save valuable time in building CoSort transform jobs.

IRI, The CoSort Company