Overview DataStage ETI*Solution Informatica OWB & ODI Others

Challenges

Most ETL and ELT tools, along with the database engines they use, cannot transform big data efficiently without:

  • an expensive parallel processing edition
  • taking database or system resources from others
  • a complex, hard-to-maintain Hadoop environment
  • other major hardware investments
  • pushing the problem to an even more expensive database

It is the large sort, join, and aggregation jobs that can take too long. Subsequent tasks like loading, analytics, or BI displays also suffer. They are also being performed in separate steps, products, or expensive appliances.
 

Solutions

If you use other ETL or ELT software, IRI software can speed any of their major processes now running on Unix, Linux, or Windows:

Operation Solution Supports Advantages
Extraction Fast Extract (FACT) Oracle, DB2, Sybase, MySQL, SQL Server, Altibase, Tibero Native DB drivers, parallel unloads, portable flat file output data, simple job scripts, easy to invoke
Transformation CoSort's SortCL Program DB-agnostic, all flat files, Informatica, DataStage, and any 'system command' call Multi-threading, task and I/O consolidation, local and remote execution in file systems, automatic metadata and job script creation
Loading Any CoSort sort engine Any RDBMS load utility, ODBC, JDBC Stream pre-CoSorted data after E or T jobs


IRI's Fast IRI's CoSort can transform big data faster, typically alongside incumbent platforms in sequential (flat) files. If you do not use those tools, consider what CoSort's SortCL program can do instead, especially in line with IRI's Fast Extract (FACT) utility and your database loader (with direct, pre-sorted files).

You can directly speed sorting (and downstream joins, aggregates, and loads) within Informatica or DataStage with CoSort's unique plug-in sort transformations. CoSort also links with Kalido, ETI, Software AG Natural, SAS, and TeraStream.

Better yet, you can also run CoSort's SortCL tool alongside your ETL tool or ELT platform. Using the metadata you have, you can exploit SortCL's ability to run, speed, and combine data transformations and operations like:

  • Sorts
  • Joins
  • Aggregates
  • Lookups
  • Perl-Compatible Regular Expressions
  • Data-type and file-format conversions
  • Field/column encryption and masking
  • Detail, delta (CDC) and summary reports
  • Row-column pivoting
  • Slowly changing dimension reports
  • Test data generation

You can call SortCL from the shell (as a batch executable or ETL tool command), via API or Eclipse GUI -- and flow data back and forth through files, pipes, or procedures as needed. In the IRI Workbench GUI environment, you construct ELT or ELT flows by connecting CoSort (and FACT) to your database sources and targets along with flat files at the same time.

Meta Integration Model Bridge (MIMB) software or services from Meta Integration Technology convert metadata defined for popular ETL tools (like Informatica's .xml and DataStage .dsx repositories) into the equivalent SortCL data definition file (.ddf) format. This automatic metadata replication preserves existing investments, facilitates SortCL job creation, and reduces migration time.
 

Request More Information

* indicates a required field.
IRI does NOT share your information.