High Volume Data Manipulation and Flat File Transformation for Developers and ETL Architects

Home » Solutions » ETL DB Acceleration » File Transforms

Transform Huge Files Faster

Smart Data Architects Use Flat Files and CoSort

Challenges:

Do you need to manipulate large volumes of data? Integrating, staging, and reporting large data volumes inside your database, ETL, or BI tool can take a very long time, and require costly software and hardware upgrades or appliances. Meanwhile, the overhead imposed during internal transformations impacts other users, ETL component runtimes, database query response times, and digital displays.

You may also need to directly manipulate or convert mainframe file formats, like Micro Focus ISAM or variable block, to and from desktop formats like record sequential, CSV, LDIF, and XML. Or, you may need to reformat and compare flat files for reporting purposes and/or what-if analyses.

Solutions:

Flat files are not only a convenient, common format for data exchange, but often the fastest way to change operational data and produce reports. If you use flat files, you should know that CoSort has long been considered one of the fastest data transformation tools for flat files available. Since 1992, CoSort has used a popular 4GL called SortCL to perform multiple data transformations in the same job script and I/O pass to manipulate large files in batch streams.

SortCL runs in the operating system (on the command line, from a shell script or batch stream), from a GUI, or from within your programs to accelerate or replace existing applications or file development environments. SortCL improves runtime efficiency because it exploits file system I/O, memory and CPUs directly, and removes the heavy burden of multi-gigabyte file transformations from your database, ETL or BI tool.

In on CoSort SortCL job script -- and I/O pass, you can:

• Input one or more large sequential sources
• Run multiple transforms (filter, sort, etc.)
• Compare files to capture changes and BI
• Re-map, re-format, and pivot fields
• Create segmented, customized reports
• Convert data types and file formats
• Protect sensitive data at the field level
• Generate test data in custom formats
• Output to multiple targets simultaneously

This diagram, where the CoSort logo is centered, illustrates SortCL's capabilities for flat-file transformation, conversion, protection and reporting -- all in the same pass.

In addition to runtime performance, coding these operations in SortCL can also be faster and easier to maintain than proprietary programs, SQL, shell scripts, and ETL projects. SortCL uses a human-readable 4GL that leverages familiar data layout syntax, SQL manipulation concepts, and centralized metadata repositories. SortCL scripts all follow the same logical process flow (input, action, output), are self-documenting, and more concise than traditional alternatives.

IRI, The CoSort Company