Sorting remains a critical component of data processing. Ordering data is part of:
- database load, index, and query/search operations
- data warehouse sort, join, and aggregate transformations
- reporting, analytical, and testing environments
But as data source sizes increase, from hundreds of megabytes into terabyte levels and beyond, sorting can place an exponential demand on computing resources.
Mainframe sort/merge utilities are expensive to operate, use cryptic JCL syntax, and are functionally limited. Sort functions in databases, ETL and BI reporting tools, operating systems, and compilers are just not designed for big data.
|sort speed and scalability in volume||sorting and related functionality|
|data and file type support||GUI and/or parm syntax simplicity|
|event monitoring and error handling||metadata framework|
|performance tuning and logging||pricing and licensing models|
|plug-in compatibility or parm conversion accuracy||technical support speed|
|third-party hardware and software interoperability||vendor capabilities and reputation|
|implementation paradigm||skills gap (e.g. Hadoop), maintenance costs|
As the volume of data grows, so grows the value of IRI CoSort. CoSort is the world's first commercial sort/merge package for open systems. It continues to sell as a robust, commercial grade, cross-compatible:
- Unix file sort utility
- Windows sort program
- ETL, BI, and DB sort verb alternative
- Mainframe JCL sort/merge replacement
with state-of-the-art performance, industry-leading functionality, and the most familiar, intuitive user interfaces ... and without more hardware, Hadoop, in-memory DBs, or appliances.
CoSort will sort any number, size, and type of structured fields, keys, records, and files -- including mainframe binary, IP addresses, multi-byte Asian characters, Unicode, and so on. The CoSort engine scales linearly in volume, and allows granular tuning of CPU, memory, disk, and related resources. Multiple gigabytes sort in seconds on multi-CPU servers.
127,268,900 rows * 405 bytes/row = 51.5GB input file
CoSort total job time w/20-byte sort key @ 131 seconds = 2m:11s
Platform: x86 Linux development server using 32 of 64 cores
CoSort can also replace or convert third-party sort functions with proven libraries, tools, or services - saving time and money in batch operations and embedded applications. Ask about special incentives for migrating from a legacy sort product, and discounts for integrated distribution.
Sorting is Just the Beginning
CoSort also delivers the unique ability to simultaneously transform, migrate, report, and protect data at risk. The CoSort Sort Control Language (SortCL) program combines these functions in the same job script and I/O pass. Map multiple sources to multiple targets and formats while you sort.
SortCL is only one of several interfaces in the CoSort package available for standalone or integrated sort/merge operations. All sorting and transformation jobs can be scheduled, monitored, logged, audited, and otherwise managed in the IRI Workbench GUI, built on Eclipse™.
Beyond the CoSort package, these same SortCL-driven operations are also integral to the CoSort-including IRI Voracity data management platform where big data discovery, integration, migration, governance, and analytics are performed and combined. In Voracity, the CoSort sort engine (and SortCL scripts) are automatically used in (and created for): ETL, change data capture, DB subsetting, pseudonymization, synthetic test data, data wrangling, and bulk DB loading operations.