IRI’s data management tools share a familiar and self-documenting metadata language called SortCL. All these tools — including CoSort, FieldShield, NextForm, and RowGen — require data definition file (DDF) layouts with /FIELD specifications for each data source so you can map your data and manage your metadata.
One of the best ways to speed up big data processing operations is to not process so much data in the first place; i.e. to eliminate unnecessary data ahead of time.
The IRI data management platform Voracity, as well as its constituent tools, can perform and speed big data warehouse extract, transform, load (ETL) operations, delaying the need for new hardware or expensive proprietary appliances: http://www.iri.com/blog/data-transformation2/a-big-data-quandary-hardware-or-software-appliances-or-cosort/
In 1992, Digital Equipment Corporation (DEC, long since acquired) asked IRI to develop a 4GL interface to CoSort in the syntax of the VAX VMS sort/merge utility.
This demonstration shows how to set up a sort job for CoSort using the IRI Workbench. The sort is accomplished using the SortCL language. This video takes a CSV input file, shows how to define the sort keys and options, and demonstrates how to define the targets for output.
This demonstration shows how to use the IRI Workbench to create an aggregation job using sums. Workbench is used to create the job script in the SortCL language.