Innovative Routines International (IRI), Inc., The CoSort Company

Delivering High-Speed Control of High-Volume Data

What Is SortCL?

A) A parallel ETL engine for big file staging and integration
B) A flat-file manipulation, comparison and conversion tool
C) A custom detail, delta, and summary report generator
D) 'Beyond the mainframe' sort/merge/sum software
E) A protection and governance tool for flat file data

Answer: All of the above. See also: FAQ > ETL

The Sort Control Language (SortCL) program is the most powerful and popular tool in IRI's flagship CoSort package. SortCL is an open, explicit 4GL for data definition and manipulation that enables high-speed and simultaneous data transformation, conversion, reporting, and protection in large file environments.

SortCL runs multiple data transformations at once, and can:

• Replace slower 3GL, Perl and SQL procedures
• Feed, speed, or reseed BI, DB and ETL jobs
• Relieve application and system overhead
• Filter, integrate, and stage large data volumes
• Generate custom reports and hand-offs
• Accelerate bulk database reorgs and loads
• Anonymize, de-ID or encrypt fields and records
• Re-host legacy sorts and convert data types
• Migrate from one flat/index file format to another
• Synthesize safe test data in real file formats
• Detect, capture, and audit changed data

What Data Transformations Can I Perform?

CoSort's Sort Control Language (SortCL) program can accept multiple inputs (files, pipes and procedures) in multiple formats, process them in many ways, and produce one or more custom-formatted reports and output targets, all at once.

Specifically, SortCL can, in one job script and I/O pass, rapidly perform and combine all of these data transformations:

Filter	At the byte, field and record level, plus duplicate removal and saving
Segment	Conditional (include/omit) selection with if-then-else, else-if logic
Sort	Multiple keys, directions, sequences
Merge	Two or more pre-sorted files
Join	Two or more un/sorted files on many conditions for file compares and change data capture (deltas)
Re-map	Resize, reposition, and realign fields
Convert	Change data types (e.g. EBCDIC<>ASCII, Packed<>Numeric)
Re-format / Interchange	Convert between file formats (e.g. Text <>XML, VS<>RS, ISAM<>Vision, LDIF<>CSV)
Aggregate	Parallel roll-up and drill-down sum, min, max, average, and count values. Accumulation. Ranking.
Calculate	Math and trig functions across detail and summary rows
Sub-string	Bit-level manipulations and Perl-compatible regular expression logic for pattern matching, etc.
Validate	Check that character and field attributes match their specifications (i.e. "iscompares", gap analysis)
Sequence	For custom indexing, reporting, and database load operations
Lookup	Discrete field substitutions, pseudonymization, etc. using "set" file field dimensions
Protect	Encrypt data at the field level and audit data security measures. Also: anonymization, de-identification, filtering, and pseudonymization
Prototype	Create randomly-generated or set-selected (safe) test data fields
Report	Custom-formatted, segmented detail and summary targets
Custom	Complex field-level user functions (e.g. 3rd-party DQ libraries)

How Do I Run SortCL?

SortCL job scripts are called from the command line, in a batch script, through a job scheduler, from a program (as an executable or C API call), or via CoSort's free Java gui2scl (Graphical User Interface to SortCL) client. The GUI allows you to import, modify, create, save, print, share, validate, and execute SortCL job scripts across the network.

What about Logging?

Application-level statistics can be output with each job; either to the screen or a file. In addition, the CoSort system logs runs in a self-appending file, and debugging information in a self-replacing file. On-screen monitoring options are available at various verbosity levels for runtime progress assessment. You can also enable and secure an XML audit log for validating compliance and performing forensic application and data analysis.

Where Can I Get More Information on SortCL?

SortCL plays in an integral role in almost every Solution area described on this web site. Explore this linked list for a description of SortCL's capabilities in:

• Business Intelligence
• Data Governance
• Data Transformation
• ETL/DB Acceleration
• Field Protection
• File Interchange
• Sort Migration