CoSort SortCL: Batch, GUI and API Calls for Single-Pass Data Processing CoSort

Home » Products » CoSort » SortCL

SortCL - Big Data Handling in One Pass

» Products

CoSort (Transform)

Unique Value Proposition

In a single pass through your single (file-based) source of the truth, CoSort's field-level transformation tool (SortCL) can: filter, migrate, integrate, stage, and present multiple, well-formatted views of big data; protect data on a need-to-know basis for outsourcing, compliance and test environments; and, prepare both real and test data for VLDB loads, ETL, analysis teams, and BI tools.

High-Speed Control of High-Volume Data

Question: Is SortCL a

A) multi-threaded big data integration and staging program
B) versatile data manipulation, comparison and conversion tool
C) custom detail, delta, and summary report generator
D) proven alternative for mainframe sort/sum/report jobs
E) targeted protection and governance tool for sensitive data
F) faster, simpler replacement for SQL, shell and 3GL apps
or G) all of the above?

Answer: G) All of the above. See also: FAQ > ETL

The Sort Control Language (SortCL) program is the most powerful and popular interface in IRI's flagship CoSort package. SortCL uses an open, explicit metadata for data definition and manipulation. SortCL is a self-documenting 4GL that anyone can read, recognize, and easily modify, and there is an Eclipse GUI supporting it. In other words, job creation, migration, and management are simple matters with a negligible learning curve.

SortCL job scripts perform - and combine - high-speed data transformation, conversion, reporting, and protection in large file environments.

From a solution standpoint, SortCL can be used to:

• Replace slower 3GL, Perl and SQL jobs
• Feed, speed, or re-seed BI/DW tools
• Relieve application and system overhead
• Filter, integrate, and stage large files
• Generate custom reports and hand-offs
• Accelerate bulk DB reorgs and loads
• Mask and encrypt fields and records
• Convert legacy sorts, files, and data types
• Migrate between flat and index files
• Synthesize safe test data in real formats
• Detect, capture, and audit changed data

What Can SortCL Do?

CoSort's Sort Control Language (SortCL) program can accept multiple inputs, including:

large files
pipes
ODBC-connected DB tables (and Excel)
user procedures

in multiple formats, process them in many ways, and produce one or more custom-formatted reports and output targets, all at once.

Specifically, SortCL can, in one job script and I/O pass, rapidly perform and combine all of these data transformations:

Filter	At the byte, field and record level, plus duplicate removal and saving
Segment	Conditional (include/omit) selection with if-then-else, else-if logic
Sort	Multiple keys, directions, sequences
Merge	Two or more pre-sorted files
Join	Two or more un/sorted files on many conditions for file compares and change data capture (deltas)
Re-map	Resize, reposition, and realign fields
Convert	Change data types (e.g. EBCDIC<>ASCII, Packed<>Numeric)
Re-format / Interchange	Convert between file formats (e.g. Text <>XML, VS<>RS, ISAM<>Vision, LDIF<>CSV)
Aggregate	Parallel roll-up and drill-down sum, min, max, average, and count values. Accumulation. Ranking.
Calculate	Math and trig functions across detail and summary rows
Sub-string	Bit-level manipulations and Perl-compatible regular expression logic for pattern matching, etc.
Validate	Check that character and field attributes match their specifications (i.e. "iscompares", gap analysis)
Sequence	For custom indexing, reporting, and database load operations
Lookup	Discrete field substitutions, pseudonymization, etc. using "set" file field dimensions
Protect	Encrypt data at the field level and audit data security measures. Also: anonymization, de-identification, filtering, and pseudonymization
Prototype	Create randomly-generated or set-selected (safe) test data fields
Report	Custom-formatted, segmented detail and summary targets
Custom	Complex field-level user functions (e.g. 3rd-party DQ libraries)

How Do I Run SortCL?

SortCL job scripts are called from the command line, in a batch script, through a job scheduler, from a program (as an executable or API call), or via the IRI Workbench GUI. The IRI Workbech GUI, built on Eclipse, allows you to import, modify, create, save, print, share, validate, and execute SortCL jobs across the network while providing direct access to your input and output files and tables. Importantly, the IRI Workbench integrates graphically with the scripts, which stay serialized in reusable text files you can run anytime in, or anywhere outside, the GUI.

What about Logging?

Application-level statistics can be output with each job; either to the screen or a file. In addition, the CoSort system logs runs in a self-appending file, and debugging information in a self-replacing file. On-screen monitoring options are available at various verbosity levels for runtime progress assessment. You can also toggle and secure an XML audit log for validating compliance and performing forensic application and data analysis.

Where Can I Get More Information on SortCL?

SortCL plays in an integral role in almost every Solution area described on this web site. Explore this linked list for a description of SortCL's capabilities in:

• Business Intelligence
• Data Governance
• Data Transformation
• ETL/DB Acceleration
• Field Protection
• File Interchange
• Third-party Sort Migration

IRI, The CoSort Company