|
Process and Output Only the Data You Need
Challenges:
You need to remove or save duplicate records, and include or exclude records based on business rules. This type of data scrubbing is often part of a separate pre-process step, which is sometimes performed by inefficient procedures, O/S commands or shell scripts -- programs that can also be difficult to maintain or re-use.
Solutions:
The SortCL tool within IRI's CoSort package offers several horizontal (record/row-level) and vertical (field/column-level) selection statements for managing the size, number, and flow of records before and after its central processing (sort, merge, join, report) actions. SortCL's elimination, reduction or hand-off of records provides for high performance, SQL-oriented data filtering during both input and output phases. Bulk data reduction increases efficiency for SortCL transformations, reports, loads, and your other downstream processes.
SortCL supports extensive condition logic (e.g. elaborate if-then-else expressions) to include, omit, and reformat records while it sorts, joins, aggregates, and reports. You can also filter records through de-duplication and byte, header, and record-specific commands. For example, you could specify to SortCL which field value ranges are valid in order to output only records outside those values. Similarly, you can validate character forms and perform other data integrity checks to reject or isolate bad records before loading them into a database.
Your specifications are based on your business logic, and are saved in text scripts for easy modification and re-use.
See also:
Solutions > Data Transformation > Scrub/Cleanse
Solutions > ETL/DB Acceleration
Products > CoSort > SortCL |
Thank you for sending us a request for information. We will get back to you shortly.
|
Case Studies
White Papers
