String-level expression logic is used to search and manipulate data according to certain patterns. This functionality is often associated with text editors and shell commands, but is rarely integrated into high volume data processing operations.
In other words, string parsing, pattern matching, and other low level manipulations must usually occur in a separate tools and I/O steps, increasing coding and processing overhead.
The SortCL program in IRI CoSort and IRI Voracity supports Perl Compatible Regular Expression (PCRE) logic for pattern matching, as well as find and replace, and other string and substring-level manipulations. SortCL also supports field padding and alignment, character validation, and field re-mapping.
More importantly, this intricate data transformation can occur in the same job script and and I/O pass with all the other simultaneous functions SortCL performs, like:
- Data Transformation (sort, join, aggregate, filter, remap, etc.)
- Data Migration (of data types and file formats)
- Data Protection (field level encryption, de-ID, masking, etc.)
- Reporting (batch, delta, detail and summary BI)
Please let us know below about your specific data manipulation requirements.