How do you manipulate VLDB table and flat-file data today?
ETL tools, Perl scripts, custom programs, and SQL procedures can be expensive and hard to maintain. Complex GUIs and coding syntax make specification difficult, and runtimes suffer under the weight of big data. Can you:
- transform huge data volumes without using our DB, or changing, your IT fabric; i.e. without Hadoop, NoSQL, or an ELT appliance?
- leverage multiple CPUs and cores for big jobs, run several tasks in the same I/O, and dynamically allocate resources to optimize performance?
- easily understand, share, and modify your data and job definitions without a major learning curve?
The SortCL program in the IRI CoSort packageis a long-proven (and now Eclipse-supported) 4GL that controls the heavy lifting of data transformation in the world's largest data warehouses, operational data stores and clickstream data webhouses.
Transform big data with speed and simplicity that no other tool or method can match. Transform without burdening the database ... without adding hardware ... without new platforms plagued by failure risks, high costs, and steep learning curves.
Optimize ETL operations as you combine sorts, joins, and aggregates in a single job script, partition, and I/O pass. At the same time, de-duplicate and filter, convert and re-map, lookup, rank, (de-)normalize, calculate, shift, encrypt to protect, and mask to re-cast.
Transform large volumes of data in many different table and file sources together. Discover, define, and expose your data and manipulation definitions in simple text file metadata repositories you can manage in the free IRI Workbench GUI, built on Eclipse. Use named fields for the mappings, as you:
- Map sources to targets
- Reduce script sizes and creation times
- Facilitate reorg and ETL operations
- Produce load and file-compare metadata
Do all the same transforms that slower and more complex SQL procedures or ETL tools do, and perform:
- change data capture
- row-column rotation (pivot/unpivot)
- slowly changing dimension reporting
- star (or snowflake) schema targeting
- static, structured, running, and windowed aggregates
- discrete and operative value lookups
- format mass and other value modifications
- data cleansing
- data protection (masking, encryption, pseudonymization, etc.)
Use Existing Metadata
SortCL and related facilities in the CoSort package accept many third-party data layouts, e.g., DB DDL, COBOL copybooks, CSV, LDIF and XML files, CLF and ELF web logs, and SQL*Loader Control File metadata. SortCL job scripts contain SQL-familiar commands that use and/or reference the layouts. Its GUI discovers and generates those layouts automatically.
AnalytixDS Mapping Manager and LiteSpeed Conversion technologies support SortCL script creation. Meta Integration Technology, Inc. (MITI) also has a metadata model bridge (MIMB) spoke to SortCL's data definition file format. If you have file layouts already defined for popular ETL tools like Informatica or DataStage, you can automatically produce equivalent layouts for use in SortCL, or accelerate those tools without having to manually redefine your metadata.
Interoperate & Accelerate
SortCL transformations work hand-in-hand with data extraction and loading utilities. SortCL can take piped data from IRI FACT (Fast Extract), and pipe it pre-sorted into database load utilities like SQL*Loader. SortCL can also connect through ODBC to other databases and Excel to acquire and deliver data.
SortCL transforms can run alongside ETL tools like Informatica and DataStage, to optimize their performance. SortCL jobs run on the command line, in batch scripts, from 3GL programs, via API calls, or in the IRI Workbench GUI, built on Eclipse™. Easily embed these transforms to accelerate your applications.
SortCL exploits CoSort's granular speed tuning and flexible CPU licensing. IRI's continuing innovation in parallel data movement, I/O and memory management, data manipulation functionality and consolidation -- along with our meaningful industry partnerships -- keep you at the leading edge of big data transformation.
Browse the Data Transformations that you can accomplish and combine in SortCL above. Learn more below: