Pump Your DB
Very large database (VLDB) tables require a high-performance unloading (extraction) method for:
- Data warehouse ETL and ELT operations
- Classic (offline) reorgs
- Archival and storage
- Migration and replication
- Data interchange
Extraction performance in DB and DW environments is constrained by high data volumes and inefficient approaches. Read about a remedy for big Oracle tables in this blog post.
ODBC Select or "Fast Extract"
IRI CoSort (for data transformation and reporting) and IRI NextForm (for data and database migration) read data directly from relational and NoSQL DBs via ODBC or native protocols. Or, you can dump huge RDB tables in parallel to flat files using IRI FACT (Fast Extract).
Source-side selection in IRI CoSort via SQL and filtering commands in the CoSort SortCL program -- as well as change data capture (CDC) scripts -- can improve acquisition performance by reducing data bulk. SortCL supports any number of input tables, files, pipes, and procedures at once, and can apply specific filtering criteria for each data source.
You can also extract values from structured and unstructured document sources into flat files, based on searches for literal strings and Java regular expressions (patterns). Multiple data discovery wizards, which also profile such sources, exist in the IRI Workbench GUI, to help you find and use dark data.
For VLDB data acquisition, IRI FACT (Fast Extract) uses native drivers and parallel query methods to turn VLBD tables into flat files when bulk unloads are needed. FACT imposes no database overhead or configuration changes. FACT also bypasses the need to set up log sniffers and complex CDC in the database.
FACT uses SQL SELECT syntax in simple configuration files to unload data from: Oracle, DB2 UDB, Sybase, MS SQL Server, MySQL, Altibase, and Tibero.
During extraction, FACT reformats (e.g. delimiter) and converts the data (e.g. date types), and silos LOB fields. FACT also writes CoSort SortCL metadata for data transformation, conversion/replication, masking, and reporting, plus loader control file metadata for the source databases. This facilitates reorgs and ETL in the same I/O pass.
The IRI Workbench GUI for FACT, CoSort, etc. supports the automatic creation of tables and loader files for additional target databases including Teradata - in Eclipse.