CoSort Captures Changed Data in Files and Writes Delta Reports.

Home » Solutions » Data Governance » CDC Delta Reporting

Capture Changed Data (Delta Reporting)

Faster Data Acquisition, Retention, Refreshment. Easier File Compares.

Challenges:

Operational data undergoes constant changes, additions, and deletions relative to historical data. Data warehouse maintenance and business groups need to compare files, as well as capture, isolate, and analyze changes in data.

Unfortunately, many data acquisition tools and methods used for comparing files or table data, and processing that changed data are cumbersome, error-prone, and limited to one database's sources, performance, or log file data.

Also, as data volumes grow, the regular reading and updating of the entire database is an impractical alternative for refreshing the warehouse. In other words, the feasibility of doing in-database work for deltas shrink as table sizes grow.

Solutions:

Trap (capture) your database and legacy data changes off-line, in faster, more functional CoSort Sort Control Language (SortCL) job scripts that compare files with integrated join and reporting logic. The resulting reports flag and format the changes in customized layouts with meaningful information, or database feeds that can be populated in real-time or batch. This removes a major workload from the DBMS (which relies on triggers to update CDC tables), and provides the simultaneous opportunity to produce business intelligence from the change details.

In a big data change capture scenaior, you might want to sStart with a SELECT query in the compatible Fast Extract (FACT) tool for Oracle, DB2, etc., or use any utility to offload those rows generated after a certain timestamp to a flat file. Unqualified unloads are also very fast using FACT, and allow the entire transaction sets to be analyzed in the subsequent step:

The CoSort SortCL tool can conditionally joins a new flat file or on-line (ODBC) data source to one or more archives in those formats. The resulting target(s) can reflect the inserts, updates, deletes, and/or unchanged data in a custom report format, pre-sorted file for a database bulk loader, ODBC feed, or application hand-off format whre -- wherever deltas are required, and all at the same time.

At the same time, you can also count the number of new, modified, or missing records in your reports, or consult SortCL's runtime statistics (which count inner and outer matches at each join). Label, log, and analyze changes in the transaction data to spot red flags and assess trends forensically.

SortCL can, at the same time, also:

• re-format
• cross-calculate
• rank

the changed data, and use its field-level commands or functions to help you:

• transform
• cleanse
• protect
• present

the changed data.

Thus, you can cost-effectively capture changes in your data, and leverage simultaneous transformation and reporting opportunities, during high-volume file compare and data refresh operations.

IRI, The CoSort Company