For the last 30 or so years, the precursor to most large scale business intelligence (BI) environments has been the Enterprise Data Warehouse (EDW). A data warehouse (DW) is usually a central database (DB) for reporting, planning, and analyzing summarized, subject-matter data integrated from disparate, historical transaction sources.
The value of good test data to DBAs is well known:
“Testing of database-intensive applications has unique challenges that stem from hidden dependencies, subtle differences in data semantics, target database schemes, and implicit business rules.
Job schedulers are computer programs controlling other program executions behind the scenes, or as part of batch processes. This workload automation is typically coordinated so that traditional background data processing can be integrated with real-time business activities without depleting resources or interfering with other applications.
Big data integration activities can happen outside the database in an extract, transform, load (ETL) environment, or inside the database in ELT:
One example of an ELT operation would be Informatica’s Pushdown Optimization option, in which users transform data in a relational database like Oracle, or in Teradata.
As IRI CoSort integrates and stages big data from a variety of sources, it plays a natural role in producing data for reporting and analytics.
CoSort not only transforms data for loading data warehouse tables, it can report at the same time, or feed data in filtered, aggregated, sorted, and properly formatted subsets (like .CSV
The IRI data management platform Voracity, as well as its constituent tools, can perform and speed big data warehouse extract, transform, load (ETL) operations, delaying the need for new hardware or expensive proprietary appliances: http://www.iri.com/blog/data-transformation2/a-big-data-quandary-hardware-or-software-appliances-or-cosort/
Big Data Problem Big data volumes are growing exponentially. This phenomenon has been happening for years, but its pace has accelerated dramatically since 2012. Check out this blog entitled Big Data Just Beginning to Explode from CSC for a similar viewpoint on the emergence of big data and the challenges related thereto.