Learn IRI Software

• Installation
• IRI Workbench
• Job Scheduling
• Data Discovery
• Data Integration
• Data & Database Migration
• Data Governance
• Analytics & BI

Self-Directed Learning Site for IRI Software

This page links you to free, self-help content in the many data management objectives that IRI software products can help you achieve.

Please note that:

1. All IRI software users, from freemium to licensed subscribers, can access this material.  Only supported users, however, can get help from IRI beyond this content.
2. The IRI software product or licensing agreement you have may or not support the functionality, or entitle you to support for that feature even if the feature itself is enabled in your product. For more information, please refer to this product-feature matrix.
3. Additional training materials are available on request, custom designed for your company's requirements and certification-oriented learning. Please describe your material requirements below and an IRI services representative will contact you.

IRI Product Installation

• IRI Product Names & Architecture
• Installation Manual for all IRI Products
  • ​Hardware Configuration Prerequisites​
    • CoSort v10 Upgrade Advice
      • CoSort 10.0 Release Notes
    • CoSort v10 Best Practices
    • What's New in CoSort v10.5
      • CoSort 10.5 Release Notes
    • CoSort 10.x Tuning FAQs
  • IRI CellShield PE & EE V2 
  • SQL# & JDBC SQL Trail Proxy DDM
  • Updating IRI Workbench
• IRI Workbench Demo Projects in Git
• Connecting to RDBs (via O/JDBC)
  • Google BigQuery
  • HIVE
  • IBM DB2
  • Informix
  • MS Access
  • ​MS SQL
  • MySQL/MariaDB
  • ​Oracle
  • PostgreSQL
  • Salesforce
  • Snowflake
  • Teradata
  • Vertica
• Using Flat Files in Cloud Stores
• Connection Registry
• DB Data Type Mapping Wizard
• DSN Files
• IRI Ripcurrent (Real-Time DB CDC)
• Multi-Table Filtering​
• Connecting to NoSQL DBs
  • Cassandra
    • using Flat (CSV) File Import/Export
    • using Native Driver, All Collections
  • Elasticsearch
  • MarkLogic
  • MongoDB
    • using FieldShield (Flat Collections)
      • using Flat File Import/Export
      • using ODBC Driver, Flat Collections
      • using Native Driver, Flat Collections
    • using DarkShield (All Collections)
       

IRI Workbench - General

• Getting Started
• Job Design Methods
  1. New Job Wizards (see Welcome > First Steps)
  2. Dialogs (use ? for in-context Help)
  3. Script Editor & Outline (see product manuals for syntax)
  4. Mapping Diagrams (see Work Flow from the Palette below)
  5. erwin Mapping Manager
  6. DataSwitch (no-code, AI-enabled data engineering)
  7. Java apps via Gulfstream API
• Flow Design (how-to articles for Voracity ETL and other batch jobs)
  • Creating a Work Flow Using Existing Job (SortCL, etc.) Scripts (3 parts)
  • Creating a Work Flow from a Wizard (2 parts)
  • Creating a Work Flow from the Palette(2 parts)
    • Video: Work Flow (Palette)
• Compatible Plugins
  • SQL Scrapbook (built-in)
    • Creating New DB Tables
  • Eclipse Marketplace
  • KNIME
  • Wicked Shell
• Job Deployment

   

  • Command line ($sortcl /spec=jobname.scl)
  • Single Job Execution Options
  • Batch Job (Work Flow) Execution Options
  • Remote Connection/Executions
  • Running Voracity Jobs in Hadoop
    • ​When to Use Hadoop
      • Connecting to HortonWorks/Ambari
      • Connecting to Cloudera
      • Connecting to MapR
  • Running Voracity Jobs with KNIME
    • Job Scheduling
    • CI/CD (DevOps) Pipeline Automation
    • IRI Workbench Task Launcher
      • Eclipse Launch Configurations
      • Launch Configuration Groups
    • Oracle Job Scheduler
    • PowerShell Real-Time (on file events)
    • Stonebranch UAC

Data Discovery & Classification

• DB Profiling (includes 1+ table searching)
  • ER Diagramming
  • Multi-Table Filtering
  • Displaying Profiling Results in Splunk
• Flat-File Profiling
• NoSQL DB Data Class Search (and Mask)
• Structured Metadata Discovery
• IRI Metadata Search (via Git)
• Structured (DB & Flat-File) Data Classification
  • Video: Data Classification Setup 
  • Data Class Validation and Customization
  • Video: Data Class Modification (Library Editing)
• Directory Data Class Search (structured files)
• Schema Data Class Search (all tables in 1 or more RDB schema)
  • Video: Schema Data Class Search Wizard
• Schema-wide Pattern Search (all tables in an RDB schema)
  • Video: Schema Pattern Search Wizard
  • Schema Pattern Search to Data Class Association
• Unstructured File Search, Extract, Structure & Profile (text, documents, images, faces)
  •  Video: Dark Data Data Classification & Discovery
  • Video: Named Entity Recognition (NER) / NLP Searches via Machine Learning
• Data Discovery Charts in IRI Workbench

Data Integration

• Data Integration Architectures (and Voracity DI paradigms)
  • Enterprise Data Warehouse 
  • Logical Data Warehouse
  • Operational Data Store / Enterprise Data Hub
  • Production Analytic Platform (4-part series)
  • Data Lake
  • ETL vs. ELT
• ETL Job Design (see IRI Workbench > Flow Design above)
  • CoSort Script Reuse 
• ETL Execution (see IRI Workbench > Job Deployment above)
• Faster Extraction
• Data Transformation
  • Date Masking
  • Database (Table) Lookups
  • Filtering
  • Video: Sort/Join/Aggregate
  • Video: Inner Join
  • Video: Left Outer Join
  • Video: Sort
    • z/OS MVS & VSE JCL (ICEMAN) conversions
    • Links to other plug'n'play sort replacements
  • Video: Summary Aggregation
  • Row-Column Pivoting
  • Set Lookups
• Fast Loading
• Change Data Capture
• Slowly Changing Dimensions
• DB-Specific Optimization (see tabs)
• Video: Voracity ETL Workflow (wizard mode; see Flow Design above)
  • Oracle to MongoDB Example
• Voracity ETL Job Preview (via live or test data)
• Legacy ETL Optimization
  • ​Accelerating Informatica
  • Accelerating DataStage
  • Accelerating Pentaho
  • Accelerating Talend
• Legacy ETL Tool Migration (via erwin Smart Connector)
  • Re-platforming Informatica

Data & Database Migration

• File-Format Conversion
• Database Migration
• Database Subsetting
• Basic Data Replication
• Incremental Data Replication
  • Incremental Changes, Oracle to MongoDB
  • Real-Time Replication for MS SQL, MySQL, Oracle & PostgreSQL
• Data Federation
• Schema Migration: Relational to DataVault 2.0
• Schema Migration: Relational to Star
• Vision File Conversion
• XML (Complex) Parse/Process

Data Governance

• Data Masking
  • Dynamic Data Masking (DDM)
    • IRI FieldShield in DB Apps
    • Triggered FieldShield functions in PL/SQL
    • Proxy-based JDBC SQL Trail driver
    • Dynamic decryption of encrypted production data
    • .NET and Java (Hadoop) API libraries 
      • FieldShield ("Sandkey") SDK
  • Static Data Masking (SDM)
    • Which IRI Data Masking Product Should I Use?
    • Getting Started with FieldShield
      • Video: Single Table / File Masking demo
    • Which Data Masking Function Should I Use?
      • Blurring (Random Noise)
      • Bucketing (Generalization)
      • Deletion (GDPR Erasure)
      • Encryption
        • FieldShield Key Management in Azure Key Vault
        • FieldShield Key Management in Townsend Alliance Key Manager
      • Hashing
      • Pseudonymization
      • Redaction
      • Tokenization
    • Multi-Table RDB Masking (with Referential Integrity)
      • Real-Time (Incremental / Refresh) Class-Based Masking
      • Rule-based SDM using Column Names Only
        • Video: Multi-table masking demo
      • Rule-based SDM using Data Classes (Recommended)
        • Video Demo: Multi-table Classify/Mask (2021)
        • Video Series: Multi-table Masking Tutorials (2020)
          • ​​Data Classification & Discovery (see links above)
          • Applying Masking Rules to Classified Data
            • Video: Assigning masking rules
          • Data Class DB Masking Wizard
            • Video: Data Class DB Masking Wizard (2020)
    • Multi-Flat-File Masking (in 1 or more directories)
      • Data Class File Masking
    • ​Applying Field Rules Using Classification
    • Structured & Semi-Structured Data Sources
      • Amazon S3 file buckets
        • in DarkShield GUI
        • in DarkShield API
      • Amazon DynamoDB NoSQL (DarkShield API example)
      • Apache Cassandra
        • Structured or unstructured collections via DarkShield
        • Structured collections (only) via FieldShield
        • ASN.1 CDR files (see Example #5)
      • Azure BLOB Storage via DarkShield API
      • CosmosDB (MS NoSQL, DarkShield API Example)
      • Couchbase, Redis & Solr NoSQL DBs
      • CSV Files
      • Dates & Ages
      • Elasticsearch via DarkShield
      • Excel Spreadsheets: comparing CellShield, DarkShield & FieldShield approaches
        • via CellShield Personal Edition (PE)
        • via CellShield Enterprise Edition (EE)
        •  Video: via DarkShield
        • via DarkShield files API
        • via FieldShield (see example #6)
      • GCP Storage (Google Cloud Platform buckets) via DarkShield API
      • HL7 and X12 EDI formats via DarkShield API for files
        • HL7 with relations in Voracity(or see X12 via DarkShield, below)
      • Kafka Streams (via DarkShield API, but is also supported in FieldShield)
      • JSON & XML Files via DarkShield path filters
      • Live Feeds
      • MongoDB
        • Structured collections via Ex/Import 
        • Structured collections via J/ODBC
        • Structured collections via Native Driver
        • Structured & semi-/unstructured collections via DarkShield
      • MQTT (IoT) Streams
      • NIDs & SSNs
      • NRIC
      • Oracle & other RDBs
        • via FieldShield (for structured/1NF columns only, with classification & search)
          • Video Series: FieldShield Data Class DB Masking
        • via DarkShield (with structured & unstructured - C/BLOB, XML, text - columns)
          • Video: DarkShield RDB Schema Search/Mask
      • Parquet Files
      • Pentaho
      • PostgreSQL
      • SAP HANA (subset and mask)
      • Salesforce
      • SharePoint
      • Snowflake DB
      • Splunk
      • Web Logs
      •  Video: X12 via DarkShield (see blog with full X12 parser and HL7 here)
    • Re-ID Risk Determination (for HIPAA Expert Determination Method security rule)
      •  Video: Risk Scoring Wizard &  Quasi-Identifier Anonymization
      • HIPAA Data Certification Course
    • Unstructured (Dark) Data Sources
      • Getting Started with DarkShield (GUI)
        • ​Finding & Masking PII in Text/EDI files, MS Office, etc.
        • Finding & Masking PII in PDF and Image files via GUI
          • Video: DarkShield Search/Mask How-to
          • Video: DarkShield NER/ML How-to & FAQ (IRI Workbench)
        • DarkShield GUI Search Logs Analytics in Datadog
        • DarkShield GUI Search Logs & Analytics in Splunk
          • Analyzing DarkShield Search/Mask Results in Splunk
          • Universal Forwarder (sending search/mask logs to Splunk)
          • Adaptive Response to log events in Splunk ES
          • Invoking DarkShield from a Splunk Phantom Playbook 
        • DarkShield (Workbench) CLI SDK
      • DarkShield RPC API SDK - Plankton Web Services Framework
        • DarkShield Base API (for all sources, silos, and feeds)
        • DarkShield Files API (CSV/XML/JSON, text, PDFs, images, Parquet, MS Word & Excel)
          • Video: DarkShield RPC API
            • Video: DarkShield API searching and masking PII in GCP file stores
          • Credit Card PANs in Images
          • DICOM (Medical Imaging) Files
          • Image Preprocessing Techniques
          • GitHub Demo Repository
          • Load Balancing & Authentication (via NGINX reverse proxy)
          • Masking Files in Amazon, Google & Microsoft Cloud Storage
          • Named Entity Recognition (NER) via Tensorflow and PyTorch
          • Restoring Masked Values (decryption and reverse pseudonymization)
             
    • Data Quality
      • Data Quality Rule Wizards
      • Filter & De-Duplication
      • Fuzzy Searching
      • Data Validation
      • Data Unification (Homogenization, Reconciliation)
      • Finding Business Rule Violations
      • Finding Format Errors
         
    • Master Data Management (MDM)
      • MD Consolidation
      • MDM Registry (pending)
      • Alternative (via GIT)
         
    • Metadata Management
      • Teamwork (via GIT)
        • Version Control
        • Lineage (via Git)
        • Lineage & Impact Analysis (via Erwin/ADS)
        • Data & Metadata Lineage (built-in, pending)
        • Asset Security (via Git)
           
    • Role Based Access Controls (RBAC)
      • AD-compatible IAM (pending)
         
    • Test Data Management (TDM)
      • Considerations for developing test sets:
        • Goal Setting & Team Building
        • Test Data Needs Assessment
        • Test Data Generation & Provisioning
        • Test Data Sharing & Persistence
      • ​Windocks Virtualized DB (Masked/Syntheized Clones)
      • Value Labs Test Data Hub (On-Demand Dataset Portal)
    • Test (Synthetic) Data Generation, for:
      • Continuous Integration / Continuous Delivery-Deployment (CI/CD)
        • in Amazon CodePipeline
        • in GitLab
        • in Azure DevOps
        • in Jenkins
      • Data Values (via IRI Workbench Rule Wizards)
      • DataVault 2.0 Models
      • ETL Ops (Voracity mapping preview example)
      • RDBs via data masking (see SDM above)
      • RDBs via auto-parse/populate/generate (synthesis)
      • Video: IRI RowGen Test File & RDB Synthesis Operations 
      • RDBs via masked table subsets (DB subsetting with optional masking)
      • Flat files via the job wizard (e.g., CSV; see more formats below)
      • Specific formats or data classes
        • ASN.1 CDR files (see Example #4)
        • COBOL files
        • Datadog (XML or JSON)
        • Excel spreadsheets (see Examples #2 & 7)
        • Customer (transaction) data
        • EDI Files (HL7 & X12)
        • Personally Identifiable Information (PII) (fake PII for DevOps)
        • Credit Card Numbers
        • Individual CSV, JSON, XML Files & ODBC (SQL) Tables
        • MarkLogic DB
        • NIDs (US & Korea SSNs, Italy CF, Netherlands BSN)
        • PDF and Image Files
        • UUIDs/GUIDs
        • Cassandra (CSV)
        • MongoDB
        • Teradata
      • Set Files: A Primer
        • Using Set Files for Data Realism
        • Set File Creation in IRI Workbench
        • Drawing Values from Set Files
      • Weighted distributions
      • Java & Hadoop (API)
        • RowGen ("Outrigger") SDK

     

    Analytics & BI

    • Production Analytic Platform (4-part series)
    • Embedded BI
      • see report examples in CoSort manual (SortCL program)
    • IoT: Aggregation on the Edge
      • Video: IoT Edge Aggregation (MQTT) demo
    • Change Data Capture
    • Slowly Changing Dimensions
    • Predictive Analytics
    • Clickstream Analytics
    • BIRT Integration
    • Datadog Integration
    • Splunk Integrations
      • Voracity add-on for Splunk
      • Voracity app for Splunk (Splunkbase download)
      • Voracity results in Splunk via Universal Forwarder
    • Data Wrangling for Other BI & Analytic Tools
      • for Cubeware
      • for DWDigest
      • for IBM Cognos
      • for KNIME
      • for Microsoft Power BI
      • for Microstrategy
      • for Oracle Visualization Desktop
      • for R
      • for Spotfire
      • for Tableau