Data pipelines today must satisfy two critical, and sometimes competing, requirements: high-performance data processing and strict governance controls. More specifically, data-driven enterprises need to transform, cleanse, mask, and move big data quickly while also maintaining security, accountability, and compliance for both their data and their data processing jobs.

Version 11 of the IRI CoSort data manipulation package introduces a major new capability designed to address this challenge: an integrated runtime Operational Governance System (OGS). OGS is an optional framework that wraps around SortCL job executions, defines and enforces access control policies, verifies script integrity, and generates detailed Audit Log content.

Because the SortCL program powers multiple IRI products, the new custom-governed infrastructure for granular permission and logging benefits users of these tools:

• IRI CoSort  – to sort, transform, and report
• IRI FieldShield – to classify and mask Relational DataBases(RDBs) and flat files
• IRI NextForm – to migrate data types, file formats, and DB data
• IRI RowGen – to subset or synthesize test data for multiple targets
• IRI Ripcurrent  – to replicate and/or mask new DB rows in real-time
• IRI Voracity – to do all of the above, plus ETL, data quality, SCD, and more.

IRI DarkShield is not powered by SortCL and has its own authorization and logging framework. ¹

This article introduces the purposes, architecture, and key components of the OGS to support compliance with IRI data processing jobs, which, in some cases, support data governance goals (e.g., by cleansing and anonymizing data sources). 

Why Operational Governance Matters

Data operations increasingly run across hybrid cloud and on-premise infrastructures, involving diverse storage platforms, databases, and processing environments.

At the same time, enterprises must comply with internal policies and regulations requiring:

• controlled execution of data processing jobs
• strict access permissions
• the ability to audit operational activity
• protection against unauthorized script changes
• traceability of data manipulation workflows

Traditional governance solutions often rely on external orchestration systems or security layers. OGS takes a different approach by embedding governance directly inside the SortCL runtime environment, where the actual data processing occurs.

This approach enables organizations to enforce governance policies without sacrificing the performance advantages of the CoSort engine, which can perform high-volume transformations, joins, and masking operations in a single pass. 

What Does the IRI OGS Do?

The Operational Governance System is a policy-driven runtime framework that controls and records the execution of SortCL jobs. It is a production permission and logging system for: 

• Granular Role-based and attribute-based access controls (RBAC / ABAC)
• Job script integrity validation
• Metadata recording (basic data cataloging) 
• Machine-readable audit trails with built-in retention and wrangling facilities

Governance rules are defined in policy files that specify access permissions and runtime behavior for SortCL jobs and their components. 

OGS Architecture Overview

The Operational Governance System operates as a layer surrounding SortCL execution, parsing jobs scripts and applying governance policies before and after runtime events.

This architecture ensures data and job governance controls defined for SortCL-compatible runtime scripts are applied at runtime, regardless of whether jobs are launched from:

• The command line
• Within a batch script
• Scheduling tools
• The IRI Workbench GUI
• ETL, AI, TDM, and other applications integrating SortCL

Key OGS Components 

1. Policy-Based Access Control

The foundation of OGS is a policy framework. There is a Policy File that determines who can execute or modify SortCL jobs based on specified rules. It can be defined for a global or application-specific Policy File managed secure utility with rules for:

• runtime users, groups, and/or roles 
• job scripts
• job items, including sources and targets, fields, functions, classes, sets, conditions, etc. 

The system supports both RBAC and ABAC, allowing the system governor to describe and enforce rules based on:

• user roles
• user attributes
• job metadata (see above) 

This flexible model allows organizations to catalog and enforce fine-grained governance policies for SortCL operations that align with internal security frameworks. 

Note that OGS Policy Files also define whether job scripts will be signature-validated at runtime, along with the content and retention rules for the execution/audit logs (see below).

2. Job Script Integrity Verification

Data governance requires assurance that operational scripts cannot be altered without authorization. OGS introduces digital signature validation for SortCL job scripts.

This mechanism:

• detects unauthorized modifications to job files
• prevents execution of tampered scripts
• ensures trusted workflows in production environments

Script signing tools and encryption utilities are included in the CoSort 11 distribution to support these verification processes. (PRWeb)

3. Metadata & User Catalog

OGS Policy Files – in addition to defining role and attribute-based access controls, signature, and auditing policies – can also be used to describe and share information about users, groups, and roles, as well as individual job elements. The Policy File can preserve details about the:

• Name of, and metadata job items defined in SortCL scripts
• Users, Groups, and Roles are defined in the Policy File.

The Policy File can also function as a rudimentary data catalog for organizations that:

• do not yet have an enterprise-scale cataloging system.
• require a separate mechanism to catalog data involved only in SortCL-compatible operations.

While limited compared to pure data catalog platforms, the Policy File offers some advantages:

Feature	IRI Policy File	Enterprise Catalogs
Integration	Embedded directly in CoSort and compatible workflows	External, connector-based
Complexity	Lightweight, minimal setup	Robust, higher overhead
Governance	Immediate linkage to transformation rules	Broader governance frameworks
Cost/Footprint	Included with CoSort, no extra infrastructure	Separate licensing and deployment

Additional Benefits

• Operational Synergy: Metadata is tied directly to the execution environment, ensuring fidelity between catalog and runtime.
• Audit Support: Documentation of Form Items, rules, and lineage simplifies compliance reporting.
• Incremental Governance: Provides a starting point for cataloging that can later feed into larger enterprise systems.
• Complementary Role: Can serve as a metadata source for external catalogs, enriching them with precise transformation-level details.

Note, however, that the responsibility for collecting, inputting, and determining the visibility of Policy File information rests with the OGS Governor.

4. JSON-Formatted Job Audit Logs

Every governance decision and runtime action can be recorded in human- and machine-readable job logs. These customizable logs capture events such as:

• execution attempts, timings, and system resources (requested and used)
• policy evaluation results (permission outcomes)
• named users, scripts, data sources, classes, functions, and conditions, etc.

Because the new logs are produced in JSON format, they can easily integrate with:

• SIEM platforms like Splunk Enterprise Security
• observability tools, including the built-in IRI audit log wrangler supporting JPath queries
• AI and machine learning pipelines

OGS policies can also define log contents and retention rules. Logs can be created empty or full, contain a minimal set of attributes or those belonging to a certain category (e.g., performance), or be built for specific users with a custom mix of attributes. 

Log retention policy can be defined to move and delete log files on a schedule, and log contents can be filtered after generation to save disk space. See this article for more information.

5. Audit Log Query and Export 

To analyze operational activity, CoSort 11 includes a built-in audit log wrangling utility that runs on the command line for interactive and batch analysis, as well as for extracting SortCL job log data.

This tool allows both job performance analysts and data security analysts to:

• query logs using JSONPath expression logic
• review and export subsets of audit records
• Use delimited data for analysis in graphical tools like Excel and Splunk

Example query:

Select : [$.execution.startTime]
Where :  [$[?(@.execution.startTime > "2025-10-31T23:59:59")]]
Where :  [$[?(@.execution.startTime < "2025-12-01T00:00:00")]]
Select : [$.statistics.jobResults.returnCode]
Select : [$.statistics.jobResults.recordsProcessed]

The wrangler enables rapid detection of policy violations, job performance settings, and more, while also supporting post-facto qualitative log file culling to save disk space (where permitted).

Sample OGS Uses

Because SortCL powers many IRI tools, OGS provides a shared governance layer across the entire platform.

Data Integration
In IRI Voracity and IRI NextForm, governance policies can control ETL and data migration jobs that move data between databases, files, and cloud platforms.

Data Protection
In IRI FieldShield, OGS policies can restrict and log execution of masking operations involving highly sensitive datasets.

Test Data Provisioning
In IRI RowGen, governance policies can ensure that synthetic data generation workflows comply with internal testing policies.

Because these tools all leverage the same SortCL runtime, OGS ensures consistent governance enforcement across diverse data workflows.

OGS Benefits for Enterprise Users

Organizations adopting CoSort 11 gain several operational advantages:

Stronger Security
Fine-grained RBAC and ABAC policies protect sensitive data operations.

Trusted Production Workflows
Digital signatures ensure job scripts remain unmodified and authorized.

Comprehensive Auditability
JSON-based logs provide a detailed record of operational activity.

Unified Governance
Policies apply consistently across multiple IRI tools and workflows.

Compliance Readiness
Structured audit trails support regulatory frameworks and internal controls.

Runtime Efficiency
By integrating governance checks into SortCL’s existing multi-threaded and/or task-consolidating runtime environment rather than relying on external orchestration layers, OGS maintains the high data throughput and speed expected by existing CoSort users.

Conclusion

The Operational Governance System (OGS) in IRI CoSort 11 represents a major advancement in custom-governed infrastructure for high-performance data management jobs.

By combining:

• policy-driven, granular attribute- and role-based access control;
• script integrity verification;
• metadata-aware governance and cataloging; and
• robust audit logs and log management, 

OGS enables organizations using IRI software to maintain secure, compliant, and verifiable data workflows while continuing to leverage the speed and flexibility of the underlying SortCL program. This new infrastructure provides a unified governance layer that strengthens operational control across the entire IRI data management product ecosystem.

Upcoming articles on the IRI blog will dive deeper into the components of OGS, revealing their features and use cases.

1. DarkShield nevertheless also produces an additional JSON audit log largely compatible with the new SortCL audit log schema. Those log files can thus also be examined in the OGS audit log wrangler utility, or via SortCL, Splunk, and other log analytics tools, too, along with SortCL jobs that may run with DarkShield, like in RowGen or Voracity.