{"id":11109,"date":"2017-06-12T12:11:12","date_gmt":"2017-06-12T16:11:12","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=11109"},"modified":"2026-02-23T15:23:47","modified_gmt":"2026-02-23T20:23:47","slug":"process-and-protect-hl7-data-with-voracity","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/data-transformation2\/process-and-protect-hl7-data-with-voracity\/","title":{"rendered":"Process and Protect HL7 Data with Voracity"},"content":{"rendered":"

Editors Note<\/strong>:\u00a0<\/em>It is now possible to discover, deliver, and de-identify PHI<\/a> in text-based HL7 files directly using the IRI DarkShield<\/a> data masking product for semi-structured and unstructured files, which is also included in the IRI data management platform mentioned in the discussion below where the HL7 files are structured. See this article on how to discovery and de-identify PHI in HL7 files directly with the fit-for-purpose HL7 search\/mask wizard in DarkShield<\/a>.<\/em><\/p>\n

This article shows how you can use <\/span>IRI Voracity<\/span><\/a> to rapidly integrate <\/span>and mask<\/span><\/i> sensitive healthcare data in the <\/span>HL7 file standard we introduced here<\/span><\/a>. <\/span><\/p>\n

The process begins as HL7 messages are converted into a relational format using <\/span>Flexter Data Liberator<\/span><\/a> software from Sonra. The output of that process will be an Oracle database — which means easy metadata discovery and data manipulation\/masking job creation in Voracity. \u00a0<\/span><\/p>\n

Flexter is XML parsing software for any industry data standard, including HL7. Voracity is a data management platform for data discovery (profiling), integration (ETL), migration, governance (DQ, masking, line age, etc.), and analytics (BI and data preparation), built on Eclipse. <\/span><\/p>\n

We will invoke Flexter\u2019s REST API to process the data from a Linux client. In the configuration file for the client we specify the REST end point, the output data store, and the connection details to our database:<\/span><\/p>\n

\"Output<\/a><\/p>\n

We execute the command: <\/span><\/p>\n

.\/push_full.sh ADT_01.messages.xml ADT_01.schema.zip<\/span><\/pre>\n
Then, we pass the filename of the XML message <\/span>ADT_01.messages.xml<\/span><\/i> and the filename of the XSD <\/span>ADT_01.schema.zip<\/span><\/i> to the command.<\/span><\/p>\n
\"XML<\/a><\/p>\n
It takes a few seconds to process the file. \u00a0XML parsing in Flexter happens in three steps:<\/span><\/p>\n
    \n
  1. Flexter analyses the XML schema and collects sample statistics from the provided XML files. It uses this information to generate an optimized and simplified relational target schema. <\/span><\/li>\n
  2. Flexter generates the mappings from the XML source elements to the relational target attributes. <\/span><\/li>\n
  3. Flexter processes the data locked away in the XML files and loads it into the target data store; e.g., an Oracle DB. It also generates the primary and foreign keys, constraints, and indexes.<\/span><\/li>\n<\/ol>\n
    Note<\/b>: Flexter uses a pluggable architecture, which allows us to easily add new input and output data stores or even different data formats such as JSON, EDI, etc. Flexter and Voracity can also run atop Spark to optionally distribute large processing loads across multiple servers.<\/span><\/p>\n
    Once Flexter has finished creating the target schema and we have successfully staged the HL7 data in our Oracle database, we can use Voracity to transform it further downstream.<\/span><\/p>\n
    HL7 Data Processing & Protection in IRI Voracity<\/b><\/h3>\n
    Now that we have processed the HL7 XML data into a usable format, we can utilize Voracity for ETL, cleansing, masking, reporting, or other processing. \u00a0<\/span><\/p>\n
    In this example, Voracity used the Oracle schema that Flexter generated, and it simply needed to execute the SQL command against the provided DDL to populate the tables. That<\/span> script was only needed because Flexter and Voracity were not on the same server; in most cases, they would and this process would be automatic (no DDL needed).<\/span><\/p>\n
    \"IRI<\/a><\/p>\n
    \"IRI<\/a><\/p>\n
    Shown above in <\/span><\/i>IRI Workbench<\/span><\/i><\/a> — the free Eclipse IDE for Voracity — are the HL7 tables in Oracle and their auto-exported content on display. On the right is the auto-built E-R Diagram and the SQL display of the target table we created and populated, again all in the same environment.<\/span><\/i><\/p>\n
    I created a single Voracity job to simultaneously (in the same script and I\/O pass):<\/span><\/p>\n