{"id":14711,"date":"2021-08-26T12:52:14","date_gmt":"2021-08-26T16:52:14","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=14711"},"modified":"2025-04-15T15:57:38","modified_gmt":"2025-04-15T19:57:38","slug":"sortcl-compatible-asn1-jobs","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/etl\/sortcl-compatible-asn1-jobs\/","title":{"rendered":"Sample CDR Data Processing Jobs"},"content":{"rendered":"

In the previous article<\/a>, new support for <\/i>SortCL<\/i><\/a>-compatible jobs in the <\/i>IRI Voracity<\/i><\/a> data processing ecosystem for ASN.1-encoded data files was introduced. This article takes a more in-depth look through sample jobs demonstrating various use cases, ASN.1 encoding rules, schema files, and the Protocol Data Units (PDUs) involved with data defined in ASN.1 schemas.<\/i><\/p>\n

Note that many of the jobs shown herein are presented in raw text script form, after serialization in the IRI Workbench<\/a> graphical job design environment, built on Eclipse. Also, a license from OSS Nokalva<\/a> is needed along with your IRI software license to support the ASN.1 Encoding Rules used in SortCL jobs.<\/p>\n

A zip file of all the examples discussed in this article may be downloaded from here<\/a>.<\/p>\n

Example 1: Output BER-encoded data from existing compliant CSV file<\/b><\/h6>\n

Here is an example of an IRI CoSort<\/a> data transformation job that sorts and maps CSV data to a BER-encoded file meeting the specifications of TAP3, a standard for exchanging data between mobile network operators for roaming calls. The number of fields have been truncated for readability.\u00a0 :<\/p>\n

\/INFILE=ASN.csv\r\n    \/PROCESS=CSV\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_SENDER, TYPE=ASCII, POSITION=1, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_RECIPIENT, TYPE=ASCII, POSITION=2, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILESEQUENCENUMBER, TYPE=ASCII, POSITION=3, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILECREATIONTIMESTAMP_LOCALTIMESTAMP, TYPE=ASCII, POSITION=4, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILECREATIONTIMESTAMP_UTCTIMEOFFSET, TYPE=ASCII, POSITION=5, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_TRANSFERCUTOFFTIMESTAMP_LOCALTIMESTAMP, TYPE=ASCII, POSITION=6, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_TRANSFERCUTOFFTIMESTAMP_UTCTIMEOFFSET, TYPE=ASCII, POSITION=7, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILEAVAILABLETIMESTAMP_LOCALTIMESTAMP, TYPE=ASCII, POSITION=8, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILEAVAILABLETIMESTAMP_UTCTIMEOFFSET, TYPE=ASCII, POSITION=9, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_SPECIFICATIONVERSIONNUMBER, TYPE=NUMERIC, POSITION=10, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_RELEASEVERSIONNUMBER, TYPE=NUMERIC, POSITION=11, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILETYPEINDICATOR, TYPE=ASCII, POSITION=12, SEPARATOR=\",\")\r\n.........\r\n\/SORT\r\n    \/KEY=TRANSFERBATCH_BATCHCONTROLINFO_SENDER\r\n\r\n\/OUTFILE=\"TAP-0310.asn,BER;tap.ber\"\r\n    \/PROCESS=ASN1<\/i>\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_SENDER, TYPE=ASCII, POSITION=1, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_RECIPIENT, TYPE=ASCII, POSITION=2, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILESEQUENCENUMBER, TYPE=ASCII, POSITION=3, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILECREATIONTIMESTAMP_LOCALTIMESTAMP, TYPE=ASCII, POSITION=4, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILECREATIONTIMESTAMP_UTCTIMEOFFSET, TYPE=ASCII, POSITION=5, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_TRANSFERCUTOFFTIMESTAMP_LOCALTIMESTAMP, TYPE=ASCII, POSITION=6, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_TRANSFERCUTOFFTIMESTAMP_UTCTIMEOFFSET, TYPE=ASCII, POSITION=7, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILEAVAILABLETIMESTAMP_LOCALTIMESTAMP, TYPE=ASCII, POSITION=8, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILEAVAILABLETIMESTAMP_UTCTIMEOFFSET, TYPE=ASCII, POSITION=9, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_SPECIFICATIONVERSIONNUMBER, TYPE=NUMERIC, POSITION=10, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_RELEASEVERSIONNUMBER, TYPE=NUMERIC, POSITION=11, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILETYPEINDICATOR, TYPE=ASCII, POSITION=12, SEPARATOR=\",\")\r\n.........<\/pre>\n
NOTE: The input data in the CSV file must meet the constraints of the ASN.1 specification (TAP-0310.asn in this case).<\/p>\n
Example 2: Sort BER-encoded file and output to Excel<\/b><\/h6>\n
Here is another example of CoSort taking a TAP3 BER-encoded call detail record, sorting it on one of the input fields, and outputting to an Excel compatible file in XLSX format.:<\/p>\n
\/INFILE=\"TAP-0310.asn,BER;tap.ber\"\r\n    \/PROCESS=ASN1\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_SENDER, TYPE=ASCII, POSITION=1, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_RECIPIENT, TYPE=ASCII, POSITION=2, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILESEQUENCENUMBER, TYPE=ASCII, POSITION=3, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILECREATIONTIMESTAMP_LOCALTIMESTAMP, TYPE=ASCII, POSITION=4, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILECREATIONTIMESTAMP_UTCTIMEOFFSET, TYPE=ASCII, POSITION=5, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_TRANSFERCUTOFFTIMESTAMP_LOCALTIMESTAMP, TYPE=ASCII, POSITION=6, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_TRANSFERCUTOFFTIMESTAMP_UTCTIMEOFFSET, TYPE=ASCII, POSITION=7, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILEAVAILABLETIMESTAMP_LOCALTIMESTAMP, TYPE=ASCII, POSITION=8, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILEAVAILABLETIMESTAMP_UTCTIMEOFFSET, TYPE=ASCII, POSITION=9, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_SPECIFICATIONVERSIONNUMBER, TYPE=NUMERIC, POSITION=10, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_RELEASEVERSIONNUMBER, TYPE=NUMERIC, POSITION=11, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILETYPEINDICATOR, TYPE=ASCII, POSITION=12, SEPARATOR=\",\")\r\n    .........\r\n\/SORT\r\n    \/KEY=TRANSFERBATCH_BATCHCONTROLINFO_RECIPIENT\r\n\r\n\/OUTFILE=\"'Sheet1',HEADER;outTAP.xlsx\"\r\n    \/PROCESS=XLSX\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_SENDER, TYPE=ASCII, POSITION=1, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_RECIPIENT, TYPE=ASCII, POSITION=2, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILESEQUENCENUMBER, TYPE=ASCII, POSITION=3, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILECREATIONTIMESTAMP_LOCALTIMESTAMP, TYPE=ASCII, POSITION=4, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILECREATIONTIMESTAMP_UTCTIMEOFFSET, TYPE=ASCII, POSITION=5, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_TRANSFERCUTOFFTIMESTAMP_LOCALTIMESTAMP, TYPE=ASCII, POSITION=6, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_TRANSFERCUTOFFTIMESTAMP_UTCTIMEOFFSET, TYPE=ASCII, POSITION=7, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILEAVAILABLETIMESTAMP_LOCALTIMESTAMP, TYPE=ASCII, POSITION=8, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILEAVAILABLETIMESTAMP_UTCTIMEOFFSET, TYPE=ASCII, POSITION=9, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_SPECIFICATIONVERSIONNUMBER, TYPE=NUMERIC, POSITION=10, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_RELEASEVERSIONNUMBER, TYPE=NUMERIC, POSITION=11, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILETYPEINDICATOR, TYPE=ASCII, POSITION=12, SEPARATOR=\",\")\r\n    .........\r\n\r\n<\/pre>\n
\"Converting
Converting a TAP3 call detail record to Excel format and sorting the records by the TADIG code of the call recipient.<\/figcaption><\/figure>\n
Octet strings from the TAP3 schema are output as hex values. Hex values can be decoded to ASCII with the decode_hex function.<\/pre>\n
Example 3: Masking Call Detail Records using IRI FieldShield<\/b><\/h6>\n
Given this original set of call detail records encoded with Basic Encoding Rules by the definitions provided in an ASN.1 schema,<\/p>\n
\"\"<\/p>\n
and running this IRI FieldShield<\/a> data masking job script in the same SortCL syntax,<\/p>\n
\/INFILES=\"{C:\/Eclipse\/Workbench_10_08\/workspace\/ASN.1\/TAP-0310.asn},BER,DataInterChange;C:\/Eclipse\/Workbench_10_08\/workspace\/ASN.1\/tapgen6.ber\"\r\n    \/PROCESS=ASN1\r\n    \/ALIAS=tapgen6 \r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_SENDER, TYPE=ASCII, POSITION=1, SEPARATOR=\",\") \r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_RECIPIENT, TYPE=ASCII, POSITION=2, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILESEQUENCENUMBER, TYPE=ASCII, POSITION=3, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILECREATIONTIMESTAMP_LOCALTIMESTAMP, TYPE=ASCII, POSITION=4, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILECREATIONTIMESTAMP_UTCTIMEOFFSET, TYPE=ASCII, POSITION=5, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_TRANSFERCUTOFFTIMESTAMP_LOCALTIMESTAMP, TYPE=ASCII, POSITION=6, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_TRANSFERCUTOFFTIMESTAMP_UTCTIMEOFFSET, TYPE=ASCII, POSITION=7, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILEAVAILABLETIMESTAMP_LOCALTIMESTAMP, TYPE=ASCII, POSITION=8, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_FILEAVAILABLETIMESTAMP_UTCTIMEOFFSET, TYPE=ASCII, POSITION=9, SEPARATOR=\",\")\r\n    \/FIELD=(TRANSFERBATCH_BATCHCONTROLINFO_SPECIFICATIONVERSIONNUMBER, TYPE=NUMERIC, POSITION=10, SEPARATOR=\",\")\r\n.......\r\n\/REPORT\r\n\/OUTFILE=masked-tap-gen\r\n    \/PROCESS=RECORD \r\n    \/FIELD=(PSEUDO_TRANSFERBATCH_BATCHCONTROLINFO_SENDER, TYPE=ASCII, POSITION=1, SEPARATOR=\",\", SET=\"C:\/IRI\/cosort105\/sets\/tadig-codes.set\")\r\n    \/FIELD=(PSEUDO_TRANSFERBATCH_BATCHCONTROLINFO_RECIPIENT, TYPE=ASCII, POSITION=2, SEPARATOR=\",\", SET=\"C:\/IRI\/cosort105\/sets\/tadig-codes.set\")\r\n    \/FIELD=(DECODE_TRANSFERBATCH_BATCHCONTROLINFO_FILESEQUENCENUMBER=decode_hex(TRANSFERBATCH_BATCHCONTROLINFO_FILESEQUENCENUMBER), TYPE=ASCII, POSITION=3, SEPARATOR=\",\")\r\n    \/FIELD=(PSEUDO_TRANSFERBATCH_BATCHCONTROLINFO_FILECREATIONTIMESTAMP_LOCALTI, TYPE=ASCII, POSITION=4, SEPARATOR=\",\", SET=\"C:\/Eclipse\/Workbench_10_08\/workspace\/ASN.1\/timestamps.set\")\r\n    \/FIELD=(DECODE_TRANSFERBATCH_BATCHCONTROLINFO_FILECREATIONTIMESTAMP_UTCTIMEOFFSET=decode_hex(TRANSFERBATCH_BATCHCONTROLINFO_FILECREATIONTIMESTAMP_UTCTIMEOFFSET), TYPE=ASCII, POSITION=5, SEPARATOR=\",\")\r\n    \/FIELD=(PSEUDO_TRANSFERBATCH_BATCHCONTROLINFO_TRANSFERCUTOFFTIMESTAMP_LOCAL, TYPE=ASCII, POSITION=6, SEPARATOR=\",\", SET=\"C:\/Eclipse\/Workbench_10_08\/workspace\/ASN.1\/timestamps.set\")\r\n    \/FIELD=(DECODE_TRANSFERBATCH_BATCHCONTROLINFO_TRANSFERCUTOFFTIMESTAMP_UTCTIMEOFFSET=decode_hex(TRANSFERBATCH_BATCHCONTROLINFO_TRANSFERCUTOFFTIMESTAMP_UTCTIMEOFFSET), TYPE=ASCII, POSITION=7, SEPARATOR=\",\")\r\n    \/FIELD=(PSEUDO_TRANSFERBATCH_BATCHCONTROLINFO_FILEAVAILABLETIMESTAMP_LOCALTIMESTAMP, TYPE=ASCII, POSITION=8, SEPARATOR=\",\", SET=\"C:\/Eclipse\/Workbench_10_08\/workspace\/ASN.1\/timestamps.set\")\r\n    \/FIELD=(DECODE_TRANSFERBATCH_BATCHCONTROLINFO_FILEAVAILABLETIMESTAMP_UTCTIMEOFFSET=decode_hex(TRANSFERBATCH_BATCHCONTROLINFO_FILEAVAILABLETIMESTAMP_UTCTIMEOFFSET), TYPE=ASCII, POSITION=9, SEPARATOR=\",\")\r\n    \/FIELD=(DECODE_TRANSFERBATCH_BATCHCONTROLINFO_SPECIFICATIONVERSIONNUMBER=decode_hex(TRANSFERBATCH_BATCHCONTROLINFO_SPECIFICATIONVERSIONNUMBER), TYPE=NUMERIC, POSITION=10, SEPARATOR=\",\")\r\n\r\n.......<\/pre>\n
results in this delimited flat file with pseudonymized data.<\/p>\n
\"\"<\/p>\n
Note that the\u00a0decode_hex<\/em> function is only applied for ease of viewing, since ASN.1 octet string values are output as the hex representation. Set files were used to pull random TADIG code and timestamp replacement values.<\/p>\n
Here is a visual overview of the script in IRI Workbench, where the orange arrows indicate fields that have had transforms applied to them.<\/p>\n
\"SortCL<\/p>\n
Example 4: Converting TAP3 Call Detail Records to other Formats and Layouts<\/b><\/h6>\n
The example included in the zip folder will output to a text file. Alternatively, data could be transported to a database using a script similar to the one shown below.<\/p>\n
Truncated script outputting to database:<\/p>\n
\/INFILE=\u201dTAP0310.asn;tap.ber\u201d\r\n    \/PROCESS=ASN1\r\n    \/FIELD=(transferBatch_batchControlInfo_sender, POSITION=1, SEPARATOR=',')\r\n    \/FIELD=(transferBatch_batchControlInfo_recipient, POSITION=2, SEPARATOR=',')\r\n    \/FIELD=(transferBatch_batchControlInfo_fileSequenceNumber, POS=3, SEP=',')\r\n    \/FIELD=(transferBatch_batchControlInfo_fileCreationTimeStamp_localTimeStamp, POS=4, SEP=',')\r\n    \/FIELD=(transferBatch_batchControlInfo_fileCreationTimeStamp_utcTimeOffset, POS=5, SEP=',')\r\n    \/FIELD=(transferBatch_batchControlInfo_transferCutOffTimeStamp_localTimeStamp, POS=6, SEP=',')\r\n    \/FIELD=(transferBatch_batchControlInfo_transferCutOffTimeStamp_utcTimeOffset, POS=7, SEP=',')\r\n    \/FIELD=(transferBatch_batchControlInfo_fileAvailableTimeStamp_localTimeStamp, POS=8, SEP=',')\r\n    \/FIELD=(transferBatch_batchControlInfo_fileAvailableTimeStamp_utcTimeOffset, position=9, SEP=',')\r\n\u2026\u2026\u2026.\r\n\r\n\/REPORT\r\n\r\n\/OUTFILE=\"asn.base;DSN=New My SQL;UID=root;PWD=$PASS;\"\r\n    \/PROCESS=ODBC\r\n    \/FIELD=(transferBatch_batchControlInfo_sender, POS=1,SEP=\u201d\\t\u201d,ODEF=\"transferBatch_batchControlInfo_sender\")\r\n    \/FIELD=(transferBatch_batchControlInfo_recipient, POS=2, SEP=\u201d\\t\u201d,ODEF=\"transferBatch_batchControlInfo_recipient\")\r\n    \/FIELD=(transferBatch_batchControlInfo_fileSequenceNumber, POS=3, SEP=\u201d\\t\u201d,ODEF=\"transferBatch_batchControlInfo_fileSequenceNumber\")\r\n    \/FIELD=(transferBatch_batchControlInfo_fileCreationTimeStamp_localTimeStamp, POS=4, SEP=\u201d\\t\u201d,ODEF=\"transferBatch_batchControlInfo_fileCreationTimeStamp_localTimeStamp\")\r\n    \/FIELD=(transferBatch_batchControlInfo_fileCreationTimeStamp_utcTimeOffset, POS=5, SEP=\u201d\\t\u201d,ODEF=\"transferBatch_batchControlInfo_fileCreationTimeStamp_utcTimeOffset\")\r\n    \/FIELD=(transferBatch_batchControlInfo_transferCutOffTimeStamp_localTimeStamp, POS=6, SEP=\u201d\\t\u201d,ODEF=\"transferBatch_batchControlInfo_transferCutOffTimeStamp_localTimeStamp\")\r\n    \/FIELD=(transferBatch_batchControlInfo_transferCutOffTimeStamp_utcTimeOffset, POS=7, SEP=\u201d\\t\u201d,ODEF=\"transferBatch_batchControlInfo_transferCutOffTimeStamp_utcTimeOffset\")\r\n    \/FIELD=(transferBatch_batchControlInfo_fileAvailableTimeStamp_localTimeStamp, POS=8, SEP=\u201d\\t\u201d,ODEF=\"transferBatch_batchControlInfo_fileAvailableTimeStamp_localTimeStamp\")\r\n    \/FIELD=(transferBatch_batchControlInfo_fileAvailableTimeStamp_utcTimeOffset, POS=9, SEP=\u201d\\t\u201d,ODEF=\"transferBatch_batchControlInfo_fileAvailableTimeStamp_utcTimeOffset\")\r\n\u2026\u2026\u2026.<\/pre>\n
The image below shows the output of TAP3 BER-encoded data (top portion of image) to a comma-delimited text file.<\/p>\n
\"\"<\/a><\/p>\n
Example 5: Synthesizing a BER-encoded TAP3 Test File<\/b><\/h6>\n
The tapgen.rcl.txt file in the examples folder linked at the beginning of the post displays an example of synthesizing BER-encoded data that adheres to the standard TAP3 schema. This script could \u00a0fine-tuned to generate a larger range of realistic and random data (but still within the constraints of the TAP3 specification), but it does show a simple method of generating a TAP3-compliant file.<\/p>\n
Truncated script:<\/span><\/p>\n
\/INFILE=rfp<\/span>\r\n    \/PROCESS=RANDOM<\/span>\r\n    \/INCOLLECT=1<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_sender, POSITION=1, SEPARATOR=\",\",SET={5052594854})<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_recipient, POSITION=2, SEPARATOR=\",\",SET={484B474854})<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_fileSequenceNumber, POSITION=3, SEPARATOR=\",\",SET={3030303033})<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_fileCreationTimeStamp_localTimeStamp, POSITION=4, SEPARATOR=\",\",SET={3230303430323034313134343438})<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_fileCreationTimeStamp_utcTimeOffset, POSITION=5, SEPARATOR=\",\",SET={2D30343030})<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_transferCutOffTimeStamp_localTimeStamp, POSITION=6, SEPARATOR=\",\",SET={3230303430323034313134343438})<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_transferCutOffTimeStamp_utcTimeOffset, POSITION=7, SEPARATOR=\",\",SET={2D30343030})<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_fileAvailableTimeStamp_localTimeStamp, POSITION=8, SEPARATOR=\",\",SET={3230303430323034313134343438})<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_fileAvailableTimeStamp_utcTimeOffset, POSITION=9, SEPARATOR=\",\",SET={2D30343030})<\/span>\r\n    \u2026\u2026\u2026.<\/span>\r\n\/REPORT<\/span>\r\n\/OUTFILE=\"TAP-0310.asn;tapgen.ber\"<\/span>\r\n    \/PROCESS=ASN1<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_sender, POSITION=1, SEPARATOR=\",\")<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_recipient, POSITION=2, SEPARATOR=\",\")<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_fileSequenceNumber, POSITION=3, SEPARATOR=\",\")<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_fileCreationTimeStamp_localTimeStamp, POSITION=4, SEPARATOR=\",\")<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_fileCreationTimeStamp_utcTimeOffset, POSITION=5, SEPARATOR=\",\")<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_transferCutOffTimeStamp_localTimeStamp, POSITION=6, SEPARATOR=\",\")<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_transferCutOffTimeStamp_utcTimeOffset, POSITION=7, SEPARATOR=\",\")<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_fileAvailableTimeStamp_localTimeStamp, POSITION=8, SEPARATOR=\",\")<\/span>\r\n    \/FIELD=(transferBatch_batchControlInfo_fileAvailableTimeStamp_utcTimeOffset, POSITION=9, SEPARATOR=\",\")<\/span>\r\n    \u2026\u2026\u2026.<\/span>\r\n\r\n<\/pre>\n
Example 6: TAP3 BER-encoded file to JSON<\/b><\/h6>\n
The input is a non-human readable, BER-encoded file based on the TAP3 ASN.1 schema.<\/p>\n
The input looks like this when opened in a basic text editor.<\/p>\n
\"\"<\/a><\/p>\n
SortCL can easily translate this to JSON, however. In the image below,the SortCL script top left is used to make this transformation to the JSON file shown in the bottom middle of the image. The job makes use of a sortcl-compatible D<\/b>ata D<\/b>efinition F<\/b>ormat (DDF) file<\/a> containing the field definitions; a small part of that is shown top right.<\/p>\n
\"\"<\/a><\/p>\n
The DDF file can be generated easily in an interactive and visual way through the Metadata Discovery\u00a0 wizard<\/a> in IRI Workbench, which is also subsumed in the Data Source definition pages of most of the fit-for-purpose new job wizards in Workbench. Alternatively, the executable asn1_2ddf<\/i><\/b> program provides a more batch-able, automated approach to generating DDF files for ASN.1-encoded data.<\/p>\n
There are many functions that could have been applied to this data, such as sorting by field(s), performing selective encryption or pseudonymization, adding fields, and so much more. However, this script simply acts as a converter of the ASN.1-encoded data to a more human readable JSON format.<\/p>\n
Example 7: Reporting on one PDU from an ASN.1 file defining several PDUs<\/b>
\n<\/b><\/h6>\n
This is for e1ap<\/a>, a protocol related to 5G.<\/p>\n
Selecting from a specific PDU:<\/i><\/p>\n
This job gets information from the PDU with the type name TimeToWait. e1ap has many PDUs, but one must be chosen. In TAP3, however, there is a single PDU that can be referenced called DataInterChange. In that case, the PDU does not need to be specified since all types map out to that single PDU.<\/p>\n
Script:<\/p>\n
\/INFILE=\u201de1ap.asn,PER,TimeToWait;E1AP-PDU_Reset.per\u201d\r\n    \/PROCESS=ASN1\r\n    \/FIELD=(FIELD1,POSITION=1,SEPARATOR=\",\")\r\n\r\n\/REPORT\r\n\r\n\/OUTFILE=e1appdu1.out<\/pre>\n
Output:<\/p>\n
v1s\r\nv1s\r\nv1s\r\nv2s\r\nv1s\r\nv1s\r\nv1s\r\nv1s\r\nv10s\r\nv1s\r\nv1s\r\nv1s\r\nv1s\r\nv1s\r\nv1s\r\nv20s\r\nv1s\r\nv1s\r\nv1s\r\nv1s\r\nv1s\r\nv1s\r\nv1s\r\nv1s<\/pre>\n
Example 8: NG application protocol for 5G -> Simple report of selected PDU to text file<\/b><\/h6>\n
This script (ngap.scl) references a specific PDU that is used to interpret the data message with, from an ASN.1 schema that contains multiple PDUs. The field layout can be generated by ASN1_2DDF. The script will report the data to a text file.<\/p>\n
Script:<\/p>\n
\/INFILE=\"ngap.asn,PER,NGAP-PDU;NGAP-PDU_HandoverRequest.per\"\r\n\/PROCESS=ASN1\r\n    \/FIELD=(initiatingMessage_procedureCode, POSITION=1, SEPARATOR=\",\")\r\n    \/FIELD=(initiatingMessage_criticality, POSITION=2, SEPARATOR=\",\")\r\n    \/FIELD=(initiatingMessage_value, POSITION=3, SEPARATOR=\",\")\r\n    \/FIELD=(successfulOutcome_procedureCode, POSITION=4, SEPARATOR=\",\")\r\n    \/FIELD=(successfulOutcome_criticality, POSITION=5, SEPARATOR=\",\")\r\n    \/FIELD=(successfulOutcome_value, POSITION=6, SEPARATOR=\",\")\r\n    \/FIELD=(unsuccessfulOutcome_procedureCode, POSITION=7, SEPARATOR=\",\")\r\n    \/FIELD=(unsuccessfulOutcome_criticality, POSITION=8, SEPARATOR=\",\")\r\n    \/FIELD=(unsuccessfulOutcome_value, POSITION=9, SEPARATOR=\",\")\r\n\r\n\/REPORT\r\n\r\n\/OUTFILE=ngapPDU.out<\/pre>\n
Output:<\/p>\n
12,reject,000006000A000320303900550003401A85001D000100000F40018A0069000F00AABBCC1001234500ABCDEF123456006500020112,,,,,,<\/pre>\n
For additional examples, check out the ASN.1 sample repository<\/a>.<\/p>\n
Wrap Up<\/b><\/h6>\n
An array of examples have been demonstrated in this article to give an idea of some basic ways to process or generate ASN.1 encoded data. In the following article, the fourth\u00a0 of the five part series, support for the ASN.1 process in IRI Workbench will be discussed and demonstrated.<\/p>\n
Article Links: Other articles in the series:<\/p>\n