IRI Blog Articles

Diving Deeper into Data Management

 

 

Post image for CDR (TAP3) Processing Example

CDR (TAP3) Processing Example

by Paul Friedland

Abstract: Previously, we introduced the unique ability of IRI CoSort to bypass mediation and directly process native ASN.1-based call detail records (CDRs) in SortCL data integration, data migration, data masking, and reporting programs. Today we are providing an example of how CoSort natively recognizes and processes data in the TAP 3.11 standard. Contact IRI if your CDRs are in a different format.

The Abstract Syntax Notation One (ASN.1) standard defines the “rules and structures for representing, encoding, transmitting, and decoding data in telecommunications and computer networking.” The TAP3 CDR decoding algorithm available to IRI CoSort users leverages the Basic Encoding Rules (BER) of that  standard.

In the CoSort SortCL program, the mediation and processing of ASN.1-based CDRs occur either simultaneously, or in tandem, in the same pass. Raw CDRs can be input from a database table, file, or pipe. The records are flattened and remapped as the values are decoded into readable ASCII characters.

In that form, records continuously move through SortCL to be processed, just as records would from other input source, and with any other source. SortCL users can apply a wide range of data management functions to multiple data sources, including data:

Because you can process multiple input streams at the same time, you can merge (mash-up) your CDR data with other CDR and data sources in the same pass. And because you can produce multiple output streams, you can also selectively filter, summarize, format, and provide data for different users at the same time.

Consider your CDR analytic, billing, archival, compliance, and other applications, and the benefits of a single-pass, single-product environment for designing and running them. See the basic example below to edify your understanding of the process:

Input

Raw CDR data is not human readable and cannot be processed by standard data integration or processing tools:

ScreenHunter_02 Aug. 18 17.26


First few records of a TAP3.11 input file with only 27 fields typically used in call detail record (CDR) analytics.

The hex representation of the first part of the sample file shown above is:

HexaDecimal +0 +1 +2 +3 +4 +5 +6 +7 +8 +9+10+11+12+13+14+15 GGNTAP3-MO-GPRSCALL.dat 1
0000(00000) 61 81 a3 63 81 a0 69 81 9d 7f 81 13 7e 7f 83 2b a..c..i....~.+
0010(00016) 18 7f 81 47 14 5f 81 01 07 62 99 92 01 91 21 10 ..G._...b....!.
0020(00032) 5f 81 18 05 62 99 99 99 10 5f 81 35 01 31 7f 59 _...b...._.5.1Y
0030(00048) 30 5f 83 17 05 62 55 55 55 10 5f 82 17 09 36 32 0_...bUUU._...62
0040(00064) 35 35 35 35 35 35 31 5f 2a 02 41 31 5f 2e 03 6a 5555551_*.A1_..j
0050(00080) 6b 74 5f 83 23 0b 31 31 31 31 31 31 31 31 31 31 kt_.#.1111111111
0060(00096) 35 5f 5a 02 35 37 7f 2c 14 50 0d 32 30 31 33 3
0070(00112) 38 30 31 31 30 30 31 31 5f 81 68 01 00
0080(00128) 01 20 5f 81 48 01 31 5f 3a 01
0090(00144) 81 1c 13 5f 81 38 0
00a0(00160) 01 21 5f
- - - - - - - - -

The SortCL program in IRI CoSort reads and flattens the CDR input. It also leverages a reusable field layout data definition file (DDF) for immediate, or downstream, data processing applications (see script below). In our example, the metadata repository, TAP3-27fields.ddf, contains:

   /FIELD_PREDICATE=(separator = '|')
      /FIELD=(callType)
      /FIELD=(imsi)
      /FIELD=(msisdn)
      /FIELD=(rapFileSequenceNumber)
      /FIELD=(calledNumber)
      /FIELD=(dialledDigits)
      /FIELD=(calledPlace)
      /FIELD=(calledRegion)
      /FIELD=(sMSDestinationNumber)
      /FIELD=(destinationNetwork)
      /FIELD=(networkAccessIdentifier)
      /FIELD=(accessPointNameNI)
      /FIELD=(accessPointNameOI)
      /FIELD=(callEventStartTimeStamp)
      /FIELD=(totalCallEventDuration)
      /FIELD=(simToolkitIndicator)
      /FIELD=(causeForTerm)
      /FIELD=(partialTypeIndicator)
      /FIELD=(chargingId)
      /FIELD=(recEntityCode)
      /FIELD=(callReference)
      /FIELD=(locationArea)
      /FIELD=(cellId)
      /FIELD=(servingNetwork)
      /FIELD=(dataVolumeIncoming)
      /FIELD=(dataVolumeOutgoing)
      /FIELD=(chargedItem)


Process

Given the input data and its layout repository above, you can write a simple analytic application in SortCL like the one below that reads the raw CDR file and uses the centrally available metadata (DDF). For example, a job script might contain these transformation and mapping specifications:

/INFILE= GGNTAP3-MO-GPRSCALL.dat   # ASN.1 BER encoded CDRs
     /PROCESS=TAP3
     /SPEC=TAP3-27fields.ddf 
     /OMIT WHERE callEventStartTimeStamp NE "2013080101021" 
     /INCLUDE WHERE servingNetwork EQ "wapoperator"
/SORT
     /KEY=totalCallEventDuration
/OUTFILE=GGNTAP3-MO-GPRSCALL.data.out
    /PROCESS=RECORD
    /HEADREC="Timestamp Duration Network V_In V_Out Item\n"
# # # selected output fields # # #
    /FIELD_PREDICATE=(separator = "\t")   # tab delimiter
        /FIELD=(callEventStartTimeStamp)
        /FIELD=(totalCallEventDuration)
        /FIELD=(servingNetwork)
        /FIELD=(dataVolumeIncoming)
        /FIELD=(dataVolumeOutgoing)
        /FIELD=(chargedItem)
    /OUTCOLLECT=10        # limit to 10 output records


Output

In this simple case, the flat output file result would contain:

Timestamp       Duration    Network      V_In   V_Out  Item
2013080101021   3232        wapoperator  17776  12726  48
2013080101021   3264        wapoperator  17952  12852  48
2013080101021   3296        wapoperator  18128  12978  48
2013080101021   3328        wapoperator  18304  13104  48
2013080101021   3360        wapoperator  18480  13230  48
2013080101021   3392        wapoperator  18656  13356  48
2013080101021   3424        wapoperator  18832  13482  48
2013080101021   3456        wapoperator  19008  13608  48
2013080101021   3488        wapoperator  19184  13734  48
2013080101021   3520        wapoperator  19360  13850  48

Again, many other output targets and formats can be produced simultaneously, as well as formatted reports, hand-offs to DBs or BI tools, or even direct feeds to BIRT via ODA for free, advanced visualizations in the same GUI.

Billing reports with aggregations and cross-calculations can also be designed directly in SortCL scripts, and field-level data masking can be applied to protect fields containing PII. Contact IRI with your specific requirements.

Print Friendly

{ 1 comment… read it below or add one }

Prasandha March 15, 2015 at 10:26 pm

Hi David,
This capability for processing the raw CDR is very interesting. Could you please show the other sample of ASN.1 file processing (other than TAP3 file)? Currently I’m evaluating the ETL tools which be able to process the ASN.1 files from Telco nodes.

Thank you.
TPrasandha

Reply

Cancel reply

Leave a Comment

Previous post:

Next post: