The sort included with each Unix-based operating system is a standard command line program that prints lines of input or specified input files in\u00a0the specified\u00a0sorted order.<\/p>\n
The bin\/sort interface provided with IRI CoSort<\/a> is a drop-in replacement for the sort utility<\/a>\u00a0usually found in the \/bin directory of most Unix\/Linux operating systems, but uses IRI’s proprietary sort and resource-optimization algorithms to perform system sort jobs much faster.<\/p>\n To function as a drop-in, it is recommended that your system administrator:<\/p>\n 1) Renames \/bin\/sort to \/bin\/sort.orig (for safe keeping). Note that the Windows replacement is named unixsort.exe upon installation of CoSort for Windows.<\/p>\n 2) Creates a link from the default system \/bin\/sort to the CoSort sort.<\/p>\n Users will then have the sort speed of CoSort without changing any of their own references (within batch scripts and programs).<\/p>\n Given this 10-record input file:<\/p>\n When we run the following from the command line:<\/p>\n the output file contains:<\/p>\n The final field (state) was sorted (as indicated by +4 with a “|” separator), and:<\/p>\n The reason these particular records were kept is that they appeared earlier than the other Florida and South Dakota records in the original input file.<\/p>\n We recommend that CoSort users eventually convert their bin\/sort commands to equivalent syntax in “SortCL<\/a>” job scripts. SortCL is the primary user interface in the IRI CoSort package and default data transformation, cleansing, masking, and reporting engine in the IRI Voracity<\/a> platform. SortCL uses a simple, and familiar (to JCL and SQL users) 4GL to define data layouts and manipulations.<\/p>\n Because SortCL uses the same underlying sort engine as CoSort’s bin\/sort replacement does, sort performance in SortCL will be just as superior to the system sort. SortCL will provide the added advantages of:<\/p>\n The bin\/sort example in the above, as performed on the date source patient_info.in:<\/p>\n would be expressed as follows in SortCL (in its simplest form):<\/p>\n Running the above will produce the same output shown in the above bin\/sort example.<\/p>\n Note the following functional equivalents between the above sortcl script and bin\/sort:<\/p>\n In bin\/sort this was done with -t “|” +4\u00a0 where 0 was the first field in bin\/sort and 1 was the first field for SortCL counting purposes.<\/p>\n ** This direct specification of key parameters with position and separator options (and optionally SIZE) can be done in lieu of simply specifying a field name, if metadata were provided in the input section (via the \/SPEC command, which was commented out above). Note that IRI Workbench features a Metadata Discovery wizard to automate the process of ascertaining metadata where possible.<\/p>\n <\/p>\n Add Functions to Your Sort Jobs via SortCL<\/strong><\/p>\n Once your bin\/sort job is converted to SortCL, as in the above example, you can now enjoy the full benefits of that interface, described at:<\/p>\n http:\/\/www.iri.com\/products\/cosort\/sortcl\/function-matrix<\/a><\/p>\n To apply format-preserving encryption to the email field, for example, you could simply modify and expand the above job script as follows:<\/p>\n This would produce the following results:<\/p>\n We still see the same sort that was performed above, but also the email field was encrypted with AES 256-bit format-preserving for the alpha-numeric characters (the @ and . were not encrypted, and the field width was preserved).<\/p>\n Multiple field protection (data masking) options are offered in CoSort, Voracity, or\u00a0IRI FieldShield<\/a>, as they all support the same metadata and job scripting syntax of SortCL.<\/p>\n","protected":false},"excerpt":{"rendered":" The sort included with each Unix-based operating system is a standard command line program that prints lines of input or specified input files in\u00a0the specified\u00a0sorted order. The bin\/sort interface provided with IRI CoSort is a drop-in replacement for the sort utility\u00a0usually found in the \/bin directory of most Unix\/Linux operating systems, but uses IRI’s proprietary<\/p>\nAdams, John|adams@gmail.com|646-834-9956|Melbourne|Florida\r\nMonroe, James|Monroe@James.com|433-758-2783|Rapid City|South Dakota\r\nJackson, Andrew|Jackson@Andrew.com|145-894-4328|Long Island|New York\r\nVan, Martin|Van@Martin.com|654-763-7612|Tulsa|Oklahom\r\nHarrison, Henry|Harrison@Henry.com|765-978-2457|Aberdeen|South Dakota\r\nTyler, John|Tyler@John.com|554-674-1289|Juneau|Alaska\r\nPolk, James|Polk@James.com|553-563-2399|Baton Rouge|Louisiana\r\nPierce, Franklin|Pierce@Franklin.com|344-891-3289|Salem|Oregon\r\nCleveland, Henry|Cleveland@henry.com|345-548-3282|Kalahari|Delaware\r\nChalse, Logan|Chalse@Logan.com|321-889-4633|Melbourne|Florida\r\n<\/pre>\n
sort -s -u -t '|' +4r patient_info.in -o patient_info.out<\/pre>\n
Monroe, James|Monroe@James.com|433-758-2783|Rapid City|South Dakota\r\nPierce, Franklin|Pierce@Franklin.com|344-891-3289|Salem|Oregon\r\nVan, Martin|Van@Martin.com|654-763-7612|Tulsa|Oklahoma\r\nJackson, Andrew|Jackson@Andrew.com|145-894-4328|Long Island|New York\r\nPolk, James|Polk@James.com|553-563-2399|Baton Rouge|Louisiana\r\nAdams, John|adams@gmail.com|646-834-9956|Melbourne|Florida\r\nCleveland, Henry|Cleveland@henry.com|345-548-3282|Kalahari|Delaware\r\nTyler, John|Tyler@John.com|554-674-1289|Juneau|Alaska\r\n<\/pre>\n
\n
\nUsing SortCL Instead of bin\/sort<\/strong><\/p>\n\n
sort -s -u -t '|' +4r patient_info.in -o patient_info.out<\/pre>\n
\/INFILE=patient_info.in\r\n#\/SPEC=C:DDF_repositorypatient_info.ddf\u00a0 # invokes the metadata for the source\r\n# (commented out here)\r\n\/KEY=(POS=5,SEP=\"|\",DESC)\u00a0\u00a0\u00a0\u00a0 # direct key specification, without requiring metadata\r\n\/STABLE\r\n\/NODUPLICATES\r\n\/OUTFILE=patient_info.out\r\n<\/pre>\n
\n
\n
\/INFILE=patient_info.in\r\n \/SPEC=C:DDF_repositorypatient_info.ddf\u00a0 # invokes the metadata for the source\r\n\/KEY=(state,desc)\u00a0 # descending sort on the state field\r\n \/STABLE\r\n \/NODUPLICATES\r\n\/OUTFILE=encrypted_email.out\r\n \/PROCESS = RECORD\r\n \/FIELD=(NAME, POS=1, SEP=\"|\")\r\n \/FIELD=(ENC_FP_EMAIL=enc_fp_aes256_alphanum(EMAIL, \"12345\"), POS=2, SEP=\"|\")\r\n # applies the encryption routine to the email field with the passphrase \"12345\"\r\n \/FIELD=(PHONE, POS=3, SEP=\"|\")\r\n \/FIELD=(CITY, POS=4, SEP=\"|\")\r\n \/FIELD=(STATE, POS=5, SEP=\"|\")\r\n<\/pre>\n
Taylor, Zachary|Hwswie@Gqfimdu.dbs|423-560-3289|Charleston|West Virginia\r\nMonroe, James|Tsafdh@Todhu.wqm|433-758-2783|Rapid City|South Dakota\r\nPierce, Franklin|Ltzglh@Mbzogtyv.szo|344-891-3289|Salem|Oregon\r\nVan, Martin|Leb@Iggeqh.vru|654-763-7612|Tulsa|Oklahoma\r\nGrant, Ulysses|Poqcn@Aaoqeat.thg|348-855-3478|Bismarck|North Dakota\r\nJackson, Andrew|Mexdkjh@Iuaddn.xwj|145-894-4328|Long Island|New York\r\nBuchanan, James|Frnulwam@Fzriv.ulf|432-453-3472|Trenton|New Jersey\r\nAdams Quincy|Hbcbe@Rklkhge.pnk|983-245-2954|Lyon|Nevada\r\nFillmore, Millard|Guyqyvll@Qcusvio.tfx|205-773-2347|Lincoln|Nebraska\r\nMadison, James|Tuqzrpq@Tonzp.nvw|563-435-7821|Biloxi|Mississippi\r\nJohnson, Andrew|Mwvptjr@Izjnmd.slp|984-587-2855|Saint Paul|Minnesota\r\nPolk, James|Hvbv@Dsedx.eqo|553-563-2399|Baton Rouge|Louisiana\r\nJefferson, Thomas|Hwyjqjpgn@Frwqqe.pvo|321-890-8293|Goodland|Kansas\r\nAdams, John|cqlai@zdnla.yxk|646-834-9956|Melbourne|Florida\r\nCleveland, Henry|Dmcykglas@wvzzl.wlf|345-548-3282|Kalahari|Delaware\r\nHayes, Rutherford|Hmjmb@Vovhfrzzio.vdv|646-344-1234|Sacramento|California\r\nTyler, John|Ftutv@Aypw.tss|554-674-1289|Juneau|Alaska\r\n<\/pre>\n