{"id":5287,"date":"2014-06-17T15:39:10","date_gmt":"2014-06-17T19:39:10","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=5287"},"modified":"2017-11-06T14:00:17","modified_gmt":"2017-11-06T19:00:17","slug":"title-utf8-good-day-script","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/","title":{"rendered":"Say Hello to UTF-8 Support in CoSort"},"content":{"rendered":"

Thanks to the Internet, the IT world is now a smaller place, too. Information conveyed in\u00a0different languages must be accurately represented in multiple data sources, email clients, and applications.<\/p>\n

In the past, unaccented English letters were represented by ASCII characters in the decimal range 000-127. Then when other groups and countries needed to represent characters that could not be designated by those below 128, their characters were\u00a0assigned into\u00a0the decimal range 128-255. The problem was that each new group had designations for their own characters within the same new range. To address the conflict, different ways of representing characters in the same range\u00a0became\u00a0known as code pages. The\u00a0user had\u00a0to designate which code page\u00a0to use on\u00a0his computer. Greek and Turkish characters are defined in different code pages; therefore, they could not be represented at the same time. Representing complex characters used in\u00a0Asia was\u00a0even more problematic.<\/p>\n

A new system was needed in which any character could be uniquely represented. Eventually a set of unifying encoding systems, or ‘Unicode’, was invented to achieve this. Unicode is a standard for the computer industry that defines the handling and encoding of text for most of the world’s writing systems. The most basic, UTF-8, stands for Unicode Transformation Format 8-bit. UTF-8 is a variable-width encoding system that represents every character in the Unicode character set. Each Unicode character can be represented by one to four 8-bit bytes. Each byte is known as a code point.<\/p>\n

Encodings that represent the ASCII characters 0x00 to 0x7f (decimal 000-127) are exactly the same in UTF-8. When a character is a multi-byte character, the first byte is one of the permitted start bytes, and is followed by continuation bytes. The start byte that is used also indicates the length in bytes of the multi-byte character. The ASCII encodings cannot be used for start bytes or continuation bytes in multi-byte encodings. It is because of this system that every character can be uniquely represented and self-synchronizing. When an encoding is corrupted, it is easy to resynchronize, since the beginning of the next character boundary can always be located.<\/p>\n

The SortCL<\/a> program in IRI CoSort<\/a> — along with spin-off tools like NextForm for data migration and FieldShield for data masking — will process UTF-8 data. Fields in UTF-8 data sources can be re-mapped to different output formats while sorting in encoding order. Consider the following SortCL job script example:<\/p>\n

\n
\/INFILE=GoodDayDelim.utf8\r\n    \/FIELD=(phrase,POSITION=1, SEPARATOR='|', TYPE=UTF8)\r\n    \/FIELD=(language,POSITION=2, SEPARATOR='|' TYPE=UTF8)\r\n\/SORT\r\n    \/KEY=phrase\r\n\/OUTFILE=GoodDayDelim.out\r\n    \/FIELD=(phrase,POSITION=1, CHARS=25, TYPE=UTF8)\r\n    \/FIELD=(language,POSITION=27, CHARS=19, TYPE=UTF8)]\r\n<\/pre>\n<\/blockquote>\n
The input file (GoodDayDelim.utf8) has two pipe-delimited fields for each record. The two fields have each been given the data type of\u00a0 UTF-8. The field phrase<\/em> has the equivalent of the English “Good Day” in several languages. The field language<\/em> gives the actual language in ASCII English for the phrase<\/em> field in that record. The records are sorted in the encoding order for the field phrase<\/em> and then written to the output file (GoodDayDelim.out). The fields are mapped to fixed positions in the output.<\/p>\n
In a record that has only ASCII characters, SortCL would use a “SIZE=number of bytes<\/em>” to indicate the byte size for each field. Because the fields have a data type of UTF-8 and characters can have more than one byte per character, the size of the field is indicated by using “CHARS=number of characters<\/em>” per field, where each character can be defined by more than one byte.<\/p>\n
So if you have the following delimited input files:<\/p>\n
\"input_utf8\"<\/a><\/p>\n
You will get this output file:<\/p>\n
\"output_utf8\"<\/a><\/p>\n
You should notice that the writing for each of the languages is displayed at the same time. This was previously impossible because they would each need to have different code pages active for the display to happen. UTF-8 is becoming the standard for the Internet, and almost all email messages are now transmitted in\u00a0UTF-8 format.<\/p>\n","protected":false},"excerpt":{"rendered":"
Thanks to the Internet, the IT world is now a smaller place, too. Information conveyed in\u00a0different languages must be accurately represented in multiple data sources, email clients, and applications. In the past, unaccented English letters were represented by ASCII characters in the decimal range 000-127. Then when other groups and countries needed to represent characters<\/p>\n
Read More<\/a><\/div>\n","protected":false},"author":10,"featured_media":5289,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[31,1],"tags":[458,460,457,456,459,454,455,453],"class_list":["post-5287","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-migration","category-data-transformation2","tag-ascii-characters","tag-character-set-support","tag-eliminate-code-pages","tag-multiple-languages-at-the-same-time","tag-unicode-transformation-format","tag-utf-8-introduction","tag-utf-8-support","tag-utf-8-support-in-cosort"],"acf":[],"yoast_head":"\nSay Hello to UTF-8 Support in CoSort - IRI<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Say Hello to UTF-8 Support in CoSort\" \/>\n<meta property=\"og:description\" content=\"Thanks to the Internet, the IT world is now a smaller place, too. Information conveyed in\u00a0different languages must be accurately represented in multiple data sources, email clients, and applications. In the past, unaccented English letters were represented by ASCII characters in the decimal range 000-127. Then when other groups and countries needed to represent charactersRead More\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/\" \/>\n<meta property=\"og:site_name\" content=\"IRI\" \/>\n<meta property=\"article:published_time\" content=\"2014-06-17T19:39:10+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-11-06T19:00:17+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/06\/output_utf8.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"362\" \/>\n\t<meta property=\"og:image:height\" content=\"300\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Susan Gegner\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Susan Gegner\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/\"},\"author\":{\"name\":\"Susan Gegner\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/87be5da567628ab9396ca81170f36d63\"},\"headline\":\"Say Hello to UTF-8 Support in CoSort\",\"datePublished\":\"2014-06-17T19:39:10+00:00\",\"dateModified\":\"2017-11-06T19:00:17+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/\"},\"wordCount\":640,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/06\/output_utf8.jpg\",\"keywords\":[\"ascii characters\",\"character set support\",\"eliminate code pages\",\"multiple languages at the same time\",\"unicode transformation format\",\"utf-8 introduction\",\"utf-8 support\",\"utf-8 support in CoSort\"],\"articleSection\":[\"Data Migration\",\"Data Transformation\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/\",\"url\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/\",\"name\":\"Say Hello to UTF-8 Support in CoSort - IRI\",\"isPartOf\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/06\/output_utf8.jpg\",\"datePublished\":\"2014-06-17T19:39:10+00:00\",\"dateModified\":\"2017-11-06T19:00:17+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/#primaryimage\",\"url\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/06\/output_utf8.jpg\",\"contentUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/06\/output_utf8.jpg\",\"width\":362,\"height\":300},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.iri.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Say Hello to UTF-8 Support in CoSort\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.iri.com\/blog\/#website\",\"url\":\"https:\/\/www.iri.com\/blog\/\",\"name\":\"IRI\",\"description\":\"Total Data Management Blog\",\"publisher\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.iri.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\",\"name\":\"IRI\",\"url\":\"https:\/\/www.iri.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"contentUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"width\":750,\"height\":206,\"caption\":\"IRI\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/87be5da567628ab9396ca81170f36d63\",\"name\":\"Susan Gegner\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/2b1ca5592a65d44483351292cf1ae00a?s=96&d=blank&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/2b1ca5592a65d44483351292cf1ae00a?s=96&d=blank&r=g\",\"caption\":\"Susan Gegner\"},\"url\":\"https:\/\/www.iri.com\/blog\/author\/susang\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Say Hello to UTF-8 Support in CoSort - IRI","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/","og_locale":"en_US","og_type":"article","og_title":"Say Hello to UTF-8 Support in CoSort","og_description":"Thanks to the Internet, the IT world is now a smaller place, too. Information conveyed in\u00a0different languages must be accurately represented in multiple data sources, email clients, and applications. In the past, unaccented English letters were represented by ASCII characters in the decimal range 000-127. Then when other groups and countries needed to represent charactersRead More","og_url":"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/","og_site_name":"IRI","article_published_time":"2014-06-17T19:39:10+00:00","article_modified_time":"2017-11-06T19:00:17+00:00","og_image":[{"width":362,"height":300,"url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/06\/output_utf8.jpg","type":"image\/jpeg"}],"author":"Susan Gegner","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Susan Gegner","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/#article","isPartOf":{"@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/"},"author":{"name":"Susan Gegner","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/87be5da567628ab9396ca81170f36d63"},"headline":"Say Hello to UTF-8 Support in CoSort","datePublished":"2014-06-17T19:39:10+00:00","dateModified":"2017-11-06T19:00:17+00:00","mainEntityOfPage":{"@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/"},"wordCount":640,"commentCount":0,"publisher":{"@id":"https:\/\/www.iri.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/#primaryimage"},"thumbnailUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/06\/output_utf8.jpg","keywords":["ascii characters","character set support","eliminate code pages","multiple languages at the same time","unicode transformation format","utf-8 introduction","utf-8 support","utf-8 support in CoSort"],"articleSection":["Data Migration","Data Transformation"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/","url":"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/","name":"Say Hello to UTF-8 Support in CoSort - IRI","isPartOf":{"@id":"https:\/\/www.iri.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/#primaryimage"},"image":{"@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/#primaryimage"},"thumbnailUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/06\/output_utf8.jpg","datePublished":"2014-06-17T19:39:10+00:00","dateModified":"2017-11-06T19:00:17+00:00","breadcrumb":{"@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/#primaryimage","url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/06\/output_utf8.jpg","contentUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/06\/output_utf8.jpg","width":362,"height":300},{"@type":"BreadcrumbList","@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/title-utf8-good-day-script\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.iri.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Say Hello to UTF-8 Support in CoSort"}]},{"@type":"WebSite","@id":"https:\/\/www.iri.com\/blog\/#website","url":"https:\/\/www.iri.com\/blog\/","name":"IRI","description":"Total Data Management Blog","publisher":{"@id":"https:\/\/www.iri.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.iri.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.iri.com\/blog\/#organization","name":"IRI","url":"https:\/\/www.iri.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","contentUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","width":750,"height":206,"caption":"IRI"},"image":{"@id":"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/87be5da567628ab9396ca81170f36d63","name":"Susan Gegner","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/2b1ca5592a65d44483351292cf1ae00a?s=96&d=blank&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/2b1ca5592a65d44483351292cf1ae00a?s=96&d=blank&r=g","caption":"Susan Gegner"},"url":"https:\/\/www.iri.com\/blog\/author\/susang\/"}]}},"jetpack_featured_media_url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/06\/output_utf8.jpg","_links":{"self":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/5287"}],"collection":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/comments?post=5287"}],"version-history":[{"count":19,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/5287\/revisions"}],"predecessor-version":[{"id":11777,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/5287\/revisions\/11777"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/media\/5289"}],"wp:attachment":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/media?parent=5287"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/categories?post=5287"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/tags?post=5287"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}