{"id":5449,"date":"2014-07-03T18:42:35","date_gmt":"2014-07-03T22:42:35","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=5449"},"modified":"2018-02-26T13:38:41","modified_gmt":"2018-02-26T18:38:41","slug":"clf-elf-web-log-formats","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/","title":{"rendered":"CLF and ELF Web Log Formats"},"content":{"rendered":"<p><em>This article is first in a 3-part series on CLF and ELF web log data, where we introduce these file formats. The next article covers\u00a0IRI solutions for <a title=\"CLF and ELF Web Log Data Processing Blog\" href=\"http:\/\/www.iri.com\/blog\/data-transformation2\/clf-elf-web-log-data-processing\/\" target=\"_blank\" rel=\"noopener\">processing web log data<\/a>, and the last\u00a0demonstrates\u00a0<a title=\"How to Mask Data in Web Logs Blog\" href=\"http:\/\/www.iri.com\/blog\/data-protection\/web-log-data-masking\/\" target=\"_blank\" rel=\"noopener\">web log data masking<\/a>\u00a0to protect visitor identities and destinations.<\/em><\/p>\n<p>The\u00a0<strong>NCSA Common Log Format<\/strong> (CLF) is a standardized text file format used by web servers when generating server log files. The format is standardized so that analytic programs can more conveniently make use of the information contained within them, though\u00a0other proprietary log formats exist.<\/p>\n<p>CLF logs are in a fixed (non-customizable) ASCII format, and record basic information about user requests. For example, a CLF record might contain:<\/p>\n<pre>125.0.0.1 user - identifier sjones [10\/Oct\/2011:13:55:36 -0700] \"GET \/examp_alt.png HTTP\/1.0\" 200 10801<\/pre>\n<p>where:<\/p>\n<ul>\n<li>A &#8220;-&#8221; in a field indicates missing data<\/li>\n<li>125.1.0.1 is the IP address of the client (remote host) which made the request to the server<\/li>\n<li>user-identifier is the identity of the client<\/li>\n<li>sjones is the userid of the person requesting the document<\/li>\n<li>[10\/Oct\/2011:13:55:36 -0700] is the date, time, and time zone when the server finished processing the request<\/li>\n<li>&#8220;GET \/examp_alt.png HTTP\/1.0&#8221; is the request line from the client; the method GET, \/examp_alt.png is the resource requested; and HTTP\/1.0 is the HTTP protocol<\/li>\n<li>200 is the HTTP status code returned to the client<\/li>\n<li>10801 is the size of the object returned to the client, measured in bytes<\/li>\n<\/ul>\n<p><b>W3C Extended Log Format <\/b>(ELF) format is a customizable ASCII format, with a variety of different fields, that is used by web servers when generating log files. ELF files provide more information and flexibility than CLF files.<\/p>\n<p>With ELF, you can include fields important to you, while limiting log file sizes by omitting unwanted fields. In addition, note that fields are separated by spaces, and that time is recorded as UTC (Greenwich Mean Time). For example, an ELF record might contain:<\/p>\n<pre>2010-05-02 15:42:15 - 40.89.255.10 \u00a034.14.255.10 80 GET \/default.htm 200 - HTTP\/1.0 Mozilla\/4.0 \u00a0(compatible: MSIE+5.5+Windows+2000+Server)<\/pre>\n<p>In this case, the format is:<\/p>\n<pre>date, time,c-ip, cs-username(-), s-ip, sport, method, cs-uri-stem, status, csUserAgent<\/pre>\n<p>Each line can contain either a directive or an entry. Entries consist of a sequence of fields relating to a single HTTP transaction. Fields are separated by spaces. A &#8220;-&#8221; in a field indicates missing data.<\/p>\n<p>Directives record information about the logging process itself. Lines beginning with the # character are directives.<\/p>\n<p>These directives are defined as follows:<\/p>\n<ul>\n<li>Version &#8211; rendition of the extended log file format used<\/li>\n<li>Fields &#8211; space in which data is recorded in the log<\/li>\n<li>Software &#8211; program that generated the log<\/li>\n<li>Start Date &#8211; date\/time when the log began<\/li>\n<li>End Date &#8211; date\/time when the log was finished<\/li>\n<li>Date &#8211; date\/time when the entry was added<\/li>\n<li>Remark &#8211; specific comment information (data recorded in this field should be ignored by analysis tools)<\/li>\n<\/ul>\n<p>See the next article on\u00a0<a title=\"CLF and ELF Web Log Data Processing Blog\" href=\"http:\/\/www.iri.com\/blog\/data-transformation2\/clf-elf-web-log-data-processing\/\" target=\"_blank\" rel=\"noopener\"><em>CLF and ELF Web\u00a0Log Data Processing<\/em><\/a><strong>\u00a0<\/strong>that introduces IRI solutions for transforming, migrating, protecting, reporting from, and prototyping huge web logs.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This article is first in a 3-part series on CLF and ELF web log data, where we introduce these file formats. The next article covers\u00a0IRI solutions for processing web log data, and the last\u00a0demonstrates\u00a0web log data masking\u00a0to protect visitor identities and destinations. The\u00a0NCSA Common Log Format (CLF) is a standardized text file format used by<\/p>\n<div><a class=\"btn-filled btn\" href=\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/\" title=\"CLF and ELF Web Log Formats\">Read More<\/a><\/div>\n","protected":false},"author":53,"featured_media":12037,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[31],"tags":[483,487,484,486,485],"class_list":["post-5449","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-migration","tag-clf-web-log","tag-common-log-format","tag-elf-web-log","tag-extended-log-format","tag-web-log-formats"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v23.4 (Yoast SEO v23.4) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>CLF and ELF Web Log Formats - IRI<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"CLF and ELF Web Log Formats\" \/>\n<meta property=\"og:description\" content=\"This article is first in a 3-part series on CLF and ELF web log data, where we introduce these file formats. The next article covers\u00a0IRI solutions for processing web log data, and the last\u00a0demonstrates\u00a0web log data masking\u00a0to protect visitor identities and destinations. The\u00a0NCSA Common Log Format (CLF) is a standardized text file format used byRead More\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/\" \/>\n<meta property=\"og:site_name\" content=\"IRI\" \/>\n<meta property=\"article:published_time\" content=\"2014-07-03T22:42:35+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-02-26T18:38:41+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/07\/clf-elf-data.png\" \/>\n\t<meta property=\"og:image:width\" content=\"706\" \/>\n\t<meta property=\"og:image:height\" content=\"500\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Chaitali Mitra\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Chaitali Mitra\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/\"},\"author\":{\"name\":\"Chaitali Mitra\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/9bae14a309616863b027c2d56f532caf\"},\"headline\":\"CLF and ELF Web Log Formats\",\"datePublished\":\"2014-07-03T22:42:35+00:00\",\"dateModified\":\"2018-02-26T18:38:41+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/\"},\"wordCount\":473,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/07\/clf-elf-data.png\",\"keywords\":[\"clf web log\",\"common log format\",\"elf web log\",\"extended log format\",\"web log formats\"],\"articleSection\":[\"Data Migration\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/\",\"url\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/\",\"name\":\"CLF and ELF Web Log Formats - IRI\",\"isPartOf\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/07\/clf-elf-data.png\",\"datePublished\":\"2014-07-03T22:42:35+00:00\",\"dateModified\":\"2018-02-26T18:38:41+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/#primaryimage\",\"url\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/07\/clf-elf-data.png\",\"contentUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/07\/clf-elf-data.png\",\"width\":706,\"height\":500,\"caption\":\"CLF ELF Data\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.iri.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"CLF and ELF Web Log Formats\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.iri.com\/blog\/#website\",\"url\":\"https:\/\/www.iri.com\/blog\/\",\"name\":\"IRI\",\"description\":\"Total Data Management Blog\",\"publisher\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.iri.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\",\"name\":\"IRI\",\"url\":\"https:\/\/www.iri.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"contentUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"width\":750,\"height\":206,\"caption\":\"IRI\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/9bae14a309616863b027c2d56f532caf\",\"name\":\"Chaitali Mitra\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/95a11f3d0b709c00df3262bab0152f3a?s=96&d=blank&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/95a11f3d0b709c00df3262bab0152f3a?s=96&d=blank&r=g\",\"caption\":\"Chaitali Mitra\"},\"sameAs\":[\"http:\/\/www.iri.com\"],\"url\":\"https:\/\/www.iri.com\/blog\/author\/chaitalim\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"CLF and ELF Web Log Formats - IRI","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/","og_locale":"en_US","og_type":"article","og_title":"CLF and ELF Web Log Formats","og_description":"This article is first in a 3-part series on CLF and ELF web log data, where we introduce these file formats. The next article covers\u00a0IRI solutions for processing web log data, and the last\u00a0demonstrates\u00a0web log data masking\u00a0to protect visitor identities and destinations. The\u00a0NCSA Common Log Format (CLF) is a standardized text file format used byRead More","og_url":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/","og_site_name":"IRI","article_published_time":"2014-07-03T22:42:35+00:00","article_modified_time":"2018-02-26T18:38:41+00:00","og_image":[{"width":706,"height":500,"url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/07\/clf-elf-data.png","type":"image\/png"}],"author":"Chaitali Mitra","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Chaitali Mitra","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/#article","isPartOf":{"@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/"},"author":{"name":"Chaitali Mitra","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/9bae14a309616863b027c2d56f532caf"},"headline":"CLF and ELF Web Log Formats","datePublished":"2014-07-03T22:42:35+00:00","dateModified":"2018-02-26T18:38:41+00:00","mainEntityOfPage":{"@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/"},"wordCount":473,"commentCount":0,"publisher":{"@id":"https:\/\/www.iri.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/#primaryimage"},"thumbnailUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/07\/clf-elf-data.png","keywords":["clf web log","common log format","elf web log","extended log format","web log formats"],"articleSection":["Data Migration"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/","url":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/","name":"CLF and ELF Web Log Formats - IRI","isPartOf":{"@id":"https:\/\/www.iri.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/#primaryimage"},"image":{"@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/#primaryimage"},"thumbnailUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/07\/clf-elf-data.png","datePublished":"2014-07-03T22:42:35+00:00","dateModified":"2018-02-26T18:38:41+00:00","breadcrumb":{"@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/#primaryimage","url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/07\/clf-elf-data.png","contentUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/07\/clf-elf-data.png","width":706,"height":500,"caption":"CLF ELF Data"},{"@type":"BreadcrumbList","@id":"https:\/\/www.iri.com\/blog\/migration\/data-migration\/clf-elf-web-log-formats\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.iri.com\/blog\/"},{"@type":"ListItem","position":2,"name":"CLF and ELF Web Log Formats"}]},{"@type":"WebSite","@id":"https:\/\/www.iri.com\/blog\/#website","url":"https:\/\/www.iri.com\/blog\/","name":"IRI","description":"Total Data Management Blog","publisher":{"@id":"https:\/\/www.iri.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.iri.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.iri.com\/blog\/#organization","name":"IRI","url":"https:\/\/www.iri.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","contentUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","width":750,"height":206,"caption":"IRI"},"image":{"@id":"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/9bae14a309616863b027c2d56f532caf","name":"Chaitali Mitra","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/95a11f3d0b709c00df3262bab0152f3a?s=96&d=blank&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/95a11f3d0b709c00df3262bab0152f3a?s=96&d=blank&r=g","caption":"Chaitali Mitra"},"sameAs":["http:\/\/www.iri.com"],"url":"https:\/\/www.iri.com\/blog\/author\/chaitalim\/"}]}},"jetpack_featured_media_url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2014\/07\/clf-elf-data.png","_links":{"self":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/5449"}],"collection":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/users\/53"}],"replies":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/comments?post=5449"}],"version-history":[{"count":18,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/5449\/revisions"}],"predecessor-version":[{"id":12038,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/5449\/revisions\/12038"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/media\/12037"}],"wp:attachment":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/media?parent=5449"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/categories?post=5449"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/tags?post=5449"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}