{"id":10898,"date":"2017-02-16T15:42:49","date_gmt":"2017-02-16T20:42:49","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=10898"},"modified":"2026-02-23T15:24:47","modified_gmt":"2026-02-23T20:24:47","slug":"when-to-use-hadoop","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/","title":{"rendered":"When to Use Hadoop"},"content":{"rendered":"

In the last couple of years since <\/span>this article<\/span><\/a> about the use of IRI CoSort as a parallel data manipulation alternative to Hadoop, IRI released the <\/span>Voracity platform<\/span><\/a> to process big data in either paradigm. With its \u201c<\/span>map once, deploy anywhere<\/span><\/a>\u201d capability, Voracity can run many popular jobs for data transformation, masking, and test data generation in the default CoSort engine, or in MapReduce 2, Spark, Spark Stream, Storm, or Tez without any changes.<\/span><\/p>\n

Voracity is a full-stack data discovery (profiling), integration (ETL), migration, governance (data quality, masking, MDM, EMM), and analytics (reporting or blending) platform built on Eclipse.<\/p>\n

\"IRI<\/a><\/p>\n

The IRI Workbench <\/span><\/i>GUI for Voracity<\/span><\/i><\/a> showing Hadoop Run Configuration options<\/span><\/i><\/p>\n

As seamless as the choice to run in Hadoop now is, the question about when to exercise it remains. The short answer is probably when input volumes approach 10TB, and your data are in HDFS. Here\u2019s a longer answer:<\/span><\/p>\n

Voracity\u2019s CoSort engine (<\/span>SortCL<\/span><\/a>) runs reliably and fast enough below that point on most multi-core Windows, Linux or Unix servers that can hold the input, output and temp files \u2026 and still do whatever else they need to do. CoSort has better algorithms for each task than Java (or a DB) does, and a rare ability to run multiple tasks in the same job and I\/O pass (consolidate). \u00a0<\/span><\/p>\n

Consider the CoSort benchmark <\/span>here<\/span><\/a>, and that there are inherent preparation and scheduling overhead in running jobs in Hadoop. If you can sort 50GB\u00a0in 2 minutes with AIX CoSort, for example, and it takes at least a minute for Oozie to launch the same job in HDFS, stay off the grid for that. <\/span><\/p>\n

Above that input volume, however, Hadoop\u2019s inherent elasticity and redundancy, and the additional power of multiple memory and I\/O backbones, should start to give it a performance edge over CoSort. You will also want to use Hadoop to process data sets that are\u00a0 or will be in HDFS if they are large. Voracity\/CoSort\/SortCL can access and process files or HIVE tables in Hadoop directly through drivers, which works fine (and is very fast) below a terabyte.<\/span><\/p>\n

In other words, using Hadoop just to gain speed is not a good strategy when the data fit comfortably in the file system where it resides; CoSort will manipulate it efficiently on most Unix, Linux and Windows servers. Use Hadoop to process big data that resides in HDFS already.\u00a0<\/span><\/p>\n

If you are going to use Hadoop, the next questions are: how does that work in Voracity, and which Hadoop engine should I choose?<\/span><\/p>\n

We explain how to run Hadoop jobs from Voracity in a generic Apache distribution <\/span>here<\/span><\/a>, and how to connect Voracity to your Cloudera, HortonWorks or MapR distribution through the \u2018VGrid\u2019 gateway <\/span>here<\/span><\/a>. Check out articles <\/span>like this<\/span><\/a> to assess which Hadoop execution option to use. Send your questions and feedback to <\/span>voracity@iri.com<\/span><\/a>.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

In the last couple of years since this article about the use of IRI CoSort as a parallel data manipulation alternative to Hadoop, IRI released the Voracity platform to process big data in either paradigm. With its \u201cmap once, deploy anywhere\u201d capability, Voracity can run many popular jobs for data transformation, masking, and test data<\/p>\n

Read More<\/a><\/div>\n","protected":false},"author":101,"featured_media":10901,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[108,8,1,776,91,29,2255],"tags":[25,1246,81,353,1248,1247,1231,789,368,1244,1140,1237,1141,1142],"class_list":["post-10898","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-big-data-2","category-data-protection","category-data-transformation2","category-etl","category-iri-workbench","category-test-data","category-archived-articles","tag-big-data","tag-cosort-speed","tag-hadoop","tag-hadoop-alternative","tag-hadoop-cluster","tag-hadoop-performance","tag-hdfs","tag-iri-voracity","tag-mapreduce","tag-oozie","tag-spark","tag-spark-stream","tag-storm","tag-tez"],"acf":[],"yoast_head":"\nWhen to Use Hadoop - IRI<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"When to Use Hadoop\" \/>\n<meta property=\"og:description\" content=\"In the last couple of years since this article about the use of IRI CoSort as a parallel data manipulation alternative to Hadoop, IRI released the Voracity platform to process big data in either paradigm. With its \u201cmap once, deploy anywhere\u201d capability, Voracity can run many popular jobs for data transformation, masking, and test dataRead More\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/\" \/>\n<meta property=\"og:site_name\" content=\"IRI\" \/>\n<meta property=\"article:published_time\" content=\"2017-02-16T20:42:49+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-02-23T20:24:47+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/02\/IRI-Workbench-GUI-for-Voracity.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1032\" \/>\n\t<meta property=\"og:image:height\" content=\"660\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Donna Davis\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Donna Davis\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/\"},\"author\":{\"name\":\"Donna Davis\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/52271b71b77d927ce9421530e2b1260b\"},\"headline\":\"When to Use Hadoop\",\"datePublished\":\"2017-02-16T20:42:49+00:00\",\"dateModified\":\"2026-02-23T20:24:47+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/\"},\"wordCount\":482,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/02\/IRI-Workbench-GUI-for-Voracity.png\",\"keywords\":[\"big data\",\"CoSort speed\",\"hadoop\",\"hadoop alternative\",\"hadoop cluster\",\"Hadoop performance\",\"HDFS\",\"IRI Voracity\",\"MapReduce\",\"Oozie\",\"Spark\",\"Spark Stream\",\"Storm\",\"Tez\"],\"articleSection\":[\"Big Data\",\"Data Masking\/Protection\",\"Data Transformation\",\"ETL\",\"IRI Workbench\",\"Test Data\",\"Archived Articles\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/\",\"url\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/\",\"name\":\"When to Use Hadoop - IRI\",\"isPartOf\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/02\/IRI-Workbench-GUI-for-Voracity.png\",\"datePublished\":\"2017-02-16T20:42:49+00:00\",\"dateModified\":\"2026-02-23T20:24:47+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/#primaryimage\",\"url\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/02\/IRI-Workbench-GUI-for-Voracity.png\",\"contentUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/02\/IRI-Workbench-GUI-for-Voracity.png\",\"width\":1032,\"height\":660,\"caption\":\"IRI Workbench GUI for Voracity\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.iri.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"When to Use Hadoop\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.iri.com\/blog\/#website\",\"url\":\"https:\/\/www.iri.com\/blog\/\",\"name\":\"IRI\",\"description\":\"Total Data Management Blog\",\"publisher\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.iri.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.iri.com\/blog\/#organization\",\"name\":\"IRI\",\"url\":\"https:\/\/www.iri.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"contentUrl\":\"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png\",\"width\":750,\"height\":206,\"caption\":\"IRI\"},\"image\":{\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/52271b71b77d927ce9421530e2b1260b\",\"name\":\"Donna Davis\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.iri.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/f109ab98ab74af3d4419d9d477bb85db?s=96&d=blank&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/f109ab98ab74af3d4419d9d477bb85db?s=96&d=blank&r=g\",\"caption\":\"Donna Davis\"},\"url\":\"https:\/\/www.iri.com\/blog\/author\/donnad\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"When to Use Hadoop - IRI","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/","og_locale":"en_US","og_type":"article","og_title":"When to Use Hadoop","og_description":"In the last couple of years since this article about the use of IRI CoSort as a parallel data manipulation alternative to Hadoop, IRI released the Voracity platform to process big data in either paradigm. With its \u201cmap once, deploy anywhere\u201d capability, Voracity can run many popular jobs for data transformation, masking, and test dataRead More","og_url":"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/","og_site_name":"IRI","article_published_time":"2017-02-16T20:42:49+00:00","article_modified_time":"2026-02-23T20:24:47+00:00","og_image":[{"width":1032,"height":660,"url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/02\/IRI-Workbench-GUI-for-Voracity.png","type":"image\/png"}],"author":"Donna Davis","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Donna Davis","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/#article","isPartOf":{"@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/"},"author":{"name":"Donna Davis","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/52271b71b77d927ce9421530e2b1260b"},"headline":"When to Use Hadoop","datePublished":"2017-02-16T20:42:49+00:00","dateModified":"2026-02-23T20:24:47+00:00","mainEntityOfPage":{"@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/"},"wordCount":482,"commentCount":0,"publisher":{"@id":"https:\/\/www.iri.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/#primaryimage"},"thumbnailUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/02\/IRI-Workbench-GUI-for-Voracity.png","keywords":["big data","CoSort speed","hadoop","hadoop alternative","hadoop cluster","Hadoop performance","HDFS","IRI Voracity","MapReduce","Oozie","Spark","Spark Stream","Storm","Tez"],"articleSection":["Big Data","Data Masking\/Protection","Data Transformation","ETL","IRI Workbench","Test Data","Archived Articles"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/","url":"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/","name":"When to Use Hadoop - IRI","isPartOf":{"@id":"https:\/\/www.iri.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/#primaryimage"},"image":{"@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/#primaryimage"},"thumbnailUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/02\/IRI-Workbench-GUI-for-Voracity.png","datePublished":"2017-02-16T20:42:49+00:00","dateModified":"2026-02-23T20:24:47+00:00","breadcrumb":{"@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/#primaryimage","url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/02\/IRI-Workbench-GUI-for-Voracity.png","contentUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/02\/IRI-Workbench-GUI-for-Voracity.png","width":1032,"height":660,"caption":"IRI Workbench GUI for Voracity"},{"@type":"BreadcrumbList","@id":"https:\/\/www.iri.com\/blog\/data-transformation2\/when-to-use-hadoop\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.iri.com\/blog\/"},{"@type":"ListItem","position":2,"name":"When to Use Hadoop"}]},{"@type":"WebSite","@id":"https:\/\/www.iri.com\/blog\/#website","url":"https:\/\/www.iri.com\/blog\/","name":"IRI","description":"Total Data Management Blog","publisher":{"@id":"https:\/\/www.iri.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.iri.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.iri.com\/blog\/#organization","name":"IRI","url":"https:\/\/www.iri.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","contentUrl":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2019\/02\/iri-logo-total-data-management-small-1.png","width":750,"height":206,"caption":"IRI"},"image":{"@id":"https:\/\/www.iri.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/52271b71b77d927ce9421530e2b1260b","name":"Donna Davis","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.iri.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/f109ab98ab74af3d4419d9d477bb85db?s=96&d=blank&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/f109ab98ab74af3d4419d9d477bb85db?s=96&d=blank&r=g","caption":"Donna Davis"},"url":"https:\/\/www.iri.com\/blog\/author\/donnad\/"}]}},"jetpack_featured_media_url":"https:\/\/www.iri.com\/blog\/wp-content\/uploads\/2017\/02\/IRI-Workbench-GUI-for-Voracity.png","_links":{"self":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/10898"}],"collection":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/users\/101"}],"replies":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/comments?post=10898"}],"version-history":[{"count":10,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/10898\/revisions"}],"predecessor-version":[{"id":12445,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/posts\/10898\/revisions\/12445"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/media\/10901"}],"wp:attachment":[{"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/media?parent=10898"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/categories?post=10898"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.iri.com\/blog\/wp-json\/wp\/v2\/tags?post=10898"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}