{"id":10530,"date":"2016-09-14T11:55:17","date_gmt":"2016-09-14T15:55:17","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=10530"},"modified":"2025-01-31T15:50:01","modified_gmt":"2025-01-31T20:50:01","slug":"optimizing-talend-transforms-with-cosort","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/data-transformation2\/optimizing-talend-transforms-with-cosort\/","title":{"rendered":"Improving Talend Performance"},"content":{"rendered":"

Talend has been on the market for several years now, and its flexible UI components make it a very reasonable choice for developers when it comes to customization. Talend Studio is built in Java, which gives it a certain degree of control and flexibility in managing its performance. That said, however, I found that Talend has two general limitations in processing large data sets:<\/span><\/p>\n

    \n
  1. It\u00a0requires too much RAM for data processing, especially sort, join, or aggregation. Talend will always try to do them in memory, and must have that memory pre-allocated to run the operation. For example, just to perform a lookup in a 2GB file, you must first allocate 2 GB as HEAP memory.<\/span><\/li>\n
  2. The performance, even after allocating enough RAM, would be dependent on the hardware resources and the configuration options used (e.g., in T-Map check mark \u201cStored on Disk\u201d or bulk loading, etc.) <\/span><\/li>\n<\/ol>\n

    IRI CoSort<\/span><\/a> continues to perform or accelerate high-volume data manipulations in DB, BI, and ETL environments. CoSort is also the default engine in the <\/span>IRI Voracity<\/span><\/a> data management platform, which, like Talend, is built on Eclipse\u2122, and now overlaps in many areas, including <\/span>ETL<\/span><\/a>. <\/span><\/p>\n

    Like other independent DW consultants have with <\/span>DataStage<\/span><\/a>, <\/span>Informatica<\/span><\/a>, and <\/span>Pentaho<\/span><\/a>, I performed multiple benchmarks on large data sets in Talend natively, with CoSort (Voracity) alone, and from a Talend workflow calling CoSort. The head-to-head comparisons speak for themselves, while the embedded use of CoSort overcame Talend\u2019s inherent processing limitations and need for additional hardware.<\/span><\/p>\n

    Configuration<\/b><\/h3>\n

    Following are the hardware and software specifications used for this POC:<\/span><\/p>\n

    Hardware<\/b><\/p>\n