{"id":10700,"date":"2016-11-30T12:11:11","date_gmt":"2016-11-30T17:11:11","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=10700"},"modified":"2021-02-16T12:59:36","modified_gmt":"2021-02-16T17:59:36","slug":"iri-iot-edge-aggregation","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/business-intelligence\/iri-iot-edge-aggregation\/","title":{"rendered":"IoT & IRI – Aggregation on the Edge"},"content":{"rendered":"

As a follow-on to my <\/span>introduction to IoT and MQTT<\/span><\/a>, this article describes how device data collected from standard climate sensors and sent through an MQTT (Message Queuing Telemetry Transport) server can be rapidly and inexpensively pre-aggregated by a single <\/span>SortCL<\/span><\/a> program (in <\/span>IRI CoSort<\/span><\/a>) on a tiny edge device or gateway.<\/span><\/p>\n

The resulting data (and metadata) can also feed a low-cost, high-performance hub — or enterprise IoT platform — where the larger IRI Voracity<\/a> data management environment can affect more data integration, analytics, and alerts.<\/p>\n

Advantages of IoT Computing on the Edge<\/b><\/h3>\n

As IoT devices become more prevalent, focus has shifted to the benefits of gathering and examining data while it is being generated, as close to the original source as possible. This relatively new concept is called edge analytics, which\u00a0<\/span>WhatIs.com<\/span><\/a> defines as “an approach to data collection and analysis in which an automated analytical computation is performed on data at a sensor, network switch or other device instead of waiting for the data to be sent back to a centralized data store.”<\/span><\/p>\n

The main advantage of edge aggregation and\/or analytics is that it decreases latency in the decision-making process. Instead of using more traditional methods on previously stored data, many enterprises learned that it can be faster, easier, and more practical to process and analyze data closer to its source. These same shops discovered this just as they realized that much of their growing data volumes did not have the relevance or utility they once thought, especially the data coming from IoT devices. \u00a0In fact, some of it is not worth storing at all:<\/span><\/p>\n

“A lot of the data that comes from IoT may not necessarily be data that we need to keep at an atomic level. I think we’re all enjoying the ability to keep more data, analyze more data, and get richer and deeper insights from all of these vast volumes of information. That said, just because you can, doesn’t mean you should.” \u00a0– Shawn Rogers, Quest Software<\/span><\/i><\/p>\n

Holding onto all that data can also be costly, and involve complexity and security issues<\/a>. Dealing with it on the edge precludes those issues and enables faster response times and easier places to decide what, if any, data is worth keeping. And, analytics that occurs on the edge can be performed in real-time, and thus be more useful. Finally, it can allow for more dexterity, scalability, and choices for distributing the data to various platforms, like Hadoop.<\/span><\/p>\n

Our First Forays<\/b><\/h3>\n

Starting at the bottom of this diagram and working up, you can see the components involved in my job, and their interaction with each other:<\/span><\/p>\n

\"POC<\/a><\/p>\n

As you review these steps, consider how they could\u00a0work in (or be modified to suit) your\u00a0environment:<\/p>\n

    \n
  1. A python script on a\u00a0<\/span>Raspberry Pi 3<\/span><\/a> Model B edge device tries to <\/span>read sensor data<\/b> every 5 seconds. <\/span>The Raspberry Pi is a single board computer running Raspbian GNU Linux 8, attached to an <\/span>HTU21DF<\/span><\/a> temperature and humidity sensor.<\/span><\/span>1<\/sup><\/a><\/span><\/li>\n
  2. If the data is read successfully, it is published <\/b>to an MQTT<\/a> broker, along with the timestamp of the sample, on a separate Linux system. We used the Eclipse \u201cMosquitto<\/a>\u201d implementation of MQTT.<\/span><\/span>2<\/sup><\/a><\/span><\/li>\n
  3. Another python script running on the collection system <\/span>subscribes <\/b>to the same topic from the MQTT broker.\u00a0<\/span><\/li>\n
  4. Each time that the second script reads a message from the broker, it <\/span>writes it to a file<\/b> that corresponds to the hourly period in which it was collected.<\/span><\/span>3<\/sup><\/a><\/span><\/li>\n
  5. When a message is received, which has a new hourly period in the timestamp, that second Python script also fires off a CoSort <\/span>SortCL<\/span><\/a> program that <\/span>aggregates<\/b><\/a> all the data in the file for the previous hour. Each file may contain up to 720 samples if there were no errors reading the sensor. This is aggregation <\/span>on the edge<\/b>, but SortCL can also do <\/span>analytics<\/span><\/a> as it reports (<\/span>embedded BI<\/span><\/a>), feed data directly into BIRT<\/a> or Splunk<\/a>, and\/or <\/span>prepare data<\/span><\/a> for another analyzer\u00a0like R<\/a> or visualizer like\u00a0Tableau<\/a>.<\/span>
    \n<\/span><\/span><\/span><\/p>\n

    Note … the Python scripts and intermediate files described in steps 3-5 above have recently been obsoleted by the CoSort v10 engine in IRI Voracity<\/a> wherein streaming IoT data over MQTT feeds in real-time to SortCL processes; see the demo video below.<\/em><\/span><\/i><\/p>\n<\/li>\n

  6. The SortCL program sends the minimum, average, and maximum temperature and humidity values, along with the hourly period and the total number of samples, to a MySQL database table via ODBC.<\/span>4<\/sup><\/a><\/span> Note that a future version of SortCL will process MQTT streams directly, which will preclude the need for the second Python script in steps 4-6 altogether.<\/li>\n
  7. Generating reports, charts, alerts, and <\/span><\/i>further analysis of the SQL (or any form of streaming or persistent data) <\/span>can also<\/span><\/i> be performed on any downstream server running the IRI Workbench <\/span>GUI for Voracity<\/span><\/a>, built on Eclipse. Voracity uses CoSort or Hadoop engines for SortCL jobs in (scheduled or ad hoc) ETL workflows, making it both an enterprise data management platform and <\/span>IoT hub<\/b>.<\/span><\/li>\n<\/ol>\n

    \"Collected<\/a><\/i><\/span><\/p>\n

    C<\/i>ollected aggregate data in MySQ<\/i>L<\/span><\/p>\n

    IRI\u2019s Advantages on the Edge<\/b>
    \n<\/span><\/h3>\n
      \n
    • Lightweight CoSort SortCL applications can run wherever Linux CoSort can be licensed, from a Raspberry Pi device, to a gateway server to which multiple devices are connected. <\/span>Its small command-line execution footprint means the program loads and runs quickly on the edge.<\/span><\/li>\n
    • The same SortCL script(s) can process small or large batches, or streaming data, so no design changes are needed when the processing requirement does.<\/span><\/li>\n
    • Since we are reading data from sensors, at some point in time, there might be a breakdown or communication fault at the edge. By quickly counting and aggregating the data collected in any given hour, we can spot missing entries and, nevertheless, maintain accurate summary information on any data that was registered.<\/span><\/li>\n
    • SortCL programs can filter, transform, reformat, mask data at the field level (to protect PII), generate reports, and create multiple file and table targets, all in the same job script and I\/O pass<\/em>. This means applications using different repositories and formats can use the same data without it needing to regenerate or synchronize it separately.<\/span><\/li>\n
    • Solutions are easier to configure and modify in SortCL than in a Python or 3GL program because SortCL uses simple <\/span>4GL<\/span><\/a>. You can tweak its self-documenting parameters in text scripts, or use IRI Workbench graphical <\/span>dialogs or diagrams<\/span><\/a> supporting its syntax.\u00a0<\/span>The same data and SortCL metadata can be used in Voracity, so you do mashups and analytics on larger data sets in the hub\/platform layer (see below). In a Voracity hub, those same job scripts can <\/span>also run in Hadoop<\/span><\/a>.<\/span><\/li>\n<\/ul>\n

      Then in the Hub …<\/b><\/h3>\n

      The <\/span>IRI Workbench GUI<\/span><\/a> for CoSort (or Voracity), built on Eclipse, can run on any Windows box. Just set up your connection to the data stores and job scripts, wherever they happen to be. Use it to design and run the jobs, and to analyze and take action on the data being collected and aggregated.
      \n<\/span>\u00a0      \"IRI<\/a><\/p>\n

      This screenshot of the IRI Workbench shows the input files, the data stored in one of them, the color-coded SortCL aggregation script (and its GUI-connected job outline), the MySQL database connection with details about the target table (lower left). The bottom console window shows the summary output of the job script. <\/span><\/p>\n

      IRI has also directly <\/span>integrated<\/span><\/a> SortCL data preparation with BIRT in Eclipse to produce visual analytics<\/span> like this:<\/span><\/p>\n

      \"Descriptive<\/a><\/p>\n

      This sort of descriptive analytics can also be <\/span>predictive<\/span><\/a>. <\/span><\/p>\n

      Watch how IRI Voracity can aggregate IoT data on the edge in this presentation:<\/p>\n