In the Internet of Energy, data persistence matters when it comes to giving power generation companies insight into their IoT implementations, says Future Grid.
Open source data grid company Hazelcast has discussed details of a new IoT platform designed by Australia-based Future Grid, which is based on its technology and provides power generation companies with a real-time view of their IoT data.
The platform combines the in-memory capability of Hazelcast’s In memory Data Grid (IMDG) technology with Apache Cassandra, to process what are described as ‘extreme’ volumes of data.
Spinning up (cloud) server starts
As a company, Hazelcast now claims to operate hundred of thousands of installed clusters and over 39 million ‘server starts’ per month.
Now that it is working to extend its engineering capabilities with Future Grid, both firms are interested in helping utilities companies to automate the processing of sensor and smart meter data that crosses energy networks.
Future Grid’s customers, meanwhile, are collecting approximately three billion data points every day. This means that post-processing of that data equates to 20 billion records, since each record has multiple, individual data points. This is clearly a massive scaling challenge.
When Future Grid first tried to solve this problem, it used traditional relational databases. The team there quickly found that these could not cope with huge volumes of data in real time – the main issue being that they couldn’t execute algorithms against incoming data fast enough. Therefore, Future Grid decided to build its own solution, combining Hazelcast IMDG with Apache Cassandra’s persistent data store capabilities.
Chris Law, co-founder and managing director at Future Grid explained that his team implemented Hazelcast IMDG at the core of its product’s in-memory capability. For example, Hazelcast IMDG is integrated with Apache Cassandra, which provides internal data storage in regards to reference data, while still maintaining a distributed grid architecture. They found it a “straightforward process”, he said.
Read more: Inmarsat research: skills gap threatens IoT innovation in energy sector
Data persistence matters
For Future Grid, Cassandra’s data persistence capabilities were pivotal. In the context of storing data in a computer system, persistence means that data survives after the process with which it was created has ended. Essentially then, Future Grid combined the strengths of two open source solutions for its energy customers. Integrating Hazelcast IMDG with Cassandra makes more data available and effective.
Importantly, the combined solution maintains the high availability and horizontal scalability of Cassandra, while delivering performance that is 1,000 times faster than disk-based approaches.
“Using Hazelcast IMDG has enabled our customers to realize the dream of real-time data without the significant cost of traditional relational database models. Out of the box speed and resilience have helped our customers deliver operationally critical production systems,” said Future Grid’s Law.
Greg Luck, CEO of Hazelcast has also commented, saying: “Hazelcast IMDG has been designed to continually process big data volumes, while ensuring end-to-end latency. Our technology is inherently quick and solves storage issues by forming storage clusters. It can also transmit reactive access patterns, to notify analysts when values change. Therefore, it can be used as a cache for big datasets during processing, while forming in-memory data lakes for frequently used data. Importantly, it is very easy to deploy.”
Read more: SAP hopes to heat up IoT energy market with Centrica
Big data: the big picture
Perhaps one of the most pleasing aspects of this story – despite it being colored with the positive paintbrush of customer platitudes – is the fact that we are highlighting data persistence, data grids and deep dataset-level mechanics… all of which matter hugely to the IoT and, in this case, the Internet of Energy.
Let’s dig deeper as we continue to work to understand the lower substrates of the IoT and what it might help businesses to achieve.