Hazelcast and Future Grid develop real-time smart meter platform
September 26, 2017
California-based Hazelcast’s open source in-memory data grid (IMDG) has been combined with the Apache Cassandra database manager to deliver a real-time smart meter IoT platform.
The IMDG has hundreds of thousands of installed clusters and over 39 million server starts per month. The IoT platform has been designed by Australian company Future Grid, the developer of an operational intelligence data platform that provides electric power generation utilities with a real-time, streamlined view of their IoT data.
The platform combines the in-memory capability of Hazelcast IMDG with Apache Cassandra to process extreme volumes of data cost effectively.
Future Grid works with several Australian utility companies to automate the processing of sensor and smart meter data that cross energy networks. Its customers are collecting approximately three billion data points every day. In terms of daily post processing this equates to 20 billion records as each record has multiple, individual data points – a massive scaling challenge.
To make the most of this information, its customers need real-time data aggregation and processing so they can make complex real-time decisions.
When Future Grid first tried to solve this problem it used traditional relational databases. However, it soon became apparent that traditional databases couldn’t cope with huge volumes of data in real time, the main issue being that they can’t execute algorithms against incoming data fast enough. Therefore, Future Grid decided to build its own system combining Hazelcast IMDG with Apache Cassandra’s persistence data store capabilities.
“We implemented Hazelcast IMDG at the core of our products in-memory capability, while also integrating it with a range of purpose built technologies to deliver the platform our customers required,” said Chris Law, managing director of Future Grid. “For example, Hazelcast IMDG is integrated with Apache Cassandra which provides internal data storage in regard to reference data while maintaining a distributed grid architecture. We found integrating Hazelcast with Cassandra was a very straightforward process.”
For Future Grid, Cassandra’s persistence capabilities were pivotal. In the context of storing data in a computer system, persistence means that data survive after the process with which they were created has ended. Therefore, Future Grid amalgamated the strengths of the two open source systems for its energy customers.
Integrating Hazelcast IMDG with Cassandra makes more data available and effective. Importantly, the combined system maintains the high availability and horizontal scalability of Cassandra, while delivering performance that is a thousand faster than disk-based approaches due to Hazelcast IMDG.
For the utility companies, these are the use cases covered:
- Power quality, interval and event derivations: Clean de-duplicate five minute power quality data and daily per device rollup that includes pre-calculations to make further analysis faster and more accurate.
- Loss of neutral detection: Using machine learning and fast data processing to monitor and predict safety issues, reducing shock instances significantly.
- Phase based substation aggregation: Transformer modelling using aggregate meter interval data to provide better visibility per phase substation usage. Used for long term asset planning, phase balancing and alerting of exceeding designed rating.
- Customer phase cross referencing: Using machine learning to investigate data correctness of meter to substation mappings including responsive, real-time visualisation.
The Hazelcast operational in-memory computing platform is helping companies such as Capital One, Chicago Board Options Exchange, Deutsche Bank, Ellie Mae and Mizuho Securities USA manage their data and distribute processing using in-memory storage and parallel execution for better application speed and scale.
“Hazelcast IMDG has been designed to continuously process big data volumes, while ensuring low end-to-end latency,” said Greg Luck, CEO of Hazelcast. “Our technology is inherently quick and solves storage issues by forming storage clusters. It can also transmit reactive access patterns to notify analysts when values change. Therefore, it can be used as a cache for big datasets during processing, while forming in-memory data lakes for frequently used data. Importantly, it is also very easy to deploy.”
Hazelcast is headquartered in Silicon Valley’s Palo Alto, with offices in Ankara, Istanbul, London and New York.