These measurements include electricity, gas, temperature and the list goes on. Thanks to the success of real-time energy monitoring and the growing number of projects we are dealing with, our servers were receiving an extensive and increasing amount of data. For this, we found it adequate to redesign our existing system with a new architecture for data insertion and storage. The same Spotify and Twitter tech is called “Storm”. The new design would help us accommodate continued growth and make the system completely scalable.
After a 6 months period of hard work and dedication from our development team, the new architecture was up and running on January the 15th of this year. The resulting architecture contains three main elements: Storm, RabbitMQ, and Redis. Up next, we’ll explain the implications of the latter and how our system has been enhanced to provide our customers with a better service.
The three main actions that the insertion system does are:
- Calculate delta values (for example the calculation of the energy consumed in an hour or a day),
- check alerts, and
- store the data.
To get those actions done the most efficient and scalable way possible, the new product had to be of an Open-source nature, suitable for stream processing and with the utmost performance. Here is where the distributed, real-time computation system, Storm, comes into play. Storm was integrated into DEXMA’s system to complete the mentioned tasks the required way.
DEXMA receives more than 12 million new measurements every day coming from 30.000 meters.
Backtype, a marketing intelligence platform acquired by Twitter, implemented Storm in 2011. It was then when it was released as an open-source project. Storm, lets you divide the process into small tasks and decide how many resources you assign to each task. This is possible since “bolts” within the system complete individual tasks. This allows us to scale wisely. To illustrate its capabilities, Twitter, Spotify, Groupon, and now DEXMA, each uses it to process millions of data every day.
Using Open-source software components reassures our customers that we control all the development process. That there is no black-box depending on some external company. Open-source licenses enable our development team to control and understand the behaviour of software components and tune their performance to the limit. Ultimately, this allows our developers to learn techniques to implement in DEXCell Energy Manager. Additionally, this type of license allows us, at times, to fix bugs and contribute to the community.
Other Open-source components used by DEXMA include PostgreSQL, Redis, and MongoDB databases. RabbitMQ queuing system, Apache and Nginx webservers, Jenkins continuous deployment platform, and Linux operating system.
Spotify and Twitter tech
Besides being Open-source, Storm is perfect for stream processing, that’s why this technology is key for both Spotify and Twitter. This is because data is captured from queues (that act as a temporal buffer) and is processed passing by each of the bolts. RabbitMQ and RedisDB are both used in the process for queueing and caching respectively.RabbitMQ is one of the most flexible and robust message-queuing platforms while RedisDB is one of the fastest databases for key-value storage.
Why do we need stream processing?
In Big data, there are two types of data processing: Batch and Stream. In contrast to batch, where data goes through more steps, in-stream, data flows continuously. Since dataloggers (the gateways that deliver data from the meters) never stop sending data, stream data processing is a must. This faster processing mechanism is essential for DEXCell Energy Manager‘s first-class performance.
Having explained DEXMA‘s new data insertion and storage architecture, it’s time to talk about what’s in it for our clients. With the renewed platform, each client has its own queue (previously, they were shared queues). This allows each client to receive an individual treatment of their data with no risk of being affected by massive data flows from other clients. On top of this, both the product and the data insertion have greater stability. Last but not least, our clients will experience faster data processing and greater scalability.
With all this cleared up, we hope you enjoy our improved DEXCell Energy Manager!
