Creating a data hub for Estonian national electricity market from scratch with Elering

As it seems that the rest of the world is catching up with building state-wide data hubs for energy market management, we thought to share how did we do it in Estonia.

We had been on several occasions involved in the world of electricity with our projects for Eesti Energia (the former energy monopoly in Estonia) self service solution, grid building management system, and creating our own solution Wizener for energy “demand-response” management. Therefore it was no big surprise we were selected by Elering (Estonian Transmission System Operator) as the project management experts in creating the data hub for national electricity market. It was February 2012 and Estonian electricity market was to be deregulated in 10 months.

Data hub, plain and simple, huh? Not quite. Figuring out what exactly had to be done down to smallest detail was a completely new challenge, as at the time it was one of the very first national data hubs to be created and there was no best practice to look at in the world. Energy markets had been opened before, but at the time there was no central data hub containing network and electricity contracts, metering data for invoicing, business rules and processes to change electricity providers online, in-time energy balance settlements etc. To add for the ambiguity, parts of the law were still being worked out while we started the process. Step by step, with energy market specialists from Elering, we wrote down the market rules and processes on top of what the law had already stipulated. We split the expected functionality into smallest pieces, defined the tasks different energy market players need to fulfill. Eventually we defined, what the data hub is and what it is not:

All the market participants (30+ DSO-s and 5+ energy operators), big and small, contributed to the process. We held several rounds of intense iteration meetings with them. During those meeting we put together a detailed picture of market processes, as they were not established before.

As energy market operates in a very country-specific environment, then no “out-of-the-box” solutions could actually meet the requirements. It was eventually understood in Elering too and the choice was to build a custom-made data hub. The decision was made in May 2012.

Trust has to be earned

Deadline for opening up the Estonian electricity market stayed on 01.01.2013. Elering had to be ready with operational data hub by this date. We were called as partners into the project.

Mutual trust was built quickly once we started working. In Codeborne we believe that agile is the only way to carry out a software project. It took some weeks for Elering to get used to agile principles. We trusted Elering as electricity specialists, they trusted us as technology specialists. Plus the personal trust between people and mutually accepted agile approach. And at some point we started hearing “you know this better, choose yourself” type of sentences.

We did not only code in this project, we also designed the data hub and managed this project for the result to be as efficient (like in most other projects!).

Development phase. Agile, of course

We started developing in May 2012 in some extra uncertainty as the law that was going to regulate these processes was still being worked out. From that moment, there were only 8 months left until the opening / deregulating the Estonian electricity market. The most active development time was between May and September. From our side there was 1 pair of software engineers involved in the project.

Almost like from agile textbooks, we had weekly iteration meetings with the customer and other market participants. User stories were written together in mutually understandable language (opposed to studying heavy document packs in separate rooms / premises). For us it was a fast and interesting learning process and our knowledge in electricity market functioning grew rapidly to the point where we started arguing already with the customer ;).

Elering had promised certain functionality and interfaces to the market participants and we were hired by Elering. Sometimes the interests between the hub-keeper and market participants differed. In some cases we also intermediated these disputes and proposed technical and procedural compromises that in the end solution would satisfy everyone.

As the time rolled, we had more and more meetings with market participants, demoing data hub with the latest functionality and learning their individual needs. We took quite many steps to accommodate the solution accordingly. Plus we did hands-on training for their specialists on how to use the system.

As the first priority, our goal was to enable market participants to be able to load their historical data into the data hub in the Fall 2012 to be completely ready by the year end. The first test loadings were done already in June. In a similar manner we went to the end of October, when active development was finished from Codeborne side. By that time, all the market participants had already been trained and assisted to upload their meter and contract information to datahub. At the end of the year, we handed the data hub over to Elering, who could now independently operate and maintain the system. The electricity market was opened 01.01.2013 as scheduled, one month later more than 65% (461 000) households had signed a new electricity contracts.

Security, privacy

Who can see my data? Privacy concerns are are a hot topic.

With our extensive background in banking, where security, data access rights etc are acute, we designed and developed a solution where each consumer can adjust, who of the sellers can see their data. Moreover, they can later review, who actually viewed it. Full control and transparency!

For authentication the data hub is using the Estonian nation-wide official Public Key Infrastructure (PKI) system. End users can use their national ID-cards or mobile-ID to log in. Those from abroad without an Estonian ID card can authenticate themselves banklink protocol provided by all major Estonian banks. Estonian Data Protection Inspectorate was also overseeing the development process.

Challenges along the way

There were several challenges we faced in this project.

From one side there were large players with automated processes and machine interfaces to handle the large volumes, while on the other side there were small players who basically wanted to upload an Excel file. Consequently they needed a graphical UI for all operations. Jumping ahead, after thorough interviews and understanding everyones’ needs, the eventual solution satisfied both small and big players. Big ones got a well documented API to mass upload their data. Smaller operators had very easy-to-use web application, where they could copy-paste metering and contract data directly from the Excel sheets.

The hub had to cope with peak volumes like receiving and distributing the hourly metering data in the beginning of each month. (In Estonian market, it is the responsibility of the grid operator to split the once-a-month read/predicted consumption data into hourly chunks based on agreed profiles.) The system had to remain responsive to all market players while processing these large datafiles. To enable this, we had a separate processing engine for metering data. The sender initially only receives an acknowledgement of receipt of a file. After the background processing ends, the sender will receive a detailed report of the processing results. Also while one large grid operator might send multiple files at the same time, we made sure that all parties’ files were processed with equal priority - the large players couldn’t “suffocate” smaller players’ message flow. The end result was efficient and cost effective, so it did not need huge hardware and could be operated on a single modern Intel-based server. A second server was added only for failover reasons.

Another example of not yet established market processes/workflows is the existing international “standard” message formats that were supposed to be exchanged between entities. They did not suit well for Estonian market, being overly complicated. Our challenge was to simplify them. The message formats were pruned iteratively during the development and changed quite many times to contain only the necessary info for maintaining contracts, metering point meta data, meter reads and a number of other uses. The end result - simple messages supporting all Estonian market particularities - made it easier for market participants to build their systems and allowed to keep amounts of transferred data smaller.

Volumes

Currently the system, serving the relatively small Estonian market has:

• 718,000 metering points
• 677,000 consumers
• with their 770,000 contracts Current size of the database is 750 GB with a growth rate of about 500 GB per year. For example, in September 2013, 2 million messages were sent into the system and 5 million messages were sent out. The system can easily be scaled up to meet even the wildest expectations of market growth.

The result

The system we built works reliably and needs little resources to maintain. No commercial software components are used in the hub - we only used open-source components. The whole system is fully under the control of Elering’s system administrator and does not require any outside intervention. The administrator has a dashboard for monitoring throughput in key components. Everything is taking place online - change of energy seller etc.

We are proud that the customer is happy and considers the decision to build the system from scratch the right one. After 1 year of operations, Elering released a press release on this topic.

What is happening in the rest of the world?

It still is quite a new topic, like we found out at the start of the project. Denmark has been working on their nationwide system for years. More can be found here. In Sweden there are regions / communes that have voluntarily combined forces to interchange relevant market data. Ontario province in Canada has a hub, although there is no free electricity market. Norway has announced a to start their Elhub.

It is very likely that it was not the last data hub to be built as the energy markets are facing new challenges with deregulation, renewables and new market processes. We are ready to share the experience.