The rise of Big Data and the struggle of utilities
One of the biggest buzz words in business of the last years is Big Data (BD). Like many other organisations, companies in the gas and power sector are now adding data to their strategic agenda as one of their core assets. However, many utilities are struggling with creating added value to their business processes from the enormous surge of data in the last few years. The time that utilities can suffice with only structured data in relational databases seems to be over soon.
Disrupting technologies to process and analyse large amounts of (un)structured data are commercialised in ever faster pace. For example, NoSQL databases for key-value and file-based data storage, Hadoop for large scale data processing and querying of multiple data sources and many machine-learning based platforms for real-time data-analysis. An increasing number of suppliers also offer these technologies in the cloud, thereby reducing the burden on organisations’ internal IT departments.
Although the potential benefit of utilising advanced analytics and big data is beyond any doubt, the utility sector lags far behind other sectors like the Financial Services and Telecom industry. In this article, we review how utilities can move forward in their Big Data and Analytics evolution. The scope of this article will be on load forecasting
The potential of Big Data for load forecasting
The business potential of BD for the load forecasting is relatively high. Partly, thanks to the roll-out of smart metering and the development of smart appliances, driving the Internet of Things (IoT) and generating far more data compared to the previously used analog meter device. This is reinforced by data availability of other data types such as machine-to-machine transaction logs, GPS data, social media data and increasingly accurate weather data including their percentile distributions per point in time.
In potential, this enables utility companies to obtain far more knowledge about their customers’ consumption patterns. However, and not without reason, the limiting factor to realize this potential is privacy regulation for consumer protection. So, the challenge is to act within the regulatory boundaries in order to harvest and utilize source data that adds value to your business processes. Of course with supply forecasting, like wind and solar, privacy is less of an issue.
Load forecasting focus areas for BD analytics implementation
Big data analytics for utility companies can be categorised in 5 main focus areas: Data governance and integration, energy data warehousing, prediction based activities by deep data analytics, real-time (high volume) data processing and KPI reporting and monitoring (Figure 1). Each of these focus areas should be targeted in the same consecutive order, from strategically to more operationally focused until finally operational performance is measured and reported.
Figure 1 : Some of the most important Big Data Analytics focus areas and their challenges for load forecasting listed. Each challenge needs to be countered to add value in the operation.
Using deep data analytics and real time data processing to reduce volume imbalance in load forecasting
Deep data analytics, should be the one of the main drivers for adding value to the operational forecasting processes. This focus area aims at obtaining more insight into consumption and intermittent production behavior by using different types of predictors. Subsequently, these insights need to be operationalized by improving the forecasting methods and models used in the daily operation. Most of the value add for deep data analytics is expected to lay in the long term forecasting activities (Figure 2). This is the domain where simple regression based models fail to provide an accurate forecast. Particularly in demand forecasting, where causal relationships between consumption behavior and its predictors become more ambiguous. This is the area where analysis is performed on a high ‘variety’ with a questionable ‘veracity’ data set with many different (structured and unstructured) input variables. For example, new models using computational intelligence like neural networks can be created. The major challenge here is to bring professionals from different disciplines together and letting them work as one analytics team. As an example, it could be needed to include people with a marketing, psychology and IT background to find relationships between data from social media and customer churn rates.
Looking at short term forecasting, big data analytics adds value by utilizing increasing ‘volumes’ of data with a higher ‘velocity’, imported in a mainly structured format. For example, with increasing numbers of smart meters being implemented worldwide (680 million by 2017), some utilities expect an explosion of incoming data. Making most of this data in forecasting requires ultra-fast processes for importing, validating, cleaning, forecast calculation and exporting for intraday trading and dispatch. As an extension, simulation models can calculate a multitude of ‘what-if’ scenarios on the influence factors before the actual forecast is send out to downstream activities in order to minimise imbalance risks.
Figure 2 : Main forecasting dimensions and their mostly used influence factors to predict power and gas load on the grid.
Start with the base and create a roadmap for advancing step-by-step
From a high-level perspective, the potential of BD for load forecasting is crystal clear. However, before moving to the more advanced ways of data handling and analytics, utilities first have to make sure that their base energy data management is under control. As anyone in the field of data analytics and modelling knows: “Garbage in is garbage out”. A rule of thumb here is that about 80% of the forecasting accuracy is determined by the quality of the input data. Concretely, this means that the import, storage and validation of meter data like allocation, real time signals and reconciliation data is first in place. Moreover, customer portfolio data and the switch register need to be up to date and easy to query by forecasting models (See No time to waste, get your data under control).
Once the base energy data management processes are under control, utility companies can start with creating a roadmap towards the future. The roadmap should build upon the base with adding BD related activities. Each of the 5 mentioned BD focus areas should be addressed in the roadmap, from data governance and integration all the way to KPI reporting and monitoring. The way the roadmap will look like is highly company specific and depends on the long term goals of the company. Sia Partners can offer assistance in getting the energy data management processes under control as well as designing a roadmap towards the future to create value from Big Data.