How Do You Use Electricity ?

Collecting data with smart-meters

Smart-meters, through their ability to communicate data instantly, are re-shaping the electricity market landscape. Indeed, these new-generation meters collect and transmit instantaneous electricity consumption data, which can then be used by various actors ranging from the user (e.g. to monitor its own usage) to the supplier (e.g. to forecast energy demand) via independent companies (like Cambridge Energy Data Lab) which help make more sense of this data.

In this short study, we will focus on identifying generic behaviours of electricity consumption within a dataset of more than 400 users for the February-March 2014 period. Because the raw dataset is impossible to interpret, we will perform what is usually referred to as a "model reduction."


Principal component analysis

The first step in the analysis is to perform a model reduction to define several types of days. Amongst all the unique day time-series, we select few thousands (8000 days exactly, out of the 60 days x 400 users = 24000 total days available) in order to perform a Principal Component Analysis.  PCA is a linear algebra method used in order to find directions of largest variance in a dataset composed of several samples of a given variable. See figure 1 for a visual example.

Figure 1: PCA, 2-dimensional example. PCA finds the orthogonal directions which maximise the variance of the samples. 

After having performed a PCA, we can order the different samples along the first principal component (PC1 in Figure 1). We perform a PCA on the dataset composed of the 8000 different days of electricity consumption and order the different days along the first principal component. The result is presented in Figure 2.

Figure 2: PCA performed on a dataset of 8000 days of electricity consumption. The days are ordered with respect to their coordinate along the first principal component.

We notice that the days are now ordered with respect to a relevant criterion since we can detect a continuous evolution from users consuming electricity during the day and in the evening (top of Figure 2) to users who mostly use electricity in the evening and at night (bottom of Figure 2). From this observation, we can therefore define different types of days.

We then simplified the full dataset thanks to this criterion, creating around 10 different "types of days." It is therefore possible to simplify the 2-month time-series by attributing a value to each day corresponding to its type. This is represented in the left panel of Figure 3.

Figure 3: Left panel: unordered "type of day" time-series. Right panel: ordered "type of day" time-series obtained by ordering along the first principal component. We notice an evolution from users who mostly use electricity during the day (top) to users who mostly use electricity at night (bottom).

By re-applying the concept of first principal component ordering, we can re-order the simplified "type of day" time-series. This is presented in the right hand side panel of Figure 3. This time, more than ordering the time-series of the days, we manage to order the users. Each separate user can therefore be attributed to a category, depending not only on the type of daily consumption, but also on the longer time-scale (weekly, monthly) behaviour. Indeed, at the top of the right side panel are represented the "type of day" time-series for the users consuming electricity mostly during the day and the evenings, whereas the bottom part of this colour plot is associated with users consuming electricity mostly at night time. We can also notice on this figure a longer time-scale behaviour ordering, and the signature of the week-ends where people tend to stay awake (and use more energy) later at night.

Conclusion

The large amount of data collected by smart-meters can only been visualised and interpreted by using advanced mathematical tools, PCA being one of them. This method allowed us to successfully define different types of days in terms of electricity usage and therefore simplify the complete users' electricity time-series. From this model reduction, another PCA was then performed to directly order the users, therefore gaining insight about the different types of electricity consumption behaviour present in the dataset.

Smart Houses with Batteries and Renewable Generation

Concept

The preferred method of electricity consumption in the UK is to buy electricity from the grid - through a contract with an electricity retailer - and use it for daily needs such as lighting, heating, cooling, cooking, etc. However, several alternatives to this simple unilateral flow of energy from the grid to devices exist. We will focus on two other types of energy consumption behaviour, enabled by both domestic energy storage and production.

Energy storage consists of storing electricity under an alternative form (potential, chemical, thermal, mechanical, etc.) and, when needed, converting it back to electrical energy. Electricity storage has begun to be applied to houses and buildings, and it shows potential to both reduce customers' energy bills and help bridge the gap between energy demand and supply. Electricity is usually more expensive at peak times, and, generally, is cheaper at night than during the day. Therefore, using batteries to store electricity at night and re-use it during the day can be profitable.

Domestic energy production relies on harvesting energy from natural sources (such as wind, solar radiation, etc.) and using this stream of energy along with energy coming from the grid to meet a home's demands. The UK has installed solar panels on half a million houses so far, and plans to extend this to 10 million by 2020 [1]. With such equipment, households are not only able to produce a portion of the energy they use, but can also directly sell the energy they produce back to the grid through feed-in tariffs.

These two approaches to energy distribution, domestic production and usage can even be combined to create smart houses (see Figure 1), which are powered by incoming electricity from the grid, battery discharge and renewable energy. We can observe that widespread implementation of such strategies smoothes the electricity peak demand, allowing energy producers to more accurately predict the overall needs of the grid.

Figure 1: Schematic of a house equipped with both a solar panel and a battery.

Proof of concept

At Cambridge Energy Data Lab, we prefer crunching actual data to help us make real energy predictions. Therefore, we analysed the electricity consumption of approximately 500 houses, all equipped with lithium-ion batteries, and approximately 40% also equipped with solar panels.

We first focused on the group of houses equipped with batteries only and analysed their daily energy usage. The aggregated (averaged over all the users, for the winter period) results are displayed in Figure 2.

Figure 2: Averaged daily electricity usage for accommodations equipped with battery but no solar panel.

We can clearly see that a sizeable fraction of the energy required during the day (when the electricity rates are expensive) is shifted to the night through the charging of the battery. This stored energy is then released during the day. When electricity becomes cheaper later in the evening, the battery starts to charge anew.

The same analysis can be carried out for the group of users with solar energy generation. The results are presented in Figure 3.

Figure 3: Averaged daily electricity usage for houses equipped with battery and solar panels.

The same observations are made for this group of users: the daytime energy demand is partly moved to nighttime. Moreover, during the day, solar power is produced and allows for less intensive usage of the battery. When a surplus of energy is present, it is sold to the grid and generates income.

When looking at the data for a single day (see Figure 4), we realise that the combination of both solar power generation and energy storage with a battery is very effective at minimising electricity purchase from the grid, especially during the day when it is the most expensive.

Figure 4: Single day analysis: the energy bought from the grid is minimised during the day.

Conclusion and further analysis

This preliminary analysis shows the potential of individual electricity generation and storage. The electricity usage in a house can be distributed along different streams and optimised to reduce the overall cost for the customer. But these installations are very costly too. How long would it take you to reimburse such an investment? We plan to analyse this in the future, so stay tuned!

[1] http://www.theguardian.com/environment/2014/jan/29/uk-10-million-homes-solar-panels-2020

Energy Surplus Trends from Domestic UK Solar Panels in October 2013 to January 2014

According to the statistics provided by the Department of Energy and Climate Change, around 1,900 solar schemes, Feed-In-Tariff (FiT) for solar panels installation, have been installed every week during the past year in the UK. By 5 January, about half million solar schemes had been installed in total.^1,2 The solar energy revolution has started, but how much energy can actually be produced using solar panels?

At our lab, we analyzed the energy surplus produced from 1 October 2013 to 31 January 2014, and here we report some basic statistics about this selection of customers.

In these 4 months, our customers had an average energy surplus of approximately 827 kWh, which is similar to the average monthly energy consumption of an American house.³ The lowest surplus obtained by a customer was approximately 340 kWh, while the highest was approximately 1483 kWh.

We recorded the highest peaks of the energy surplus between November and December, although two others significant peaks were recorded in the first half of October and at the beginning of January.

References

1. Solar panels on half a million UK buildings, figures suggest, Jessica Shankleman for BusinessGreen, part of the Guardian Environment Network theguardian.com
2. Weekly solar PV installation and capacity based on registration date
3. U.S. Energy Information Administration website

From energy consumption data to energy profiles

We are all using electricity, non-stop, 24 hours a day. Electricity is our constant companion, but hardly anyone thinks about electricity when we turn on a light or watch the news on TV. Our utility bill is only some abstract amount of electricity we used over a long period of time, so we pay and forget about it.

At Cambridge Energy Data Lab, we think there is much more value in the details of how you use electricity. We believe that if you know and understand your own electricity usage better, you can save money and energy. Our aim is to create insights from your day-to-day electricity consumption to find the best price plan for you, and to convey this knowledge to help you adjust your habits and take control of your electricity usage.

Electricity consumption is not necessarily very regular and depends on many different factors. This is the electricity consumption of a household over a period of days:

The raw energy usage of an exemplary household.  

We can see immediately how the energy usage of this household is extremely irregular, so will have to do some further data analysis to provide the insights we are aiming for. Let's try to create an aggregated energy profile for this household. Using time-series analysis, we first decompose the original data into a periodic day-to-day component, a trend component, and a remainder which cannot be explained by the periodic and trend component:

The decomposition of the raw electricity usage data of a particularly low-usage household. The first panel shows the original data set which we decompose into 3 separate components: a periodic day-to-day element (called seasonal in timeseries analysis), the trend component indicating a smooth overall trend, and the remainder which is the partial data which cannot be explained by the other components.

This decomposition based on a daily interval is just the first step. Let's look at the periodic day-to-day component and decompose it a second time on an hourly frequency interval:

The second decomposition of the periodic component from the previous decomposition. We now have an periodic element with a hourly frequency. The remainder has been omitted. 

From the seemingly irregular raw data, we arrive at an aggregated energy profile which shows a clear trend of high electricity usage between the morning and evening hours. We can also see the periodic element of appliances, such as a refrigerator, in the seasonal component. This is just the first step, however, and much more sophisticated analysis is still to come. Nevertheless, it demonstrates that even your day-to-day electricity usage, though it doesn't look like much, is full of valuable insights.

Our goal is to develop methods and tools to make your electricity consumption data accessible to you! 

Eneberg - Domestic PV Generation Forecasting and Trading Software

Our Product Eneberg for PV Operators/ aggregators. 

Adding "Energy" and "Bloomberg" to evoke the image of an energy-trading software, we proudly present Eneberg as one of our core services. Eneberg is a web-based software for PV operators/aggregators to provide accurate predictions for the amount of electricity will sell to the wholesale market or their consumers (usually industrial or commercial customers). As savvy readers might know already, electricity supply and demand needs to balance at a 30 minute scale for stability in the most countries. Renewable energy generators have paid tremendous resources to forecast energy generation as accurately as possible in order to minimise “imbalance cost," which they are exposed to if they generate more or less electricity than their commitment.

As illustrated below, "Eneberg" is mainly focusing on "Aggregated Domestic PV," which makes the Eneberg unique in the market. Aggregated domestic electricity generations from rooftop PVs, which is still niche but is quickly growing, is our main focus, and having smart meters allows the generation data to be easily available.

Imbalance Mechanism

Prior to generation, an electricity provider must make a commitment to the amount of electricity they will supply. Deviations from this amount are penalized with an "imbalance cost," but the penalties are asymmetrical. If the generator produces too much, it will absorb the additional costs. If the generator produces too little, there are harsh regulatory penalties to pay. Therefore, forecasting accuracy should be conservative, underestimating the required electricity supply provisions to avoid paying the regulatory penalties.

Case study in Japan

We provided this forecast for one of our major Japanese customers. The electricity market in Japan, called JPEX (Japan Electricity Power Exchange), has their imbalance mechanism represented by the blue line. Up to +3% over the supply requirement, JPEX will buy electricity at an agreed rate. Further generation, however, will be unpaid, so the profit will start to decrease after +3%. On the other hand, falling short of the supply requirement is heavily penalised, so profit will be more significantly damaged by not meeting the supply requirement. This demonstrates that the provider's commitment should be more conservative, and our calculated forecast, shown in the red distribution, reflects this.

Our Target and Approach

Our target is to minimise the imbalance cost by optimising the bidding strategies on the electricity wholesale market and improving forecasting algorithms. Our accuracy target is about 5% error, which yields about 90% of maximum profit.

There are many challenges that lie ahead of us; such as quantifying the measures for uncertainty of human behaviour and weather forecasts for geospatially distributed households. In order to have reliable prediction with these uncertainties and without the large datasets of household electricity usage that our model will eventually use, we have adopted Bayesian approaches, some of which will be explained in this blog later. 

3 Ways Smart Meters Could Save You Money

One of the biggest critiques of smart meters is that while they are informative to the user, they do not actively save the user money. This is partly true: currently, to see any savings on a smart meter investment, the user must actively adjust their habits based on the readings. Studies have been inconclusive on the value of smart meters in terms of reducing consumer energy expenditure, and with good reason. A fancy screen on the wall is still a fancy screen on the wall unless the user does something with the information.

The smart meters previously studied require valuable resources. They require time, dedication, and behavioral change at the household level to earn savings. In today’s hectic world that includes any combination of long commutes, hard work days, familial obligations, and much sought after personal time, there is little time left for the consumer to spend micromanaging their energy consumption. Current passive methods involve the whole-home integration of energy-saving, “smart” electronics that either consume less energy or work with the smart meter to periodically turn off at points where the rate rises too high, but these options aren’t cheap.

The question remains: Is it possible, or fair, to expect a smart meter to save us money without adopting a behavioral change or buying brand new appliances and electronics?             

The answer is yes! While it may take time to see the savings, the UK smart meter rollout is intended to ultimately help energy consumers and producers alike, and I’d like to speculate on three ways that the smart meter rollout could ultimately help consumers save money without having to adjust habits:

1. Smart meters open the door to short switching periods, allowing for more competitive pricing.

For those without a smart meter, the billing process involves a “meter reader” employee of the energy provider physically coming to the customer’s building and taking the measurement. It takes time—usually weeks—to schedule an appointment, take the measurement, send the final bill, and finish the paperwork to allow a customer to switch. With a smart meter, the reading can be queried instantly, drastically reducing the processing time. With customers able to change their provider in as little as a day, energy producers will need to ensure that their rates remain competitive. Predictive models (such as those being created at Cambridge Energy Data Lab) also increase in accuracy with a known number of consumers, so, in addition to other planning benefits, it is in the producer’s interest to incentivize customers to make a long term commitment to an energy plan. One method of achieving both of these would be in the form of reduced prices on their fixed rate, long-term plans.

2. More accurate, up-to-date energy readings will help energy producers waste less.

With smart meters, companies can gather data on customer energy expenditure habits not only on a monthly basis, but as often as every half-hour. Developing predictive models allows them to more accurately predict the energy they need to generate in a given time period. This allows for less fuel to be wasted and cost savings to be realized, a savings which could theoretically be passed down to the consumer and fuel (pun intended) more competitive rates.

3. Your data is valuable!

Predictive modeling allows for energy providers to minimize costs and provide efficient service. Smart meters (supposedly) encourage consumers to use less energy, which helps to forward the government’s current goals of emissions reduction. Both the energy companies and the government need your data. The widespread, smart meter rollout is recognition of the value of this data in achieving each party’s goals. Both energy providers and the government are literally saying, “We want to know how and when you use energy so badly that we are willing to give you a free smart meter to find out!” Because the use of new smart meters are in the best interests of all parties involved, these meters will be widely available to consumers and will pass these savings unto the user.

Rather empowering, isn’t it? Once the energy providers realize these savings and the government starts to see measurable progress toward emissions reductions, the value of your data goes up over time.

If your data becomes integral to the operation of your energy provider, why give away your data for free? We hope to develop the tools to realize a future scenario where the consumer can be compensated for using a smart meter in the form of significantly reduced energy bills.

4. As an added bonus, customer service will improve.

Most customers who have tried switching providers themselves have been met with lengthy phone calls, paperwork, and frustration. The scenario probably went something like this: You probably had to track down your last bill and the energy provider’s number, call and wait for a representative, wait a few weeks for someone to come read the meter, wait a few more weeks for your final bill to come, and finally get confirmation over a month later that you were taken off the plan, in addition to setting up your next plan with the new energy provider. Of course, a sizeable portion of customers have also reported being double billed for months after because their original supplier did not correctly take them off their supply list, resulting in additional hassle as the consumers fought to correct the issue. Smart meters should help to reduce these errors, or at least reduce the time taken to correct them. As they say, time is money!

Have something to say about these decidedly (if not overly) optimistic speculations? Leave a comment below! For more information on the smart meter rollout, here is the UK government page on the consultations and policies in place/being developed: https://www.gov.uk/government/policies/helping-households-to-cut-their-energy-bills/supporting-pages/smart-meters

Talent over CVs

As a young and dynamic startup, we are continuously looking for great new talent to join our team of data scientists. But talent is hard to find in a pile of CVs and, as a data science company, it seemed logical to use a data-driven approach to asses applicants. That's why we designed three simple data science challenges (which you can find on GitHub). The 3 different tasks target the different objectives of our company:

• Data Analysis and Visualisation
• Data Modelling, Machine Learning, and Prediction
• Web Development

Each of the tasks is designed to see which programming style you use and how well you document and communicate your code. Code that is not only of high-quality, but also is well-documented and easy to understand is our priority. Please take extra care that you push a polished version of your code.
Second, quality comes before quantity. The objective is not to find the best overall method, so please focus on a single approach rather than trying several methodologies. Remember that you work on an unknown dataset, so don't assume too much. Just try to satisfy the requirements of specialised methodologies.
Finally, we are always happy to see people addressing all three challenges at once, but this is certainly not required!
But enough of the instructions and let's showcase some great examples which we have received:

Cluster analysis of energy consumption data

credit: Dimitry Foures

credit: Philip Squires

Predicting energy production using Bayesian networks

credit: Jan Teichmann