We are Cambridge Energy Data Lab, a smart energy startup based in Cambridge, UK.
This blog, named "Cambridge Energy Data Analysis", aims to incrementally unveil our big data analysis and technologies to the world. We are a group of young geeks: computer scientists, data scientists, and serial entrepreneurs, having a passion for smart energy and sustainable world.

Monday 31 March 2014

From energy consumption data to energy profiles

We are all using electricity, non-stop, 24 hours a day. Electricity is our constant companion, but hardly anyone thinks about electricity when we turn on a light or watch the news on TV. Our utility bill is only some abstract amount of electricity we used over a long period of time, so we pay and forget about it.
At Cambridge Energy Data Lab, we think there is much more value in the details of how you use electricity. We believe that if you know and understand your own electricity usage better, you can save money and energy. Our aim is to create insights from your day-to-day electricity consumption to find the best price plan for you, and to convey this knowledge to help you adjust your habits and take control of your electricity usage.

Electricity consumption is not necessarily very regular and depends on many different factors. This is the electricity consumption of a household over a period of days:
The raw energy usage of an exemplary household.  

We can see immediately how the energy usage of this household is extremely irregular, so will have to do some further data analysis to provide the insights we are aiming for. Let's try to create an aggregated energy profile for this household. Using time-series analysis, we first decompose the original data into a periodic day-to-day component, a trend component, and a remainder which cannot be explained by the periodic and trend component:

The decomposition of the raw electricity usage data of a particularly low-usage household. The first panel shows the original data set which we decompose into 3 separate components: a periodic day-to-day element (called seasonal in timeseries analysis), the trend component indicating a smooth overall trend, and the remainder which is the partial data which cannot be explained by the other components.
This decomposition based on a daily interval is just the first step. Let's look at the periodic day-to-day component and decompose it a second time on an hourly frequency interval:

The second decomposition of the periodic component from the previous decomposition. We now have an periodic element with a hourly frequency. The remainder has been omitted. 

From the seemingly irregular raw data, we arrive at an aggregated energy profile which shows a clear trend of high electricity usage between the morning and evening hours. We can also see the periodic element of appliances, such as a refrigerator, in the seasonal component. This is just the first step, however, and much more sophisticated analysis is still to come. Nevertheless, it demonstrates that even your day-to-day electricity usage, though it doesn't look like much, is full of valuable insights.

Our goal is to develop methods and tools to make your electricity consumption data accessible to you! 

Monday 24 March 2014

Eneberg - Domestic PV Generation Forecasting and Trading Software

Our Product Eneberg for PV Operators/ aggregators. 

Adding "Energy" and "Bloomberg" to evoke the image of an energy-trading software, we proudly present Eneberg as one of our core services. Eneberg is a web-based software for PV operators/aggregators to provide accurate predictions for the amount of electricity will sell to the wholesale market or their consumers (usually industrial or commercial customers). As savvy readers might know already, electricity supply and demand needs to balance at a 30 minute scale for stability in the most countries. Renewable energy generators have paid tremendous resources to forecast energy generation as accurately as possible in order to minimise “imbalance cost," which they are exposed to if they generate more or less electricity than their commitment.

As illustrated below, "Eneberg" is mainly focusing on "Aggregated Domestic PV," which makes the Eneberg unique in the market. Aggregated domestic electricity generations from rooftop PVs, which is still niche but is quickly growing, is our main focus, and having smart meters allows the generation data to be easily available.

Imbalance Mechanism

Prior to generation, an electricity provider must make a commitment to the amount of electricity they will supply. Deviations from this amount are penalized with an "imbalance cost," but the penalties are asymmetrical. If the generator produces too much, it will absorb the additional costs. If the generator produces too little, there are harsh regulatory penalties to pay. Therefore, forecasting accuracy should be conservative, underestimating the required electricity supply provisions to avoid paying the regulatory penalties.

Case study in Japan

We provided this forecast for one of our major Japanese customers. The electricity market in Japan, called JPEX (Japan Electricity Power Exchange), has their imbalance mechanism represented by the blue line. Up to +3% over the supply requirement, JPEX will buy electricity at an agreed rate. Further generation, however, will be unpaid, so the profit will start to decrease after +3%. On the other hand, falling short of the supply requirement is heavily penalised, so profit will be more significantly damaged by not meeting the supply requirement. This demonstrates that the provider's commitment should be more conservative, and our calculated forecast, shown in the red distribution, reflects this.

Our Target and Approach

Our target is to minimise the imbalance cost by optimising the bidding strategies on the electricity wholesale market and improving forecasting algorithms. Our accuracy target is about 5% error, which yields about 90% of maximum profit.

There are many challenges that lie ahead of us; such as quantifying the measures for uncertainty of human behaviour and weather forecasts for geospatially distributed households. In order to have reliable prediction with these uncertainties and without the large datasets of household electricity usage that our model will eventually use, we have adopted Bayesian approaches, some of which will be explained in this blog later. 

Wednesday 19 March 2014

3 Ways Smart Meters Could Save You Money

One of the biggest critiques of smart meters is that while they are informative to the user, they do not actively save the user money. This is partly true: currently, to see any savings on a smart meter investment, the user must actively adjust their habits based on the readings. Studies have been inconclusive on the value of smart meters in terms of reducing consumer energy expenditure, and with good reason. A fancy screen on the wall is still a fancy screen on the wall unless the user does something with the information.

The smart meters previously studied require valuable resources. They require time, dedication, and behavioral change at the household level to earn savings. In today’s hectic world that includes any combination of long commutes, hard work days, familial obligations, and much sought after personal time, there is little time left for the consumer to spend micromanaging their energy consumption. Current passive methods involve the whole-home integration of energy-saving, “smart” electronics that either consume less energy or work with the smart meter to periodically turn off at points where the rate rises too high, but these options aren’t cheap.

The question remains: Is it possible, or fair, to expect a smart meter to save us money without adopting a behavioral change or buying brand new appliances and electronics?             
The answer is yes! While it may take time to see the savings, the UK smart meter rollout is intended to ultimately help energy consumers and producers alike, and I’d like to speculate on three ways that the smart meter rollout could ultimately help consumers save money without having to adjust habits:

1. Smart meters open the door to short switching periods, allowing for more competitive pricing.

For those without a smart meter, the billing process involves a “meter reader” employee of the energy provider physically coming to the customer’s building and taking the measurement. It takes time—usually weeks—to schedule an appointment, take the measurement, send the final bill, and finish the paperwork to allow a customer to switch. With a smart meter, the reading can be queried instantly, drastically reducing the processing time. With customers able to change their provider in as little as a day, energy producers will need to ensure that their rates remain competitive. Predictive models (such as those being created at Cambridge Energy Data Lab) also increase in accuracy with a known number of consumers, so, in addition to other planning benefits, it is in the producer’s interest to incentivize customers to make a long term commitment to an energy plan. One method of achieving both of these would be in the form of reduced prices on their fixed rate, long-term plans.

2. More accurate, up-to-date energy readings will help energy producers waste less.

With smart meters, companies can gather data on customer energy expenditure habits not only on a monthly basis, but as often as every half-hour. Developing predictive models allows them to more accurately predict the energy they need to generate in a given time period. This allows for less fuel to be wasted and cost savings to be realized, a savings which could theoretically be passed down to the consumer and fuel (pun intended) more competitive rates.

3. Your data is valuable!

Predictive modeling allows for energy providers to minimize costs and provide efficient service. Smart meters (supposedly) encourage consumers to use less energy, which helps to forward the government’s current goals of emissions reduction. Both the energy companies and the government need your data. The widespread, smart meter rollout is recognition of the value of this data in achieving each party’s goals. Both energy providers and the government are literally saying, “We want to know how and when you use energy so badly that we are willing to give you a free smart meter to find out!” Because the use of new smart meters are in the best interests of all parties involved, these meters will be widely available to consumers and will pass these savings unto the user.

Rather empowering, isn’t it? Once the energy providers realize these savings and the government starts to see measurable progress toward emissions reductions, the value of your data goes up over time.

If your data becomes integral to the operation of your energy provider, why give away your data for free? We hope to develop the tools to realize a future scenario where the consumer can be compensated for using a smart meter in the form of significantly reduced energy bills.

4. As an added bonus, customer service will improve.

Most customers who have tried switching providers themselves have been met with lengthy phone calls, paperwork, and frustration. The scenario probably went something like this: You probably had to track down your last bill and the energy provider’s number, call and wait for a representative, wait a few weeks for someone to come read the meter, wait a few more weeks for your final bill to come, and finally get confirmation over a month later that you were taken off the plan, in addition to setting up your next plan with the new energy provider. Of course, a sizeable portion of customers have also reported being double billed for months after because their original supplier did not correctly take them off their supply list, resulting in additional hassle as the consumers fought to correct the issue. Smart meters should help to reduce these errors, or at least reduce the time taken to correct them. As they say, time is money!

Have something to say about these decidedly (if not overly) optimistic speculations? Leave a comment below! For more information on the smart meter rollout, here is the UK government page on the consultations and policies in place/being developed:

Talent over CVs

As a young and dynamic startup, we are continuously looking for great new talent to join our team of data scientists. But talent is hard to find in a pile of CVs and, as a data science company, it seemed logical to use a data-driven approach to asses applicants. That's why we designed three simple data science challenges (which you can find on GitHub). The 3 different tasks target the different objectives of our company:
  • Data Analysis and Visualisation
  • Data Modelling, Machine Learning, and Prediction
  • Web Development

Each of the tasks is designed to see which programming style you use and how well you document and communicate your code. Code that is not only of high-quality, but also is well-documented and easy to understand is our priority. Please take extra care that you push a polished version of your code.
Second, quality comes before quantity. The objective is not to find the best overall method, so please focus on a single approach rather than trying several methodologies. Remember that you work on an unknown dataset, so don't assume too much. Just try to satisfy the requirements of specialised methodologies.
Finally, we are always happy to see people addressing all three challenges at once, but this is certainly not required!
But enough of the instructions and let's showcase some great examples which we have received:

Cluster analysis of energy consumption data

credit: Dimitry Foures

credit: Philip Squires

Predicting energy production using Bayesian networks

credit: Jan Teichmann

Hello World. We are "Cambridge Energy Data Lab".

Hello, World!

We are an energetic data-driven start-up based in Cambridge, UK, and this is our first blog post. The company, Cambridge Energy Data Lab (CEDL), was founded in 2013 by a group of young geeks: computer scientists, data scientists, and serial entrepreneurs. As you can easily guess from the company's name, we are focused on "energy data" in the form of electricity generation/consumption numbers, and we claim ourself as a "Lab" since we are research-orientented engineering team. All of our team members either have or are pursuing masters or PhD's in engineering/computer science/mathematics/etc. from an array of international top-tier institutions, including the University of Cambridge.

Our Business 

Currently, our business is based in Asia and the UK. CEDL is providing smart-energy data analysis to utility companies, wind and PV developers, and other large energy-sector players. Our first product will involve forecasting electricity generation from domestic solar panels and electricity consumption in domestic households. Large amounts of domestic electricity consumption/generation data have been collected from households via smart-meter throughout our business partners, and we hope to use this data to develop several products to address the critical needs of the energy market.

This Blog's Mission

CEDL has been developing our forecasting algorithm for the past few months, but hereafter, we will incrementally unveil our research and ideas on this blog. It is important to note that all data in our blog posts will be fully anonymized. Within this limitation, we are going to try to share the essence of our technology to the world. Hopefully, it will inspire some of the people who share the same motivation with us: to make the world sustainable by optimising the inefficiency of electricity usage.

Why Cambridge?

A significant portion of our founding members are graduates/students at the University of Cambridge, so unlike London, Tokyo, San Francisco, or Oxford, it feels like home. With the University being perennially ranked as one of the best in the world, especially in engineering/technology, attracting brilliant talent with expertise in Machine Learning, Bayesian Statistics, and Mathematics is significantly easier. Moreover, the spirit of Newton, Darwin and other prominent scientists is still alive here. We feel that the rigorous atmosphere makes us think forward, unleashing us from short-sighted issues. Many startups have found inspiration and success in Cambridge, and we hope to do the same. We want to ignite the next innovation in smart-meter derived data solutions, and we hope you will follow us along the way.


Director of Cambridge Energy Data Lab.

The picture is on the Clare Bridge, the oldest remaining bridge in Cambridge (since 1640). Feeling the footsteps of Cambridge giants, I am walking on my own way.