Visit to Telekom Malaysia R&D

I’ve just returned from a visit to Telekom Malaysia Research & Development during which I delivered a 5 day course on non-intrusive appliance load monitoring. The course aimed to provide a broad overview of the range of research happening in both academia and industry as well as an in-depth description of the current state of the art. Below is a photo of the group taken outside their research facility:

With a roughly 50/50 split between presentations and practical exercises, the course covered the following areas:

• Introduction & history
• Data sets & preprocessing
• Event based & non-event based methods
• Data sets & accuracy metrics
• Disaggregation exercise using factorial hidden Markov models

I really feel that although many of the attendees started the course with little or no background in energy disaggregation, everyone left the course with a broad understanding and appreciation of the successes and limitations within the field. It was truly a pleasure to be invited by TM R&D and I hope they are able to invest their newly found knowledge in NIALM into practical solutions.

If you’re interested in a similar course for your company, please feel free to drop me an email.

GridCarbon Android app v2 released

I'm very happy to announce the release of an update to the GridCarbon Android app. The app allows you to track the carbon intensity of the UK electricity grid on your phone or tablet.

The demand for electricity in the UK varies throughout the day, and thus, the mix of generators supplying this electricity continually changes. As a result, the carbon intensity of the electricity – the quantity of CO2 produced for 1 kWh of electricity consumed – also varies continually. Deferring your use of electricity to off-peak times, when the carbon intensity is low, can help reduce your carbon footprint.

The app features a breakdown of the the current electricity generation sources:

The landscape view shows how both the carbon intensity of the grid and the generation sources varied over the previous 24 hours:

There is also an iOS version of GridCarbon which shares the same functionality as the Android app.

NILM in the New Scientist

A couple of weeks ago, Google acquired Nest, a smart thermostat manufacturer. The Nest thermostat aims to combine manually set temperature preferences with automatically learned household occupancy schedules. The acquisition seemed to prompt a few privacy concerns, given the unification of household occupancy data with the information Google currently stores about its customers. This is where NILM got its mention in the New Scientist, given that private information can also be inferred from household aggregate electricity data. A summary article even referenced research which showed that the film being watched on a plasma TV can be inferred from 2 Hz smart meter data, assuming two 5 minute periods of data can be extracted when no other appliances are changing state. Unfortunately though, none of these articles seemed to mention that such methods would require higher frequency data than will be automatically uploaded to utilities by smart meters, and therefore would require explicit consent from the customer to opt-in to such a system.

NILM 2014: Second International Workshop on Non-intrusive Load Monitoring

The Second International Workshop on Non-intrusive Load Monitoring is being organised by Mario Bergés and Zico Kolter in partnership with Pecan Street. The workshop is a follow up to the 2012 NILM Workshop held in Pittsburgh, which brought academics and vendors interested in energy disaggregation together for the first time. Full information about the upcoming workshop can be found at nilmworkshop.org.

Some important information:

• When: 3 June 2014
• Where: Austin, TX, USA
• Objective: To review the main types of approaches that have been explored to date to solve the problem of electricity disaggregation, and to then discuss possible paths forward knowing what has been tried and what has yet to be experimented.

The workshop will feature talks from invited speakers, paper presentations and a poster session. Attendees are be able to register for the workshop at nilmworkshop.org. Authors will also be able to submit papers via the same website, for which submission system is live and will close on 28 March 2014. The workshop will also be co-located with a two day workshop hosted by Pecan Street on 4-5 June.

Thesis defence complete!

Earlier this week I had to defend my PhD thesis against the critique of both an internal and external examiner in what is referred to as a viva. Despite my best efforts to prepare, I was still pretty nervous going into the exam. However, it turned out to be a lot more enjoyable than I had expected. My examiners managed to tease out the finer details of my thesis without being at all aggressive, which I think had a really positive effect on the atmosphere of the viva. The result of the viva was a number of minor corrections which I'll need to make to my thesis over the next couple of months. This means I'll hopefully make the final version of my thesis available online by the end of February 2014, but potentially sooner depending on how busy I am at the start of next year.

Energy disaggregation research at MLSUST 2013 workshop

The 2013 workshop on Machine Learning for Sustainability was recently held at at the NIPS conference in Lake Tahoe, NV, USA. The workshop was organised by Edwin Bonilla, NICTA and ANU, Tom Dietterich, Oregon State University, Theodoros Damoulas, NYU CUSP and NYU-Poly and Andreas Krause, ETH Zurich. The workshop invited papers which propose and apply machine learning algorithms to solve sustainability problems such as climate change, energy management and biodiversity monitoring. The workshop featured two poster sessions, in which the authors of the accepted papers were invited to present their work. Both poster sessions featured a paper on energy disaggregation, which I have briefly summarised below.

Interleaved Factorial Non-Homogeneous Hidden Markov Models for Energy Disaggregation. Mingjun Zhong, Nigel Goddard, Charles Sutton.

This paper proposes a method for disaggregating 2 minute energy consumption data into individual appliances. The approach is based upon an extension of the factorial hidden Markov model (FHMM), in which the appliance transition probabilities are dependent upon the time of day (non-homogeneous), and also the appliances are constrained such that only one appliance can change state per time slice (interleaved). The authors evaluate their approach on 100 homes from the Household Electricity Study, in which 20-30 days of sub-metered data from each household is used for training, while 5-10 days of data is held out for testing. The results show that both the interleaved and non-homogeneous extensions individually provide better performance than the basic FHMM, while a combination of the two provides the best performance. Finally, the authors identify a key finding in that the disaggregation accuracy varies greatly across different households, and raise this as an open problem for the NIALM community.

Using Step Variant Convolutional Neural Networks for Energy Disaggregation. Bingsheng Wang, Haili Dong, Chang-Tien Lu.

This paper proposes a method for disaggregating 15 minute interval aggregate energy data into individual appliances. The approach is based on Step Variant Convolutional Neural Networks (SVCNN), which use the aggregate energy consumption in the intervals t-2, t-1, t, t+1, t+2 to predict the energy consumption of each individual appliance in interval t. The authors evaluate their approach via cross validation using REDD, in which 3 houses are used to train the model while 2 other houses are used to test the performance. The results show that the SVCNN model achieves greater accuracy than both discriminative sparse coding models and factorial hidden Markov models. However, the results still show a relatively high whole home normalised disaggregation error of approximately 0.8, confirming the difficulty of the disaggregation of 15 minute energy data.

Further details on both the REDD and HES data sets are available in my post summarising the publicly available NIALM data sets.

Accuracy metrics for NIALM

Accuracy metrics are essential when evaluating the performance of an energy disaggregation algorithm in a given scenario. However, each paper seems to use a different metric when comparing their proposed approach to the state of the art. As a result, it is impossible to compare numerical results between papers. I don't believe this is the fault of the authors, since there is no single accuracy metric which is unquestionably better than all other accuracy metrics. Instead, the relevance of each metric depends largely on the intended use of the disaggregated data.

For example, if the use of disaggregated data is to be used to provide a breakdown of the energy consumption in a home, an accuracy metric which allows errors to cancel out over time would be suitable. However, if the use of the disaggregated data is to be used to suggest appliance loads to be deferred to a different time of day, a less forgiving accuracy metric would be required.

Therefore, inspired by discussions at the EPRI NILM 2013 workshop and in my recent involvement in the foundation of an open source disaggregation toolkit, I have decided to collect and categorise a list of commonly used accuracy metrics as shown below.

Event based metrics

Event based metrics assess how well a disaggregation algorithm detects appliance change events (e.g. washing machine turns on). However, it is not trivial to determine appliance events from sub-metered power data to be used as ground truth, and as a result often involve some subjective judgement (e.g. should a washing machine changing state mid-cycle from spin to drain constitute an event?). Furthermore, deciding whether a detected event matches a ground truth event is also not trivial (e.g. should a detected event that is 1 second apart from a ground truth event be matched?).

• True positives, false positives, false negatives, true negatives
• Confusion matrices
• True positive rate, false, negative rate, precision, recall, F-score

Non-event based metrics

Non-event based metrics assess how well a disaggregation algorithm is able to infer the power demand of individual appliances over time. As such, it is highly dependent upon the sampling rate of the sub-metered appliance data used as the ground truth. Such metrics have the advantage that sub-metered appliance data is easily collected by hardware installations, and requires little subjective judgement. However, non-event based metrics suffer from the disadvantage that disaggregation algorithms can score very highly by predicting all appliances to always draw zero power. This occurs as a result of most appliances remaining off for the majority of each day, and therefore the disaggregation algorithm is able to correctly predict each appliances power for the majority of each day.

• Mean error or disaggregation error (often normalised or squared)
• Hamming loss

Overall metrics

Overall metrics assess how well a disaggregation algorithm is able to infer the total energy consumed by individual appliances over a period of time. Such metrics are often the most intuitive, since they directly correspond to the pie chart of household energy consumption (e.g. as provided by Neurio). However, overall metrics allow errors to cancel out over time (e.g. an appliance's power is overestimated on day 1, while it is underestimated on day 2, resulting in the algorithm being assigned 100% accuracy since these errors cancel each other out).

• Error in allocated energy (often normalised or squared)
• Percentage energy correctly allocated

This list is mostly intended as a starting point for discussion regarding accuracy metrics, so please leave a comment if you notice any metrics I've left out!