UK energy disaggregation meet-up

The field of energy disaggregation has expanded so much since I started my PhD in 2010, with new companies and research groups joining the field nearly every week. As a result, it's becoming increasingly difficult to keep track of who is working in this space, even when those people are working in your own country. The NILM workshops are starting to address this by bringing the international community together in Austin, Texas, although I'm sure the inherent cost of travel will prohibit some people attending the workshop.

For this reason, I've been talking with Peter Davies of Green Running, and we are keen to organise an event to provide the opportunity to meet other people working in the field of energy disaggregation in the UK. It will likely be 1 day event held this July in London, free for anyone to attend, and feature a range of presentations and demos, as well as many opportunities for networking. We have a location in central London already to host the event which should hopefully encourage attendees.

Please let us know via email (osp@ecs.soton.ac.uk or p.davies@greenrunning.com) if you would be interested in attending this event, and also feel free to pass on this information on to anyone else who might be interested.

NILM 2014 workshop schedule released

The schedule for the NILM 2014 workshop in Austin has just been released. The workshop will include 3 sessions of paper presentations, a set of lightning talks and poster presentations, an invited keynote from Shankar Sasty, and a demo NILMTK. I'm really excited about the workshop, and look forward to seeing you there!

Training disaggregation algorithms without sub-metered data

I'm keen to include an unsupervised disaggregation algorithm (one that doesn't require appliance data for training) in NILMTK. At the moment, the toolkit only include two supervised benchmark disaggregation algorithms, which I think really limits its usefulness. This post is intended to be a first step towards a simple, intuitive and robust approach to learn models of household appliances using only household aggregate data. I would be really interested in any feedback from the community regarding any improvements or extensions.

Extracting step changes and clustering via a Gaussian mixture model

The approach can be summarised as follows:

1. Extract a set of step changes by taking the differences between sequential aggregate power readings
2. Take the absolute value of these differences such that both positive and negative step changes are identical
3. Discard small step changes (e.g. < 200 W) since there is too much noise at this range to extract any meaningful structure
4. Discard large step changes (e.g. > 3000 W) since these are most likely generated by multiple appliances changing state simultaneously.
5. Cluster the remaining set of step changes using a Gaussian mixture model

Evaluation using data from real households

I applied this approach to data set of aggregate data collected from real households. Unfortunately, the data set does not contain any sub-metered data from such households, so no quantitative results can be provided regarding its accuracy. However, a visual inspection of the extracted step changes and identified clusters shows some encouraging results.

This approach worked very well on some houses, such as the one shown below. The plot shows a black and white histogram of the extracted step changes, in which peaks corresponding to appliances are clearly visible at roughly 1000 W, 1800 W, and 2300 W. The plot also shows coloured probability density functions (PDFs) corresponding to the clusters found. Interestingly, the clustering algorithm successfully finds the three appliances, as shown by the cyan and yellow curves. However, it's also worth noting that many other clusters were found which do not correspond to appliances.

Click to enlarge

There were also households in which no structure was present in the extracted step changes, and as a result none of the clusters correspond to individual appliances, such as the plot shown below. This is likely due to a large amount of measurement noise in the aggregate data, or a number of appliances with highly variable step changes.

Click to enlarge

Conclusions and future work

These experiments have shown that even a very simple model can successfully learn models for appliances using only aggregate data. However, it has also shown that the performance is likely to vary widely between different houses. An important challenge that has not been tackled here is that of labelling identified clusters, e.g. cyan cluster = lighting, red cluster = noise.

Thesis code release

Today I'm releasing the code I wrote for the experiments in my thesis. The code includes an implementation of a Bayesian hidden Markov model, and its application to various appliance modelling tasks. The implementation is in C# and uses the Infer.NET framework for Bayesian inference. The input data I used came from the Tracebase and REDD data sets, and therefore I have not included it in the code release. As a result, I think the BayesianHMM class is probably the most useful to the community.

The code can be found via a link on my publications page.

Paper accepted at NILM 2014

My paper titled 'A Scalable Non-intrusive Load Monitoring System for Fridge-Freezer Energy Efficiency Estimation' was recently accepted for presentation at NILM 2014. The paper gives an overview of chapter 6 of my thesis, which describes a case study deployment of the theory presented in my thesis applied to the disaggregation of fridge/freezers in 117 households in the UK. This involved the collection of a huge amount of aggregate electricity data, for which we used AlertMe current clamps as shown below:

AlertMe current clamp

Here is the full abstract of the paper:

In this paper we propose an approach by which the energy efficiency of individual appliances can be estimated from an aggregate load. To date, energy disaggregation research has presented results for small data sets of 7 households or less, and as a result the generality of results are often unknown. In contrast, we have deployed household electricity sensors to 117 households and evaluated the accuracy by which our approach can identify the energy efficiency of refrigerators and freezers from an aggregate load. Crucially, our approach does not require training data to be collected by sub-metering individual appliances, nor does it assume any knowledge of the appliances present in the household. Instead, our approach uses prior models of general appliance types that are used to first identify which households contain either a combined fridge-freezer or separate refrigerator and freezer, and subsequently to estimate the energy efficiency of such appliances. Finally, we calculate the time until the energy savings of replacing such appliances have offset the cost of the replacement appliance, which we show can be as low as 2.5 years.

Introducing NILMTK: an open source toolkit for non-intrusive load monitoring

Today, Nipun Batra, Jack Kelly and Oliver Parson are really pleased to announce the release of NILMTK: an open source toolkit for non-intrusive load monitoring. The toolkit will allow researchers to easily develop algorithms which disaggregate a household’s total electricity consumption into individual appliances.

Specifically, the toolkit includes:

• a number of parsers to read public data sets into a common format
• a suite of statistical functions to analyse such data sets and identify potential problems
• two benchmark energy disaggregation algorithms
• a suite of evaluation metrics to compare disaggregation algorithms

Further details can be found in the accompanying paper recently accepted at e-Energy 2014 available via arXiv and Soton ePrints:

• Batra, N., Kelly, J., Parson, O., Dutta, H., Knottenbelt, W., Rogers, A., Singh, A., Srivastava, M. (2014). NILMTK: An Open Source Toolkit for Non-intrusive Load Monitoring. In Fifth International Conference on Future Energy Systems (ACM e-Energy). Cambridge, UK. arXiv:1404.3878

This release is hopefully just the beginning of the toolkit’s contribution to energy disaggregation, and as such we welcome feedback and contributions to all aspects of the project.

This has been cross posted via Nipun Batra’s blog, Jack Kelly’s blog and the ORCHID project blog.

Thesis available online

Today I can finally say that the finished version of my thesis has been submitted and is now available online. Here's the full reference:

Parson, Oliver (2014) Unsupervised Training Methods for Non-intrusive Appliance Load Monitoring from Smart Meter Data. University of Southampton, Electronics and Computer Science, Doctoral Thesis.

Here's a Wordle of my thesis:

And also here's the abstract:

Non-intrusive appliance load monitoring (NIALM) is the process of disaggregating a household’s total electricity consumption into its contributing appliances. Smart meters are currently being deployed on national scales, providing a platform to collect aggregate household electricity consumption data. Existing approaches to NIALM require a manual training phase in which either sub-metered appliance data is collected or appliance usage is manually labelled. This training data is used to build models of the house- hold appliances, which are subsequently used to disaggregate the household’s electricity data. Due to the requirement of such a training phase, existing approaches do not scale automatically to the national scales of smart meter data currently being collected.

In this thesis we propose an unsupervised training method which, unlike existing approaches, does not require a manual training phase. Instead, our approach combines general appliance knowledge with just aggregate smart meter data from the household to perform disaggregation. To do so, we address the following three problems: (i) how to generalise the behaviour of multiple appliances of the same type, (ii) how to tune general knowledge of appliances to the specific appliances within a single household using only smart meter data, and (iii) how to provide actionable energy saving advice based on the tuned appliance knowledge.

First, we propose an approach to the appliance generalisation problem, which uses the Tracebase data set to build probabilistic models of household appliances. We take a Bayesian approach to modelling appliances using hidden Markov models, and empirically evaluate the extent to which they generalise to previously unseen appliances through cross validation. We show that learning using multiple appliances vastly outperforms learning from a single appliance by 61–99% when attempting to generalise to a previously unseen appliance, and furthermore that such general models can be learned from only 2–6 appliances.

Second, we propose an unsupervised solution to the model tuning problem, which uses only smart meter data to learn the behaviour of the specific appliances in a given house-hold. Our approach uses general appliance models to extract appliance signatures from a household’s smart meter data, which are then used to refine the general appliance models. We evaluate the benefit of this process using the Reference Energy Disaggregation Data set, and show that the tuned appliance models more accurately represent the energy consumption behaviour of a given household’s appliances compared to when general appliance models are used, and furthermore that such general models can per- form comparably to when sub-metered data is used for model training. We also show that our tuning approach outperforms the current state of the art, which uses a factorial hidden Markov model to tune the general appliance models.

Third, we apply both of these approaches to infer the energy efficiency of refrigerators and freezers in a data set of 117 households. We evaluate the accuracy of our approach, and show that it is able to successfully infer the energy efficiency of combined fridge freezers. We then propose an extension to our model tuning process using factorial hidden semi-Markov models to model households with a separate fridge and freezer. Finally, we show that through this extension our approach is able to simultaneously tune the appliance models of both appliances.

The above contributions provide a solution which satisfies the requirements of a NIALM training method which is both unsupervised (no manual interaction required during training) and uses only smart meter data (no installation of additional hardware is required). When combined, the contributions presented in this thesis represent an advancement in the state of the art in the field of non-intrusive appliance load monitoring, and a step towards increasing the efficiency of energy consumption within households.