infer-hmm: An Infer.NET implementation of the hidden Markov model

The hidden Markov model is a popular statistical tool for modelling sequential data, and as such has received much attention from the field of non-intrusive load monitoring. However, the community has lacked general tools to perform scalable approximate Bayesian inference in HMMs, which has limited the speed of research in this field. For this reason, today I'm open sourcing infer-hmm: An Infer.NET implementation of the hidden Markov model. The aim of the project is to make it easy to run approximate Bayesian inference over both the model parameters and states of a hidden Markov model. The model is built using the Infer.NET framework for Bayesian inference in graphical models, and as such can make use of industry strength algorithms for running approximate inference. Special thanks go to Microsoft Research for adding support for chain models, and to Matteo Venanzi for his expertise in increasing the efficiency of the model.

NILMTK Survey

We've seen some really encouraging adoption of NILMTK (our open-source toolkit for non-intrusive load monitoring) since we started work on it a year ago. However, it's quite hard to keep track of how people our using the toolkit, what features they'd like to see, and what direction the toolkit should be heading in. For this reason, we've created a NILMTK Survey, which will hopefully solve these problems. Please fill out the survey if you have any interest in energy disaggregation research, and let us know what's important to you. Thanks!

Energy disaggregation bloggers

There's a great amount of blogging going on in the NILM research field, so this post aims to collect all these resources together in a single hub. As always, please let me know if I've missed anyone!

Nipun Batra

Nipun is a final year student at IIIT Delhi, who writes about energy disaggregation research, as well as some more general programming solutions he has come across. Nipun also kicked off the NILMTK project.

Mario Bergés

Mario is an assistant professor at CMU, who mostly writes about upcoming workshops and conferences which are highly relevant to NILM. Mario is an incredibly busy person, which somewhat explains why he describes his own blog as having a tendency to remain silent.

Kyle Bradbury

Kyle is a postdoctoral energy fellow at Duke University, who writes about a broad spectrum of energy issues beyond that of just NILM. Kyle is involved with a number of interdisciplinary energy projects at Duke, which draw from disciplines such as engineering, economics, policy, and behavioural science to solve energy problems.

Suman Giri

Suman is one of Mario's students at CMU, who writes about recent developments at their Intelligent Infrastructure Research Laboratory. Suman has done some great work around high-frequency NILM, and has released code for both data collection and disaggregation.

Jack Kelly

Jack is a final year student at Imperial College London, who frequently writes on his energy disaggregation blog. He's very active in the community, and has released a UK data set as well as open sourcing his metadata project. Jack is also a collaborator (and chief architect) on the NILMTK project.

Stephen Makonin

Stephen is a postdoctoral research fellow at Simon Fraser University. Stephen has completed a large amount work in the domain of smart meter data analytics, as well as releasing the AMPds data set.

Work with ITG

I've recently been doing some work with ITG, an IT and management consulting firm based in Washington DC, specialising in Real Estate IT. ITG has also been working in the field of non-intrusive load monitoring and is in the process of building an energy monitoring system. I was happy to provide a program of presentations and discussions to bring them up to speed with my experiences in the field. I really enjoyed the time I spent working with ITG, and hope I have the chance to return in the future.

ITG's office Washington DC, USA

Energy disaggregation at BuildSys 2014

I attended the BuildSys 2014 conference over the last two days, which was held on 5-6 November in Memphis, TN, USA. I was really impressed at the amount of NILM research in the main conference track, including a dedicated session on Thursday morning focusing entirely on energy disaggregation. This prompted some great discussions during the breaks, mostly centred around the potential for a third party evaluation tool and/or a NILM competition, similar to the one organised by Belkin. Both are very difficult problems and are probably worthy of their own blog post, which I'll hopefully write in the next few days.

I presented a demo of NILMTK v0.2, our latest release of the open source toolkit, which adds support for data sets which are too large to fit into memory, as well as providing support for a common data set schema via the NILM Metadata project. The demo was really well received, and ended up winning the best demo award. I presented the demo using an iPython notebook, similar to the demo that Jack Kelly presented at the NILM 2014 conference, which is available for all to view at the following URL:

http://nbviewer.ipython.org/github/nilmtk/nilmtk/blob/master/notebooks/BuildSys_2014_demo.ipynb

Update (12.11.2014): Jack Kelly has written an excellent proposal for an MSc group project covering the range of issues which a third party validation tool / disaggregation competition must address.

List of NILM conferences

This post aims to collect a number of NILM-specific workshops and conferences which have taken place over the past few years. As always, please feel free to suggest any I might have missed, and I'll do my best to keep this post updated.

NILM 2012

The first international workshop on non-intrusive load monitoring was organised by Mario Bergés and Zico Kolter, and took place on 7 May 2012 at Carnegie Mellon University, Pittsburgh, PA, USA. The workshop attracted roughly 50 attendees from academia, industry, government and not for profits. The workshop agenda followed a traditional academic style, in which accepted 2 page extended abstracts were presented across 4 sessions. The workshop also included a poster session and a session of presentations from 3 NILM companies.

EPRI NILM 2013

The Electrical Power Research Institute hosted a workshop on non-intrusive load monitoring on 12-13 November 2014 in Palo Alto, CA, USA. The workshop was organised by Chris Holmes, Cody Taylor, Krish Gomatom and Ashley Kelley-Cox, and consisted of a mixture of presentations and discussion sessions on high-level topics such as standardisation and use cases. The event brought together roughly 40 attendees from mostly utilities and NILM companies, although the workshop was also attended by a few academics. Since the workshop consisted of invited talks and discussions sessions, there were no formal proceedings.

NILM 2014

The second international workshop on non-intrusive load monitoring was again organised by Mario Bergés and Zico Kolter, and took place on 3 June 2014 at the University of Texas, Austin, USA. The workshop attracted roughly 100 attendees with an interest in NILM from a wide range of sectors. Proceedings of the workshop consisted of peer reviewed 4 page papers, which were presented as either 15 minute talks or 1 minute lightning talks. The agenda followed a less dense format than that of NILM 2012, with more time allocated to coffee breaks and networking. In addition to the poster session, the workshop also included a keynote talk from Shankar Sastry and a demo of NILMTK by Jack Kelly. The workshop was co-located with the International WikiEnergy Data Conference, which was held on 4 June.

EU NILM 2014

The first European NILM workshop was organised by Oliver Parson, Peter Davies and Jack Kelly, and took place on 3 September 2014 at Imperial College London, UK. The workshop brought together roughly 20 academics and industry experts for a combination of high level talks and discussion sessions, for which the slides are available online. The meet up concluded that there is sufficient demand for a regular European NILM venue, the next of which will likely take place in March 2015 in London. The meet up was followed by a NILMTK masterclass given by Jack Kelly.

EU NILM 2015

The second European NILM workshop was also organised by Oliver Parson, Peter Davies and Jack Kelly, and took place on 8 July 2015 at Imperial College London, UK. The workshop was attended by about 65 people from academia and a range of disaggregation companies, and was also streamed live online. Videos of all talks are available as a playlist on YouTube.

EPRI NILM 2015

The Electric Power Research Institute (EPRI) hosted a NILM workshop on 13 November 2015 in Orlando, FL, and was attended by disaggregation vendors, utilities, universities, a U.S. Department of Energy National Lab and research organisation. The workshop covered topics such as current EPRI research, use cases, utility and consultant experiences with NILM and product labelling (see full slide deck). One of the key outcomes of workshop was the recognition of gaps related to NILM metrics and how to address them as an industry through collaborative efforts.

NILM 2016

The third international workshop on non-intrusive load monitoring was organised by Stephen Makonin, and will take place on 14-15 May 2016 at the Simon Fraser University, Vancouver, Canada. Proceedings of the workshop consisted of peer reviewed 4 page papers, which were presented as either 15 minute talks or 1 minute lightning talks. Keynote talks will be given by Lyn Bartram and George Hart.

Post NILM 2014 @ London

On Wednesday 3 September we held the NILM 2014 @ London meet up at Imperial College London. The aim of the meet up was to provide a local forum for European energy disaggregation researchers to get to know each other and discuss their work. The agenda included introductions of the attendees, an overview of NILM events to date, a presentation of energy disaggregation at DNO level, an overview of the NILMTK project, as well as many discussion around data collection and evaluation. The meet up was attended by representatives from AlertMe, British Gas, Green Running, Greeniant, Navetas, ONZO and Wattgo, as well as academics from Imperial College London, University of Klagenfurt and University of Southampton. Slides from the presentations are now online via the meet up website, while photos from the day are available via Jack Kelly's flickr album.

It became clear from the meet up that there's real enthusiasm for a regular NILM event hosted in Europe. Currently, we're planning to host the next event in London around February-March 2015, so please get in touch if you'd like to attend.

Special thanks to Peter Davies of Green Running for sponsoring the event and Jack Kelly of Imperial College London for the local organisation.

NILM 2014 @ London update

With 1 week to go until the meet up, I wanted to give an update on the recent developments. We currently have attendees signed up from 7 companies and 4 academic institutions, and will continue to accept registrations up until the day before the event. The location for the meet up has also been confirmed as a lecture room at Imperial College London. Finally, we'll likely hold a NILMTK hackday the following day (Thursday 4th September 2014) for anyone who is interested.

I look forward to seeing you there!

50 day access to my Artificial Intelligence article

The published version of my recently accepted paper in Artificial Intelligence is now available online for free for 50 days. However, after the 4 October 2014, access to the published version will require a subscription to the journal, although a pre-print of the article will always be available via the University of Southampton ePrints repository.

NILM 2014 @ London, UK

Peter Davies and I would like to invite you to a meetup on the topic of non-intrusive load monitoring, to be hosted in central London on 3 September 2014. The purpose of the workshop is to provide a forum to share common research goals and identify common themes for collaboration. The workshop will be free to attend, although registration via the meetup web page is required. Further details can be found at:

http://www.oliverparson.co.uk/nilm-2014-london

We look forward to seeing you there!

Paper accepted to Journal of Artificial Intelligence

Last week we got the notification that our paper had been accepted to appear in the Journal of Artificial Intelligence. The paper describes a method for building generalisable appliance models from existing data sets, and also a method which tunes such models for previously unseen households using only aggregate smart meter data. This material also appeared in chapters 4 and 5 of my thesis.

The final reference for the article is:

Oliver Parson, Siddhartha Ghosh, Mark Weal, Alex Rogers. (2014). An Unsupervised Training Method for Non-intrusive Appliance Load Monitoring. In: Artificial Intelligence, 217, 1–19.

and the abstract is:

Non-intrusive appliance load monitoring is the process of disaggregating a household's total electricity consumption into its contributing appliances. In this paper we propose an unsupervised training method for non-intrusive monitoring which, unlike existing supervised approaches, does not require training data to be collected by sub-metering individual appliances, nor does it require appliances to be manually labelled for the households in which disaggregation is performed. Instead, we propose an approach which combines a one-off supervised learning process over existing labelled appliance data sets, with an unsupervised learning method over unlabelled household aggregate data. First, we propose an approach which uses the Tracebase data set to build probabilistic appliance models which generalise to previously unseen households, which we empirically evaluate through cross validation. Second, we use the Reference Energy Disaggregation Data set to evaluate the accuracy with which these general models can be tuned to the appliances within a specific household using only aggregate data. Our empirical evaluation demonstrates that general appliance models can be constructed using data from only a small number of appliances (typically 3-6 appliances), and furthermore that 28-99% of the remaining behaviour which is specific to a single household can be learned using only aggregate data from existing smart meters.

PhD Graduation

This week I finally graduated from my PhD, nearly four years after I started my work on energy disaggregation (and this blog) back in October 2010. It was cool to finally get closure on the whole process, and celebrate our achievements along with four other PhD students from our lab. Fortunately, I'm still working in the field of non-intrusive load monitoring, so this blog will remain active for at least the next six months. In case you missed it, here's a link to my thesis, which I already feel like was finished a long time ago!

Workshop on Human Centric Energy Management

Last week I attended a workshop on human-centric energy management organised by Dominik Egarter, Wilfried Elmenreich and Martin Krch in Klagenfurt, Austria. The schedule featured a series of keynote talks, discussions, and a trip to Kraftwerk Forstsee.

I presented some work from my group on disaggregating a home's fridge-freezer energy consumption from smart meter data, and also some work on providing home heating feedback via the Joulo project. My full presentation slides are available via my website.

The workshop also included talks from Wilfried Elmenreich on work from their group aiming to bring the smart grid into people's homes, and also Andreas Reinhardt on the opportunities for novel services based on appliance-level power consumption monitoring.

I really want to thank the organisers for inviting me to attend such a great workshop. I'm particularly excited about the potential for collaborations within NILMTK, and hope it leads to some interesting joint projects.

WikiEnergy Data Set Statistics

I recently wrote a post about the WikiEnergy data set released by Pecan Street Inc, and have since written a downloader and converter for the data set as part of NILMTK. In total, the data set contains 71 feed feeds monitored across 239 buildings over the period 1 Jan 2014 - 31 May 2014. However, only a subset of feeds were monitored for each building, and many buildings were not monitored for the full 5 months. This post provides a bit more insight into the content of the data set at the time of writing.

Feeds per building:

The histogram below shows the number of feeds monitored in each of the 239 buildings. It can be seen that the mode of the distribution is around 12 feeds per building, and therefore most of these buildings will be useful for evaluation energy disaggregation approaches.

Duration per building:

The histogram below shows the number of months for which each of the 239 buildings were monitored. It can be seen that vast majority of buildings were monitored for the full 5 months, while the remaining buildings were distributed between 1-4 months. However, this distribution will change dramatically once data for 2012-2013 is released.

Percentage energy sub-metered:

The histogram below shows the percentage of energy sub-metered in 235 of the 239 buildings. The remaining 4 buildings appeared to have energy sub-metered greater than 100%, and were therefore excluded from this plot. This distribution has two distinct peaks; one centred around 70% and another which peaks around 5%. The 63 buildings for which less than 40% of the energy was sub-metered are likely to be of limited use for evaluating energy disaggregation methods.

Buildings per feed:

The table below shows the number of buildings in which each of the 71 feeds were present. A description of each of the feeds is available from the Wiki-Energy Knowledge Base. It can be seen that the presence of feeds in buildings is quite sparse. However, the following feeds are present in the majority of buildings: the household aggregate power (use), air conditioning (air1), washing machine (clotheswasher1), dishwasher (dishwasher1), clothes dryer (drye1), electric heating (furnace1) and refrigerator (refrigerator1).

Feed	Buildings
use	239
air1	224
air2	38
air3	5
airwindowunit1	3
aquarium1	1
bathroom1	57
bathroom2	7
bedroom1	65
bedroom2	30
bedroom3	4
bedroom4	0
bedroom5	0
car1	62
clotheswasher1	133
clotheswasher_dryg1	28
diningroom1	20
diningroom2	1
dishwasher1	150
disposal1	85
drye1	141
dryg1	29
freezer1	13
furnace1	184
furnace2	29
garage1	25
garage2	3
gen	116
grid	0
heater1	2
housefan1	2
icemaker1	1
jacuzzi1	13
kitchen1	46
kitchen2	17
kitchenapp1	103
kitchenapp2	73
lights_plugs1	79
lights_plugs2	40
lights_plugs3	16
lights_plugs4	4
lights_plugs5	2
lights_plugs6	0
livingroom1	64
livingroom2	10
microwave1	113
office1	31
outsidelights_plugs1	16
outsidelights_plugs2	3
oven1	89
oven2	3
pool1	4
pool2	0
poollight1	2
poolpump1	17
pump1	3
range1	61
refrigerator1	164
refrigerator2	14
security1	7
shed1	3
sprinkler1	9
unknown1	16
unknown2	6
unknown3	1
unknown4	1
utilityroom1	5
venthood1	19
waterheater1	21
waterheater2	2
winecooler1	4

Hart, G.W., Prototype Nonintrusive Appliance Load Monitor, 1985

Since Hart founded the field of energy disaggregation back in the '80s, most papers since have cited his 1992 summary article published in the Proceedings of the IEEE. However, I've seen many references to other papers but have rarely managed to get my hands on the full text. For this reason, I was particularly excited when the following technical report surfaced recently:

Hart, G.W., Prototype Nonintrusive Appliance Load Monitor, MIT Energy Laboratory Technical Report, and Electric Power Research Institute Technical Report, September 1985

Apparently, Hart had received a few requests for older papers over the years, which so far he'd been unable to locate. However, he recently came across a copy of the above paper in the online catalog of a library in Singapore. Mario Bergés then requested the paper via an inter-library loan, Suman Giri scanned the paper copy and Simon Leigh (Jack Kelly's MSc student) applied some post-processing to provide you with the beautiful copy you see today. Quite a good community effort in my opinion!

New data sets released by WikiEnergy and University of California, Berkley

I've recently come across two new data sets which have been released in the past month:

WikiEnergy data

Pecan Street Inc have released a large amount of domestic electricity data via the WikiEnergy project. The data set currently contains data from 200 homes, in which both the household aggregate power demand and individual appliance power demands are monitored at 1 minute intervals. The data set currently contains 4 months of data from January-April 2014, although more data is likely to be released soon. The data is freely available to University members of the WikiEnergy community, and full details for database access can be found on the WikiEnergy Knowledge Base after registering.

BERDS - BERkeley EneRgy Disaggregation Data Set

The University of California, Berkley, have released electricity data collected from the Cory Hall on the UC Berkeley campus. The data set contains data collected from 4 categories of sub-metered loads: lighting, HVAC, receptacle (sockets) and other, for which many feeds are available for each load category. The data set contains measurements of active, reactive and apparent power which were collected at 20 second intervals. The data is available for free via Mehdi Maasoumy's website, and a paper briefly describing the data set appeared at the Big Learning workshop at NIPS 2013.

I've updated my blog post of publicly available data sets to include both of these releases.

Attending NILM 2014

I'm really excited to be travelling to Austin tomorrow to attend NILM 2014. I'm particularly looking forward to meeting other people working in the field, so please come and introduce yourself if you're also attending! I've been involved in two papers which will be presented at the workshop:

1. A Scalable Non-intrusive Load Monitoring System for Fridge-Freezer Energy Efficiency Estimation. Oliver Parson, Mark Weal, Alex Rogers. This paper summarises the final chapter of my thesis, which covers a large scale application of my work to 117 UK households. I'll be presenting this work in the 'new perspectives' afternoon session of the workshop, and also as a poster in the poster session at the end of the workshop.
2. NILMTK: An Open Source Toolkit for Non-intrusive Load Monitoring. Nipun Batra, Jack Kelly, Oliver Parson, Haimonti Dutta, William Knottenbelt, Alex Rogers, Amarjeet Singh, Mani Srivastava. This paper summarises the initial release of NILMTK; an open source toolkit for energy disaggregation research. This will be presented by Jack Kelly as a 30 minute demo in the final session of the workshop, and also as a poster in the following poster session.

I'll also be attending the Pecan Street WikiEnergy Data conference on the 4th of June, which also promises a very exciting list of speakers.

See you all in Texas!

UK energy disaggregation meet-up

The field of energy disaggregation has expanded so much since I started my PhD in 2010, with new companies and research groups joining the field nearly every week. As a result, it's becoming increasingly difficult to keep track of who is working in this space, even when those people are working in your own country. The NILM workshops are starting to address this by bringing the international community together in Austin, Texas, although I'm sure the inherent cost of travel will prohibit some people attending the workshop.

For this reason, I've been talking with Peter Davies of Green Running, and we are keen to organise an event to provide the opportunity to meet other people working in the field of energy disaggregation in the UK. It will likely be 1 day event held this July in London, free for anyone to attend, and feature a range of presentations and demos, as well as many opportunities for networking. We have a location in central London already to host the event which should hopefully encourage attendees.

Please let us know via email (osp@ecs.soton.ac.uk or p.davies@greenrunning.com) if you would be interested in attending this event, and also feel free to pass on this information on to anyone else who might be interested.

NILM 2014 workshop schedule released

The schedule for the NILM 2014 workshop in Austin has just been released. The workshop will include 3 sessions of paper presentations, a set of lightning talks and poster presentations, an invited keynote from Shankar Sasty, and a demo NILMTK. I'm really excited about the workshop, and look forward to seeing you there!

Training disaggregation algorithms without sub-metered data

I'm keen to include an unsupervised disaggregation algorithm (one that doesn't require appliance data for training) in NILMTK. At the moment, the toolkit only include two supervised benchmark disaggregation algorithms, which I think really limits its usefulness. This post is intended to be a first step towards a simple, intuitive and robust approach to learn models of household appliances using only household aggregate data. I would be really interested in any feedback from the community regarding any improvements or extensions.

Extracting step changes and clustering via a Gaussian mixture model

The approach can be summarised as follows:

1. Extract a set of step changes by taking the differences between sequential aggregate power readings
2. Take the absolute value of these differences such that both positive and negative step changes are identical
3. Discard small step changes (e.g. < 200 W) since there is too much noise at this range to extract any meaningful structure
4. Discard large step changes (e.g. > 3000 W) since these are most likely generated by multiple appliances changing state simultaneously.
5. Cluster the remaining set of step changes using a Gaussian mixture model

Evaluation using data from real households

I applied this approach to data set of aggregate data collected from real households. Unfortunately, the data set does not contain any sub-metered data from such households, so no quantitative results can be provided regarding its accuracy. However, a visual inspection of the extracted step changes and identified clusters shows some encouraging results.

This approach worked very well on some houses, such as the one shown below. The plot shows a black and white histogram of the extracted step changes, in which peaks corresponding to appliances are clearly visible at roughly 1000 W, 1800 W, and 2300 W. The plot also shows coloured probability density functions (PDFs) corresponding to the clusters found. Interestingly, the clustering algorithm successfully finds the three appliances, as shown by the cyan and yellow curves. However, it's also worth noting that many other clusters were found which do not correspond to appliances.

Click to enlarge

There were also households in which no structure was present in the extracted step changes, and as a result none of the clusters correspond to individual appliances, such as the plot shown below. This is likely due to a large amount of measurement noise in the aggregate data, or a number of appliances with highly variable step changes.

Click to enlarge

Conclusions and future work

These experiments have shown that even a very simple model can successfully learn models for appliances using only aggregate data. However, it has also shown that the performance is likely to vary widely between different houses. An important challenge that has not been tackled here is that of labelling identified clusters, e.g. cyan cluster = lighting, red cluster = noise.

Thesis code release

Today I'm releasing the code I wrote for the experiments in my thesis. The code includes an implementation of a Bayesian hidden Markov model, and its application to various appliance modelling tasks. The implementation is in C# and uses the Infer.NET framework for Bayesian inference. The input data I used came from the Tracebase and REDD data sets, and therefore I have not included it in the code release. As a result, I think the BayesianHMM class is probably the most useful to the community.

The code can be found via a link on my publications page.

Paper accepted at NILM 2014

My paper titled 'A Scalable Non-intrusive Load Monitoring System for Fridge-Freezer Energy Efficiency Estimation' was recently accepted for presentation at NILM 2014. The paper gives an overview of chapter 6 of my thesis, which describes a case study deployment of the theory presented in my thesis applied to the disaggregation of fridge/freezers in 117 households in the UK. This involved the collection of a huge amount of aggregate electricity data, for which we used AlertMe current clamps as shown below:

AlertMe current clamp

Here is the full abstract of the paper:

In this paper we propose an approach by which the energy efficiency of individual appliances can be estimated from an aggregate load. To date, energy disaggregation research has presented results for small data sets of 7 households or less, and as a result the generality of results are often unknown. In contrast, we have deployed household electricity sensors to 117 households and evaluated the accuracy by which our approach can identify the energy efficiency of refrigerators and freezers from an aggregate load. Crucially, our approach does not require training data to be collected by sub-metering individual appliances, nor does it assume any knowledge of the appliances present in the household. Instead, our approach uses prior models of general appliance types that are used to first identify which households contain either a combined fridge-freezer or separate refrigerator and freezer, and subsequently to estimate the energy efficiency of such appliances. Finally, we calculate the time until the energy savings of replacing such appliances have offset the cost of the replacement appliance, which we show can be as low as 2.5 years.

Introducing NILMTK: an open source toolkit for non-intrusive load monitoring

Today, Nipun Batra, Jack Kelly and Oliver Parson are really pleased to announce the release of NILMTK: an open source toolkit for non-intrusive load monitoring. The toolkit will allow researchers to easily develop algorithms which disaggregate a household’s total electricity consumption into individual appliances.

Specifically, the toolkit includes:

• a number of parsers to read public data sets into a common format
• a suite of statistical functions to analyse such data sets and identify potential problems
• two benchmark energy disaggregation algorithms
• a suite of evaluation metrics to compare disaggregation algorithms

Further details can be found in the accompanying paper recently accepted at e-Energy 2014 available via arXiv and Soton ePrints:

• Batra, N., Kelly, J., Parson, O., Dutta, H., Knottenbelt, W., Rogers, A., Singh, A., Srivastava, M. (2014). NILMTK: An Open Source Toolkit for Non-intrusive Load Monitoring. In Fifth International Conference on Future Energy Systems (ACM e-Energy). Cambridge, UK. arXiv:1404.3878

This release is hopefully just the beginning of the toolkit’s contribution to energy disaggregation, and as such we welcome feedback and contributions to all aspects of the project.

This has been cross posted via Nipun Batra’s blog, Jack Kelly’s blog and the ORCHID project blog.

Thesis available online

Today I can finally say that the finished version of my thesis has been submitted and is now available online. Here's the full reference:

Parson, Oliver (2014) Unsupervised Training Methods for Non-intrusive Appliance Load Monitoring from Smart Meter Data. University of Southampton, Electronics and Computer Science, Doctoral Thesis.

Here's a Wordle of my thesis:

And also here's the abstract:

Non-intrusive appliance load monitoring (NIALM) is the process of disaggregating a household’s total electricity consumption into its contributing appliances. Smart meters are currently being deployed on national scales, providing a platform to collect aggregate household electricity consumption data. Existing approaches to NIALM require a manual training phase in which either sub-metered appliance data is collected or appliance usage is manually labelled. This training data is used to build models of the house- hold appliances, which are subsequently used to disaggregate the household’s electricity data. Due to the requirement of such a training phase, existing approaches do not scale automatically to the national scales of smart meter data currently being collected.

In this thesis we propose an unsupervised training method which, unlike existing approaches, does not require a manual training phase. Instead, our approach combines general appliance knowledge with just aggregate smart meter data from the household to perform disaggregation. To do so, we address the following three problems: (i) how to generalise the behaviour of multiple appliances of the same type, (ii) how to tune general knowledge of appliances to the specific appliances within a single household using only smart meter data, and (iii) how to provide actionable energy saving advice based on the tuned appliance knowledge.

First, we propose an approach to the appliance generalisation problem, which uses the Tracebase data set to build probabilistic models of household appliances. We take a Bayesian approach to modelling appliances using hidden Markov models, and empirically evaluate the extent to which they generalise to previously unseen appliances through cross validation. We show that learning using multiple appliances vastly outperforms learning from a single appliance by 61–99% when attempting to generalise to a previously unseen appliance, and furthermore that such general models can be learned from only 2–6 appliances.

Second, we propose an unsupervised solution to the model tuning problem, which uses only smart meter data to learn the behaviour of the specific appliances in a given house-hold. Our approach uses general appliance models to extract appliance signatures from a household’s smart meter data, which are then used to refine the general appliance models. We evaluate the benefit of this process using the Reference Energy Disaggregation Data set, and show that the tuned appliance models more accurately represent the energy consumption behaviour of a given household’s appliances compared to when general appliance models are used, and furthermore that such general models can per- form comparably to when sub-metered data is used for model training. We also show that our tuning approach outperforms the current state of the art, which uses a factorial hidden Markov model to tune the general appliance models.

Third, we apply both of these approaches to infer the energy efficiency of refrigerators and freezers in a data set of 117 households. We evaluate the accuracy of our approach, and show that it is able to successfully infer the energy efficiency of combined fridge freezers. We then propose an extension to our model tuning process using factorial hidden semi-Markov models to model households with a separate fridge and freezer. Finally, we show that through this extension our approach is able to simultaneously tune the appliance models of both appliances.

The above contributions provide a solution which satisfies the requirements of a NIALM training method which is both unsupervised (no manual interaction required during training) and uses only smart meter data (no installation of additional hardware is required). When combined, the contributions presented in this thesis represent an advancement in the state of the art in the field of non-intrusive appliance load monitoring, and a step towards increasing the efficiency of energy consumption within households.

International WikiEnergy Data Conference 2014

The WikiEnergy Data Conference has just been announced, and will be co-hosted by Carnegie Mellon University and Pecan Street Research Institute. The conference will feature presentations from computer science, public policy, and engineering graduate students selected as finalists for the Pike Powers energy research fellowship.

Some important information:

• When: 4-5 June 2014
• Austin, TX, USA
• Objective: To convene the WikiEnergy researcher community, highlighting the results of research conducted by graduate students selected as finalists for the Pike Powers Energy Research Fellowship.

The first day of the workshop will be dedicated to the Pike Powers Fellowships, while the second day is scheduled as an industry day. Attendance is free, although attendees should  register via the conference website. The workshop will also be co-located with the NILM 2014 workshop on 3 June.

Visit to Telekom Malaysia R&D

I’ve just returned from a visit to Telekom Malaysia Research & Development during which I delivered a 5 day course on non-intrusive appliance load monitoring. The course aimed to provide a broad overview of the range of research happening in both academia and industry as well as an in-depth description of the current state of the art. Below is a photo of the group taken outside their research facility:

With a roughly 50/50 split between presentations and practical exercises, the course covered the following areas:

• Introduction & history
• Data sets & preprocessing
• Event based & non-event based methods
• Data sets & accuracy metrics
• Disaggregation exercise using factorial hidden Markov models

I really feel that although many of the attendees started the course with little or no background in energy disaggregation, everyone left the course with a broad understanding and appreciation of the successes and limitations within the field. It was truly a pleasure to be invited by TM R&D and I hope they are able to invest their newly found knowledge in NIALM into practical solutions.

If you’re interested in a similar course for your company, please feel free to drop me an email.

GridCarbon Android app v2 released

I'm very happy to announce the release of an update to the GridCarbon Android app. The app allows you to track the carbon intensity of the UK electricity grid on your phone or tablet.

The demand for electricity in the UK varies throughout the day, and thus, the mix of generators supplying this electricity continually changes. As a result, the carbon intensity of the electricity – the quantity of CO2 produced for 1 kWh of electricity consumed – also varies continually. Deferring your use of electricity to off-peak times, when the carbon intensity is low, can help reduce your carbon footprint.

The app features a breakdown of the the current electricity generation sources:

The landscape view shows how both the carbon intensity of the grid and the generation sources varied over the previous 24 hours:

There is also an iOS version of GridCarbon which shares the same functionality as the Android app.

NILM in the New Scientist

A couple of weeks ago, Google acquired Nest, a smart thermostat manufacturer. The Nest thermostat aims to combine manually set temperature preferences with automatically learned household occupancy schedules. The acquisition seemed to prompt a few privacy concerns, given the unification of household occupancy data with the information Google currently stores about its customers. This is where NILM got its mention in the New Scientist, given that private information can also be inferred from household aggregate electricity data. A summary article even referenced research which showed that the film being watched on a plasma TV can be inferred from 2 Hz smart meter data, assuming two 5 minute periods of data can be extracted when no other appliances are changing state. Unfortunately though, none of these articles seemed to mention that such methods would require higher frequency data than will be automatically uploaded to utilities by smart meters, and therefore would require explicit consent from the customer to opt-in to such a system.

NILM 2014: Second International Workshop on Non-intrusive Load Monitoring

The Second International Workshop on Non-intrusive Load Monitoring is being organised by Mario Bergés and Zico Kolter in partnership with Pecan Street. The workshop is a follow up to the 2012 NILM Workshop held in Pittsburgh, which brought academics and vendors interested in energy disaggregation together for the first time. Full information about the upcoming workshop can be found at nilmworkshop.org.

Some important information:

• When: 3 June 2014
• Where: Austin, TX, USA
• Objective: To review the main types of approaches that have been explored to date to solve the problem of electricity disaggregation, and to then discuss possible paths forward knowing what has been tried and what has yet to be experimented.

The workshop will feature talks from invited speakers, paper presentations and a poster session. Attendees are be able to register for the workshop at nilmworkshop.org. Authors will also be able to submit papers via the same website, for which submission system is live and will close on 28 March 2014. The workshop will also be co-located with a two day workshop hosted by Pecan Street on 4-5 June.