Reusable hold out test sets for NILM

Overfitting is a well-cited problem in the field of Non-intrusive Load Monitoring. Overfitting refers to the high accuracy of an algorithm on one small data set, while the same algorithm generalises poorly to other data sets. In NILM, this often corresponds to algorithms which work well on data covering a short period of time or a small number of houses, but performs poorly on data covering longer time periods or other houses.

One solution to this problem is through the use of a competition, in which the organiser releases part of the data set for training, while holding out the remaining data for testing. This approach works fine for one-off competitions where each participant can only submit one solution, but it weakens when participants are allowed to submit multiple entries. The reason for this is that participants can use information learned from their accuracy score on the test set to inform their algorithm choice.

Belkin organised exactly this type of competition for NILM via the Kaggle platform a few years ago. The competition allowed two entries per day per participant, each of were evaluated using half of a private hold out data set and displayed on a public leaderboard. However, the competition ended after 4 months, at which point the final standings were determined by each solution's performance on the other half of the private hold out data set which had never previously been released. Through such a format, the final standings can only be calculated once and the competition cannot be re-run, as the final standings convey information about the private hold out data set which could inform the design of future algorithms.

A recent Google Research blog post describes this exact problem in a much more general sense than NILM. Most interestingly, their recent paper even proposes a solution to this problem through the use of a reusable holdout set. The approach is that the reusable hold out set is only accessed through a differentially private algorithm, which effectively samples the holdout set in order to produce a different sample each time it is accessed.

At the 2015 European NILM workshop, an MSc group from Imperial supervised by Jack Kelly presented a platform which could potentially be used to host NILM competitions in the future. I’d be really interested to see whether such a platform could use such a reusable hold out set in order to allow the competition to run for much longer without compromising the results relative to the classical method of evaluation.

NILM 2015 Workshop Summary

During July 2015, Imperial College London played host to nearly 70 attendees from all over the world for the European Non-Intrusive Load Monitoring (NILM) workshop, bringing together energy disaggregation researchers and professionals for this 2-day event.

The purpose of the workshop was to provide a forum for energy disaggregation enthusiasts to learn about recent developments in the field, as well as network and discuss projects for future collaboration. The workshop was attended by academics, employees of energy disaggregation companies, multinational utility companies and a few hobbyists.

Prof Mario Bergés, Assistant Professor at Carnegie Mellon University gave the keynote speech which focussed on the relevance of NILM within the emerging domain of the Internet of Things (IoT). Mario covered recent trends in energy disaggregation, as well as his projection of the field’s relevance into the future. His talk also proposed four ‘million dollar’ ideas which he believes will have significant impact on the domain of energy disaggregation. You can watch Mario’s full talk below.

Workshop attendees also enjoyed talks from both academic and industry aspects, with speakers including Mingjun Zhong from the University of Edinburgh and Stephen Makonin from Simon Fraser University representing academic findings, and focussed on models for energy disaggregation, socioeconomic concerns and accuracy evaluation. From an industry perspective the workshop welcomed Bruno Charbonnier from EDF R&D, and Hjalmar Nilsonne from Watty who cemented the importance and benefits of disaggregating electricity and announced the release of a new dataset.

Delegates were invited to bring a poster to present at a dedicated ‘lightning talk’ session, giving each presenter a chance to disseminate and discuss a NILM related topic of their choice for 1 minute. During the lunch and break sessions, posters were displayed on the walls, giving the presenter an opportunity to engage in one-to-one discussions with other attendees. The posters are available via a dropbox folder, while the lightning talk session can be watched below.

An MSc group from Imperial College London presented a tool for evaluating NILM algorithms without requiring the NILM algorithm’s code to be released. There was a real buzz of excitement around such an initiative, and a number of improvements were suggested around the need for a real-world private data set.

As the NILM Workshop came to a close, an agenda was decided for topics to discuss the following morning at an informal user group designed to encourage collaboration and potential projects among attendees, which included funding applications and data sharing. The afternoon session explored NILMTK; an open source toolkit for non-intrusive load monitoring and included an overview of the toolkit as well as discussions on how to encourage contributions from the community. The need for a collaborative knowledge base, where items such as public data sets can be described in an easily comparable and searchable format was also discussed, with the result being a web based wiki which will be available on the nilm.eu website soon.

The most obvious learning from the workshop was the increasing momentum in this domain. The 2014 European Workshop was attended by around 20 people while this year saw nearly 70 attendees from around the globe. In addition, the diversity of the problems being studied by each of the attendees was also clear from the poster session, as each start up or academic project has a subtle but significantly different take on the problem of energy disaggregation. Lastly, the problem of evaluation accuracy cropped up regularly throughout the workshop. The need for standard data sets, metrics and methodologies is now more important than ever.

The workshop was streamed live on YouTube, and videos of all talks can be seen via our YouTube playlist, while each presenter's slides can be downloaded from our dropbox folder.

The two day workshop finished with a discussion of plans for the 2016 European workshop. While the location of the workshop is yet to be decided, it was clear that there was sufficient demand for a future meeting. We will soon be announcing a call for hosts, with the aim of hosting the workshop in a city with convenient international transport links with the rest of Europe.

NILM 2015 presentation videos

In case you missed the live stream of the Second European NILM Workshop, we've also uploaded each talk to a YouTube playlist.

Oliver Parson, University of Southampton - Introduction

Mario Bergés, CMU - NILM in the era of IoT

A series of 1-minute lightning talks by each poster presenter

Stephen Makonin, Simon Fraser University - From socioeconomic concerns to standardising accuracy to water NILM

Mingjun Zhong, University of Edinburgh - Incorporating long-term and population-level information into Machine-Learning based NILM

Imperial MSc Group - green gauge: Comparing Algorithms for Energy Disaggregation

Bruno Charbonnier, EDF - Decortic: A method for detecting and estimating the consumption of electrical space heating

Hjalmar Nilsonne - Connecting the world’s energy data

NILM 2015 Live Stream

As the upcoming European NILM Workshop is now fully booked, we're also hoping to stream the presentation sessions via a Hangout On Air. The link to the Hangout On Air event is:
https://plus.google.com/events/chkt7cmig57pp6n9hfbv1dboas4

The agenda for the day is as follows:

• 10.00 Welcome and workshop overview - Oliver Parson
• 10.30 Keynote talk - Mario Bergés
• 11.30 Poster lightning talks - All poster presenters
• 12.00 Lunch & poster session (not streamed)
• 13:40 Academic talks - Mingun Zhong & Stephen Makonin
• 14.30 NILM evaluation tool - Imperial MSc group
• 15.00 Coffee & networking (not streamed)
• 15.40 Industry talks - Bruno Charbonnier & Hjalmar Nilsonne
• 16.30 Roundup and discussion for future workshops
• 17.00 Stream ends

Please note all times are local London time (BST = GMT+1).

We might need to restart the Hangout if we encounter technical problems on the day, so please keep an eye on NILM_Workshop on twitter for links to new streams.

Videos of the talks should be available on YouTube shortly after the event so long as everything goes smoothly on the day.

We have also set up a LinkedIn group to allow people to introduce themselves, continue workshop discussions, or catch up on anything they've missed.

What even is supervised/unsupervised disaggregation?

I've noticed a fair amount of disagreement regarding exactly what type of learning is being used by a specific energy disaggregation method. I think the confusion arises from a discrepancy between the definition of supervised learning in the general machine learning literature and the practical assumptions of energy disaggregation methods:

Machine learning definition

General purpose machine learning defines supervised learning methods as those which require labelled training data to train a model. Labelled data refers to both the input and answers to the problem, which in the case of energy disaggregation corresponds to both household aggregate and individual appliance energy consumption. Conversely, unsupervised learning refers the use of only unlabelled data (household-level) data to construct models.

Practical energy disaggregation

In the energy disaggregation field, a fundamental problem exists due to the variation in appliances between different houses. As a result, scalable methods must not require appliance-level data from the houses in which disaggregation is to be performed (test houses). As such, practical approaches can apply supervised learning to appliance-level data from houses other than the test house, but can only apply unsupervised learning to aggregate-level data from the test house.

Semi-supervised learning

General purpose machine learning defines semi-supervised learning as the combination of a small amount of labelled training data with a large amount of unlabelled training data. Although this sounds similar to the scenario described above, the crucial difference is that energy disaggregation requires that the supervised and unsupervised learning takes place on data from difference domains (buildings), while general purpose machine learning assumes both the labelled and unlabelled training data are drawn from the same domain. Furthermore, energy disaggregation training methods could even make use of a large amount of labelled training data from non-test houses, and only a small amount of unlabelled training data from the test house.

Summary

I've been apprehensive to use the term semi-supervised learning to describe practical energy disaggregation methods due to the domain-specific requirements of the field. Instead, I generally refer to methods as unsupervised if they use appliance-level data from only non-test houses, which often leads to confusion. I'd be interested to hear other people's opinions on the matter, and hopefully we can reach some consensus!

Announcing the 2015 European NILM workshop

I'm really excited to announce that the Second European Workshop on Non-intrusive Load Monitoring will be held on 8th July 2015 at Imperial College London. The workshop is the follow up to last year's NILM @ London workshop, which provided the first European venue which brought together both academics and companies with an interest in energy disaggregation. Updates and registration information can be found at the new website: www.nilm.eu

Some important information:

• When: 8th July 2015
• Where: Imperial College London, UK
• Cost: Free
• Objective: To provide a European venue for disaggregation researchers to discuss recent developments in the field and fuel future collaborations

This workshop will have a more technical focus than the first workshop, and will feature a keynote from Mario Bergés in the morning and a technical session of invited talks in the afternoon. Furthermore, we're inviting all attendees to bring a poster on a topic of their choice, which could be a recent piece of work, their company's current direction, or even an invitation for collaboration on a joint project. Last, we're hoping to live stream the event online for anyone who can't be there in person, though this is a little experimental!

EEme release disaggregation accuracy calculated by Pecan Street

I recently came across EEme, a spin-out from Carnegie Mellon University, who apply energy disaggregation to 15-minute smart meter data to provide demand side management analytics. EEme are particularly interesting, since they’re the first company (to the best of my knowledge) to go completely public regarding the accuracy of their product, as calculated by a third party. EEme used Pecan Street’s 3rd party evaluation tool, which provided EEme with 15-minute aggregate smart meter data and weather data from hundreds of homes, and required EEme to return monthly totals for four of the largest energy consuming appliances. Since Pecan Street also measured the appliance-level energy consumption as part of their deployment, they are able to calculate the exact level of accuracy of EEme’s disaggregation. The full 7 page report is available for request from EEme’s website. Although EEme focus on lower resolution data than many other energy disaggregation companies, I’m interested to see whether this report sparks the beginning of a more public competition between energy disaggregation companies to release accuracy statistics as confirmed by a third party.