Tuesday 12 November 2013

Belkin Energy Disaggregation Competition - Completed

A while ago, I wrote a post about the Belkin Energy Disaggregation Competition, which concluded a couple of weeks ago. The competition drew entries from 165 participating teams, who each provided up to 169 submissions. The top 3 participants shared prizes from a combined pot of $24,000, while a separate data visualisation competition had a prize of $1,000.

It seemed like most entrants were regular Kagglers, with little participation from NIALM companies or academics with the NIALM field. Although I understand that many companies are likely unwilling to participate to prevent their secrets being divulged, I wonder if greater participation from the existing NIALM academic field could have been achieved by hosting the competition at a relevant conference or workshop. I would love to see the winners of the competition invited give a talk about their approaches!

The data used in the competition consisted of a public (training) and a private (test) data set, collected from 4 households. The training data included both household aggregate data and individual appliance sub-metered data. However, only the aggregate test data was released, while the sub-metered data was kept private for the evaluation of each submission. As a result, although a cross-validation evaluation technique was used, the participants were crucially not required to generalise to new households since sub-metered data was available from each test household for training.

With each submission, the public leaderboard was updated showing the best performance for each user over half of the private test data, while their undisclosed performance over the other half of the test data was used to calculate the final standings. Interestingly, the winner of the competition shown by the final standings was actually only ranked 6th on the public leaderboard. This suggests that many participants might have been overfitting their algorithms to the half of the test data for which the performance was disclosed, while the competition winner had not optimised their approach in such a way.

An interesting forum thread seems to show that most successful participants used an approach based on only low-frequency data, despite the fact that high-frequency data was also provided. This seems to contradict most academic research, which generally shows that high-frequency based approaches will outperform low-frequency methods. A reason for this could be that, although high-frequency based approaches perform well in laboratory test environments, their features do not generalise well over time, and as a result algorithm training quickly becomes outdated. However, another reason could have been that the processing of the high-frequency features was simply too time consuming, and better performance could be achieved by concentrating on the low-frequency data given the deadline of the competition.

Overall, I think the competition was very successful in provoking interest in energy disaggregation from a new community, and I hope that any follow up competitions follow a similar format. Furthermore, I think that hosting a prize giving and presentation forum at a relevant conference and workshop would inspire greater participation from academics already working in the field in NIALM.

Tuesday 5 November 2013

Thesis Finished!

Today I finally finished my thesis titled: 'Unsupervised Training Methods for Non-intrusive Appliance Load Monitoring from Smart Meter Data'. My viva (defence) is scheduled for December, and I plan to upload a final version upon addressing any revisions it brings up. In the meantime, I've included the thesis abstract below:

Non-intrusive appliance load monitoring (NIALM) is the process of disaggregating a household's total electricity consumption into its contributing appliances. Smart meters are currently being deployed on national scales, providing a platform to collect aggregate household electricity consumption data. Existing approaches to NIALM require a manual training phase in which either sub-metered appliance data is collected or appliance usage is manually labelled. This training data is used to build models of the household appliances, which are subsequently used to disaggregate the household's electricity data. Due to the requirement of such a training phase, existing approaches do not scale automatically to the national scales of smart meter data currently being collected.

In this thesis we propose an unsupervised training method which, unlike existing approaches, does not require a manual training phase. Instead, our approach combines general appliance knowledge with just aggregate smart meter data from the household to perform disaggregation. To do so, we address the following three problems: (i) how to generalise the behaviour of multiple appliances of the same type, (ii) how to tune general knowledge of appliances to the specific appliances within a single household using only smart meter data, and (iii) how to provide actionable energy saving advice based on the tuned appliance knowledge.

First, we propose an approach to the appliance generalisation problem, which uses the Tracebase data set to build probabilistic models of household appliances. We take a Bayesian approach to modelling appliances using hidden Markov models, and empirically evaluate the extent to which they generalise to previously unseen appliances through cross validation. We show that learning using multiple appliances vastly outperforms learning from a single appliance by 61-99% when attempting to generalise to a previously unseen appliance, and furthermore that such general models can be learned from only 2-6 appliances.

Second, we propose an unsupervised solution to the model tuning problem, which uses only smart meter data to learn the behaviour of the specific appliances in a given household. Our approach uses general appliance models to extract appliance signatures from a household's smart meter data, which are then used to refine the general appliance models. We evaluate the benefit of this process using the Reference Energy Disaggregation Data set, and show that the tuned appliance models more accurately represent the energy consumption behaviour of a given household's appliances compared to when general appliance models are used, and furthermore that such general models can perform comparably to when sub-metered data is used for model training. We also show that our tuning approach outperforms the current state of the art, which uses a factorial hidden Markov model to tune the general appliance models.

Third, we apply both of these approaches to infer the energy efficiency of refrigerators and freezers in a data set of \117 households. We evaluate the accuracy of our approach, and show that it is able to successfully infer the energy efficiency of combined fridge freezers. We then propose an extension to our model tuning process using factorial hidden semi-Markov models to model households with a separate fridge and freezer. Finally, we show that through this extension our approach is able to simultaneously tune the appliance models of both appliances.

The above contributions provide a solution which satisfies the requirements of a NIALM training method which is both unsupervised (no manual interaction required during training) and uses only smart meter data (no installation of additional hardware is required). When combined, the contributions presented in this thesis represent an advancement in the state of the art in the field of non-intrusive appliance load monitoring, and a step towards increasing the efficiency of energy consumption within households.