Data set released by EDF Energy

I've only just come across this data set, despite it being released almost a year ago! I've also updated my post of public data sets.

EDF Energy released a data set in 2012 containing energy measurements made at a single household in France for a duration of 4 years. Average measurements are available at 1 minute resolution of the household aggregate active power, reactive power, voltage and current, as well as the active power of 3 sub-metered circuits. Although each circuit contains a few appliances, this is the largest data set in terms of duration of measurement. The complete data set is openly available from the UCI Machine Learning Repository.

The pros and cons of using HMMs to model appliances

In the last few years, hidden Markov models (HMMs) have become a very popular mathematical representation for appliances (Zia et al. 2010, Kim et al. 2011, Kolter et al 2012, Parson et al 2012). As a result, I'm often asked whether I think HMMs are the future of disaggregation. However, I'm yet to find an objective analysis of the advantages and disadvantages of such approaches, which is why I've done my best to list them here:

Advantages

• The HMM is a well studied probabilistic graphic model, for which algorithms are known for exact and approximate learning and inference
• HMMs are able to represent the variance of appliances' power demands through probability distributions
• HMMs capture the dependencies between consecutive measurements, as defined by Hart as the switch continuity principle

Disadvantages

• HMMs represent the behaviour of an appliance using a finite number of static distributions, and therefore fail to represent appliances with a continuously varying power demand
• Due to their Markovian nature, they do not take into account the sequence of states leading into any given state
• Again, due to their Markovian nature, the time spent in a given state is not captured explicitly. However, the hidden semi-Markov model does capture such behaviour
• Features other than the observed power demand are not captured (e.g. time of day). However, the input-output HMM allow such such state durations to be modelled
• Any dependency between appliances cannot be represented. However, the conditional-HMM can capture such dependencies

In summary, the basic HMM provides a useful model for many appliances. However, the appliances it can represent are limited by the intrinsic structure of the model. Many extensions exist that increase the representational power of the HMM, although the additional parameters required often complicate the learning and inference tasks.

AAAI 2012 Code Release

A while ago, I wrote a post stating that I was planning to release my NIALM code at the end of my PhD. I also mentioned in the post that I'd been happily giving out an archive of my code upon request. Since then, I've had far more requests than I'd expected, as well as quite a few technical questions regarding how to run it. As a result, I've decided to make my code from my AAAI 2012 paper available via my github for anyone to clone or contribute to.

The reason why I hadn't previously uploaded my code is that I simply do not have time to provide documentation or tutorials for using my code. Therefore, my code is provided "as is", so apologies in advance if you don't find it easy to use!

Update 15.09.2015: updated link to point to github

Trip to the Minnesota, New York and North Carolina

I'll be visiting the US states of Minnesota, New York and North Carolina in the coming weeks, so please give me a shout if you're nearby and would like to talk disaggregation!

DECC meeting on disaggregating UK smart meter data

Last week I attended an expert panel meeting for the Department of Energy and Climate Change to discuss how smart meter data could be used to better understand household energy use. The meeting was organised by Cambridge Architectural Research Ltd, and brought together a wide range of stakeholders from government, industry and academia. Among the many potential projects and barriers which were discussed, I've categorised what I believe to be the important facts:

Data availability

UK smart meters will only automatically upload 30 min data for billing purposes. However, 10 second data will also be available to Consumer Access Devices (CAD), via short range WiFi. This creates two possibilities for disaggregation from 10 second data:

1. A disaggregation system could be installed in each household as a CAD
2. A CAD could upload data to cloud storage via the home broadband connection

The second option seems the most realistic to me, given the intrinsic opt-in nature of disaggregation and the benefits of performing disaggregation in the cloud.

Data granularity

The most recent smart meter specification (SMETS v2 2013) states that only 10 second apparent power data will be available to CADs. However, it would theoretically be possible to increase the reporting rate up to 1 second data through a smart meter firmware upgrade. This rate of 1 report per second is the theoretical maximum rate of smart meters as a result of hardware limitations. Furthermore, current, voltage, harmonics, reactive power etc. will not be reported by smart meters at any sub-10 second rate.

Appliance database

Another topic discussed was the potential for a UK appliance database, similar to the Tracebase database, or a disaggregation test set, similar to the REDD data set. One potential source of data is the Powering the Nation database, which DECC/DEFRA plan to release in the near future. The study collected data from 250 homes which were monitored for either 1 month or 1 year to investigate domestic energy consumption habits.

New data set released by Pecan Street Research Institute

Pecan Street Research Institute recently announced the release of a new data set designed specifically to enable the evaluation of electricity disaggregation technology. A free sample data set is available to members of its research consortium, which has now been opened up to university researchers. The sample data set contains 7 days of data from 10 houses in Austin, TX, USA, for which both aggregate and circuit data is also available containing power readings at 1 minute intervals. In addition to common household loads, 2 of the houses also have photovoltaic systems and 1 house also has an electric vehicle.

NIALM helpful terminology

When discussing related research, most papers group existing disaggregation approaches into distinct categories. As a result, many taxonomies have emerged, and unfortunately they are not always well defined before they are used. I therefore decided to compile the following list of the terminology which I've seen in recent years:

Intrusive vs non-intrusive monitoring

• Intrusive metering refers to the deployment of one monitor per appliance. This is clearly intrusive since it requires access to each appliance to install such equipment. This has the benefit in that the only uncertainty in such monitoring is due to inaccuracies in the metering hardware.
• Non-intrusive metering refers to the deployment of one (or sometimes two) meters per household. This is clearly less intrusive, since it does not provide any inconvenience besides the installation of government mandated smart meters. However, this has the disadvantage that the disaggregation process is likely to introduce further inaccuracies.

Supervised vs unsupervised training

• Supervised training (a.k.a manual setup) refers to performing disaggregation with the aid of labelled appliance data, generally from the same home in which disaggregation is performed. The training data normally consists of sub-metered appliance power data, or a phase in which appliances are turned on one by one, and labelled manually.
• Unsupervised training (a.k.a automatic setup) refers to performing disaggregation without any training data from the household in which disaggregation is being performed. However, without any notion of what appliances exist or how they behave, at best a system can only identify distinct appliances (e.g. appliance 1, appliance 2), and cannot label them with an appliance name (e.g. refrigerator or washing machine).

Event-based vs non event-based disaggregation

• Event-based disaggregation refers to methods which have distinct event detection (e.g. something switched on at 12pm) and event classification (e.g. it was the washing machine). These approaches are often identifiable by a sequential pipeline of algorithms (data collection -> smoothing -> edge detection -> classification). A core advantage of event-based approaches is that decisions are made sequentially, and therefore can easily be deployed as a real-time system.
• Non event-based disaggregation refers to methods which combine event detection and classification into a single process, in which both are inferred simultaneously. These are often identifiable by their use of time series models, which are able to reason over a sequence of data. The advantage of non event-based approaches is that high confidence decisions can affect those that are likely to surround it (e.g. a refrigerator is likely to turn on 30 minutes after its last cycle ended).

High frequency vs low frequency sampling

• High frequency sampling generally refers to meters which sample the current and voltage of a wire at a rate in the order of thousands of times per second (kHz). At this rate, information such as reactive power and current harmonics can be calculated, which are useful features for classification. However, few smart meters are likely to report data at this granularity.
• Low frequency sampling generally refers to meters which sample at between once per second and once per hour. When reported at this rate, active power is indistinguishable from reactive power, and no harmonic content is available. This is the reporting rate of most smart meters.

Steady-state vs transient-state analysis

• Steady-state analysis divides a power series into periods of constant power during which no appliances change state. The differences between these levels of constant power are then used to infer which state change(s) had taken place.
• Transient-state analysis uses the patterns between steady states to classify appliance state changes. However, it is necessary to sample at a high frequency in order to extract transient features for most appliances.

As always, please leave a comment if you come across any terminology you think should be in this list.