Monday 28 February 2011

Load Signature Study—Part I: Basic Concept, Structure, and Methodology

Liang J, Ng S, Kendall G, Cheng J. Load Signature Study - Part I: Basic Concept, Structure, and Methodology. Power Delivery, IEEE Transactions on. 2010;25(2):551-560.

This is the first of a two part study of domestic appliance load signatures. The authors spend the bulk of this article defining concepts and mathematical terms which are built upon in subsequent sections. A description of existing load concepts, signature features and disaggregation methods was given, each of which was assigned an acronym to aid identification.

The paper's contribution lies in the proposal of:
  • Committee Decision Mechanisms - methods which combine various feature extraction and classification methods in order find the best final solution.
  • Disaggregation algorithm accuracy measures - metrics which can be used to evaluate the individual accuracy of event detection or classification, or the combination of the two.
  • Appliance signature similarity measures - metrics which can be used to compare the similarity between various signatures.
  • Disaggregation algorithm Complementary Ratio - metrics which can be used to calculate whether a disaggregation algorithm adds any additional information to that given by a second disaggregation algorithm.
I am now planning to read the second part of this study:

Friday 25 February 2011

Using smoothness to detect appliances

This post follows up on a previous post describing how if we subtract an appliance's signature from the aggregate consumption, we end up with a smoother plot. This post gives an evaluation of the effectiveness of using smoothness to detect the operation of the fridge, washing machine and dishwasher from the household aggregate signal.

Aggregate & Truth Values

The graph below shows a plot of the household aggregate power demand (W) and sub-metered appliances over a 24-hour period. The household aggregate is shown in black, the fridge in blue, the washing machine in green and the dishwasher in red.

Calculating the individual appliance's demand from the household aggregate is equivalent to calculating the blue, green and red lines from the black line. To do this, at each possible position within the 24 hour period, the known signature of the appliance was subtracted from the aggregate, and the smoothness of the resulting data was calculated. This was repeated for each appliance in turn. Below are plots of the smoothness property at each position in the 24 hour period for the fridge, washing machine and dishwasher.


Below is a plot of the smoothness function calculated for each possible subtraction of the fridge's signature from the aggregate plot. The y-axis is the smoothness measure, where a positive value represents a smoother plot after the signature was subtracted, while a negative value represents a less smooth plot.

We can see that this method works very well during the overnight period. However, during the day there are a number of false positives in addition to the correct detections. This is due to the fridge's relatively small power demand (90W) compared to the household average (500W).

Washing Machine

Below is a plot of the smoothness function calculated for each possible subtraction of the washing machine's signature from the aggregate plot.

This has worked well compared to the fridge. We can see one peak at around 300 on the x axis, which corresponds to a correct identification of the washing machine's cycle. However, there are also two false positives later in the day.


Below is a plot of the smoothness function calculated for each possible subtraction of the dishwasher's signature from the aggregate plot.

The application of this technique to the dishwasher has been the most successful. We can see there are two clear peaks, both of which correspond to the two times the dishwasher was run in the 24 hour period. There are no false positives as with the other appliances, although the peaks do have some close matches of slightly lower values. The success of this method was due the dishwasher's unique signature; two cycles separated by a short interval.


This method works best for appliances:
  • With large power demands compared to the average step change in the aggregate demand
  • With signatures which describe a unique overall pattern

IEEE Pervasive Computing 10 (1): Special Issue on Smart Energy Systems - part 3

Marchiori A, Hakkarinen D, Han Q, Earle L. Circuit-Level Load Monitoring for Household Energy Management. Pervasive Computing, IEEE. 2011;10(1):40-48.

This article compares a heuristic and a probabilistic classification method for steady-state features extracted from a household's aggregate power consumption. The heuristic approach used a clustering approach to train the classifier of each appliance's states. The probability mass function of each cluster was calculated, before combining all possible combinations to find the maximum joint probability. In the probabilistic approach a naive Bayesian classifier was trained using each appliance's states and state transitions. The classifier could then calculate the probability that an appliance in a given state would undergo a specific state transition. In each case, circuit level metering was used.

Although it was not the main focus of the article, it was interesting to learn that authors believe circuit-level metering allows appliances with power demands as small as 10W to be recognised. This extends the potential of NIALM beyond the recognition of appliances drawing as little as 100 or 150W as was possible through premise-level metering.

The authors describe some very promising results for the recognition of a small number of appliances tested. However, the approaches are yet to be tested in real household settings with a greater range of appliances and less usage predictability.

Wednesday 23 February 2011

IEEE Pervasive Computing 10 (1): Special Issue on Smart Energy Systems - part 2

Froehlich J, Larson E, Gupta S, et al. Disaggregated End-Use Energy Sensing for the Smart Grid. Pervasive Computing, IEEE. 2011;10(1):28-39.

This article provides an excellent overview of a wide range current approaches towards NIALM. In addition, the authors provide detail of their approach using high-granularity sampling of voltage noise. Support Vector Machines were used to classify the extracted features against a database of known appliance signatures.

Although the described approach uses a very different method to sample data, it raises some issues relevant to my work. The discussion of which features are able to be generalised between appliance types or households is of particular relevance to any unsupervised NIALM systems. The authors also mention that current smart meters report consumption data with 15-minute granularity. I'm not sure where this figure came form so is therefore worth investigating.

Other papers I want to read from this special issue:
  • Marchiori A, Hakkarinen D, Han Q, Earle L. Circuit-Level Load Monitoring for Household Energy Management. Pervasive Computing, IEEE. 2011;10(1):40-48.
  • Bergman DC, Jin D, Juen JP, et al. Nonintrusive Load-Shed Verification. Pervasive Computing, IEEE. 2011;10(1):49-57.
Interesting papers from elsewhere:
  • L.G. Swan and V.I. Ugursal, “Modeling of End-Use Energy Consumption in the Residential Sector: A Review of Modeling Techniques,” Renewable and Sustainable Energy Reviews, vol. 13, no. 8, 2009, pp. 1819–1835.
  • Liang J, Ng SKK, Kendall G, Cheng JWM. Load Signature Study—Part II: Disaggregation Framework, Simulation, and Applications. IEEE Transactions on Power Delivery. 2010;25(2):561-569. Available at:

IEEE Pervasive Computing 10 (1): Special Issue on Smart Energy Systems

I been reading more from the special issue on smart energy systems. Here are my thoughts:

Paradiso J, Dutta P, Gellersen H, Schooler E. Guest Editors╩╝ Introduction: Smart Energy Systems. Pervasive Computing, IEEE. 2011;10(1):11-12.

This introductory article provides an overall motivation for the application of smart energy systems within the domestic environment. In doing so, it describes the need for agent controlled energy systems and a motivation to study human-agent interaction within the domestic energy domain.

This article focuses on how domestic energy monitoring systems can induce positive behavioural change. The authors report findings from the first of three stages of deployment. The first stage involved the installation of household aggregate electricity meters to 77 houses in the UK. The hardware used was the CurrentCost clamp-on meters, which collected power readings every six seconds. The next stage of deployment will involve the installation of household aggregate electricity meters on more houses in addition to 10 individual appliance sub-meters. The aim is to deploy this equipment in 250 houses in the UK and Bulgaria.

Although the focus of the article was on energy feedback technology, it raised many points of interest to the field of non-intrusive load monitoring. Most interesting. the next deployment stage will investigate the financial evaluation of deployment appliance sub-meters. This will provide a clear motivation for NILM should the study find that the cost of deployment exceeds the financial savings through reduction of energy consumption. The article also covers areas of interest to NILM, such as consumer energy awareness and data privacy.

Next on my list of things to read are three papers directly related to NILM:

Tuesday 22 February 2011

Look Back before Leaping Forward: Four Decades of Domestic Energy Inquiry

I recently read the following article on domestic energy monitoring in the IEEE Pervasive Computing magazine:

Hazas M, Friday A, Scott J. Look Back before Leaping Forward: Four Decades of Domestic Energy Inquiry. Pervasive Computing, IEEE. 2011;10(1):13-19.

This article really highlights the interdisciplinary nature of domestic energy management. It covers the following areas to a shallow level of detail:
  • Personalised feedback
  • Energy monitoring hardware
  • Consumption analysis software (including NIALM)
  • Consumer behaviour
It was actually this article which made me aware of Microsoft's Hohm, among many other existing products for domestic energy monitoring. The authors also mention the AlertMe system as one solution to online energy management. However, the AlertMe system is described as a 'closed system', I assume due to lack of public knowledge of their data APIs.

Although the article covers the area of non-intrusive energy monitoring in little depth, it was still of great value to me. The wide range of references has encouraged me to look specifically at domestic energy audits and studies of energy feedback mechanisms. These will be most relevant to my field when describing the real-world setting of the problem and the barriers faced so far.

Microsoft Hohm

I've just come across Microsoft's Hohm, an energy visualisation web app similar to Google's powermeter. It's only open to US buildings at the moment, but looks pretty sharp and seems to pull data from a variety of sources. Here's a screen shot of the main dashboard:

This shows the building's overall Hohm score, a measure of its efficiency from 0 to 100, 100 being the best. It also compares the annual cost of energy use of the building compared to the local area's average.

From the dashboard you can select an option to breakdown this annual consumption by appliance type, giving the screen show below:

Now the information shown here is very interesting. Without uploading any data collected from meters at the building, Hohm has estimated the breakdown of energy use. Not only is the overall consumption broken down by appliance type, but selecting an appliance type breaks it down further by individual appliances.

The estimates shown are based on data from a range of sources:
  • Building properties (size, age etc.)
  • Appliance properties (chosen boiler type etc.)
  • Weather data
I'd be really interested in how these estimated breakdowns change when the household aggregate energy data is uploaded. In addition, I'd like to see how these estimations differ from those produced by NIALMs.

Monday 21 February 2011

Weekly supervisor meeting

Following this week's meeting with my supervisors, my plan for this week is:
  1. Generalise the fridge signature to ensure a high smoothness value if the series matches a previously recognised signature
  2. Assign values to peaks indicating likelihood that peak a result of the fridge's cycle
  3. Repeat process washing machine/dishwasher signature
  4. Finish AlertMe coding with Tassos
  5. Read up on aggregate pattern matching

Wednesday 16 February 2011

Pattern matching by maximising smoothness

I've recently been trying a new approach to detecting an appliance's signature. The approach consists of:
  1. Generating a typical appliance cycle
  2. For each possible location of the cycle:
  • Subtract the typical appliance cycle from the aggregate readings
  • Calculate the smoothness of the resulting readings
The smoothness function I have used calculates the sum of the absolute values of the changes in power for the whole day.

Below are two plots of the smoothness function for each possible cycle location in the day. The graphs represent the smoothness function for two slightly different typical cycles.

We can see that both graphs identify the overnight fridge cycles well. However, both graphs also generate a large number of false positives. There's such a large number of of these that they obscure any correct positives during the day.

Friday 11 February 2011

Identifying high confidence cycles

I now have two data sets that can be classed as possible cycles:
  • Each possible interval identified by thresholding
  • Every pair of 'on' and 'off' steady states
We know from the analysis of the fridge, the 'on' duration of each cycle has mean = 15 minutes and variance = 2 minutes. If we assume the duration of a cycle is normally distributed about the mean, we can create the following likelihood function:

Now we can filter out possible 'on' cycles below a likelihood threshold, leaving a set of cycles containing possible duplicates. We can remove duplicates by:
  • If two 'on' cycles overlap, discard the one with the lower likelihood
  • If two 'on' cycles are within the minimum 'off' duration of each other, discard the one with the lower likelihood
This produces a list of high confidence 'on' cycles. The next step is to identify any missing cycles between these, using a combination of:
  • Low confidence state-transitions
  • Possible intervals for fridge cycles
  • Applying the model of typical fridge behaviour using surrounding high confidence cycles

New website

I've finally got round to creating a proper personal website. I used Google sites, and the url is:

My ECS homepage ( also redirects there.

Thursday 10 February 2011

Eliminating steady-state detected transitions using bruteforce thresholding

I now have two sets of information for when the fridge cycles occur, created using two different algorithms:
  1. A list of possible 'on' and 'off' events, selected through steady-state analysis of the house power data
  2. A list of every possible intervals in which the fridge could be on, selected through brute-force thresholding of the house power data
The steady-state analysis identified an unreliable list of 'on' and 'off' events. Some transitions correctly identified a state transition of the fridge, while others identified state transitions of other appliances. In addition, the steady-state analysis failed to detect some state transitions of the fridge due to noise generated from other appliances.

My aim for today was therefore to remove state transitions caused by other appliances, leaving only those generated by the fridge. I did this by eliminating state transitions using a combination of:
  • the possible intervals of operation collected through brute-force thresholding
  • the fridge's known signature.
I generated 3 rules which I used to remove incorrect state transitions:
  1. No state transition belongs to the fridge if it occurs outside of the intervals of possible operation
  2. No 'on' transition belongs to the fridge if it occurs less than the minimum 'on' duration after an impossible interval of operation
  3. No 'off' transition belongs to the fridge if it occurs less than the minimum 'on' duration prior to an impossible interval of operation
This filtered the list of state transitions, leaving the list of plausible transitions shown below.
  • The black line represents the household's power level
  • The blue line represents the fridge's power level
  • The green negative peaks represent detected 'on' transitions
  • The red negative peaks represent detected 'off' transitions

This is a fairly good start. Nearly all cycles during the night were correctly identified, while most cycles during the day had at least either their 'on' or 'off' transition detected. However, there are still a number of erroneous state transitions which must be removed.

The remaining steps to get the best estimate of each fridge cycle are:
  1. Identify cycles for which we have a high confidence in their correctness
  2. Fill in gaps between correctly classified cycles using a combination of:
  • Low confidence state-transitions
  • Possible intervals for fridge cycles
  • Applying the model of typical fridge behaviour using surrounding high confidence cycles

Wednesday 9 February 2011

Brute-force thresholding to detect possible periods for fridge cycles

The chart below shows data recorded from the house's aggregate power meter and the fridge's meter over 1 day. My aim for this week is to calculate the fridge's power demand given the household's power demand.

Before doing any steady-state analysis, it is useful to use some brute-force methods to eliminate intervals where the household's aggregate demand is less than the fridge requires. Here is some pseudo code describing the algorithm I devised:

baseline = min(value_of_household_aggregate)
for each value_of_household_aggregate
if value_of_household_aggregate >= baseline + appliance_demand
mark value as a possible area for fridge cycle

The following graph shows the areas for which the fridge could be on. The upper line represents the same day's household aggregate as the previous graph, while the lower line represents the areas detected for which it is possible the fridge is operating.

We can see that this approach works well in the overnight period, accurately identifying the fridge's on periods. However, the approach is less useful during the day, when there are long periods for which the aggregate demand is continuously above the baseline + appliance demand.

Since we know the fridge's minimum 'on' duration, we can improve this algorithm by eliminating time periods which are shorter than the appliance's minimum duration. This is illustrated by the following pseudo code:

for each possible_fridge_interval
if possible_fridge_interval less_than minimum_appliance_duration
remove interval

This produces the following areas:

This is an improvement on our last results, as all the short intervals have now been removed.

The next step is to take the longer intervals and break them up in to intervals of the expected duration. My plan for the next few days:
  1. I am going to apply my steady-state algorithm to detect 'on' and 'off' events of a magnitude which we would expect from the fridge
  2. The detected 'on' and 'off' events which fall outside of the possible intervals detected through the brute-force method will be removed
  3. 'On' events followed by an 'off' event which are separated by the expected appliance 'on' duration will be assumed to be a correctly identified cycle
  4. 'On' or 'off' events with no other near events will have their corresponding 'on' or 'off' event estimated using the expected appliance duration
  5. The remaining long intervals will be filled by finding the best fit of cycles within the interval. A constant 'on' duration will be assumed. The 'off' duration will be estimated by fitting the 'duration between on cycles' curve to the nearest cycles before and after the long interval

Tuesday 8 February 2011

Analysis and prediction of Alex's fridge's consumption

I've been analysing data collected from a meter connected to Alex's fridge to further investigate its consumption pattern. I've written an algorithm to detect the 'on' and 'off' transitions in the power data, and corrected these timestamps using the process described in my previous post. From this sequence of state transitions I have analysed the following 4 features:

On transition:
mean = 89W
variance = 13

Off transition:
mean = -83W
variance = 3

On duration:
mean = 15 minutes
variance = 2

Off duration:
mean = 35 minutes
variance = 38

Interestingly, the changes in power during transitions have a low variance, which is encouraging for classification purposes.

However, although the duration for which the fridge stays on also has a low variance, the duration for which it stays off has a high variance. Surprisingly, this suggests the fridge's temperature is regulated by varying the length of time between 'on' cycles. Should we want to predict the cycle of the fridge we need to model the pattern in the length of the 'off' periods. The graph below plots the duration of the 'off' cycle over a 4 day period.

Duration (minutes) of the fridge's 'off' cycle over a 4 day period

There is a clear pattern visible here which repeats every 24 hours (4 times on this graph). There is a steady increase overnight, peaking at around 6am. This is followed by a sharp trough most likely due to opening the fridge door at breakfast. The pattern over the remainder of the day is less predictable, although I would expect to see another trough around 7pm due to opening the fridge door when preparing dinner.

A suitable generalisation of this data would be useful to predict the time instants of 'on' and 'off' transitions in the fridge's cycle. This could be used to estimate the corresponding 'on' or 'off' transition should only one be extracted by steady-state analysis. In addition, it could be used to estimate both transitions in a cycle given either (or both of) the previous or following cycle's transitions.

In an attempt to model this data, I plotted a scatter of these day's data over the 24 hours and fitted a cubic function.

This would be of most use for long periods for which there have been no state transitions detected. A typical day has a period of roughly 17 hours where the fridge's signature might have not been detected at all. In this case, the curve would be used to estimate the off duration of each cycle, while keeping 'on' and 'off'power changes and the 'on' duration constant.

Calculating accurate appliance state transitions using power and energy readings

Suppose we are sampling instantaneous power readings at regular intervals and energy consumption over each interval of a household's aggregate load.

Let the two time instants be t0 and t1 such that they are separated by the sampling interval, t1 - t0
Let the two power readings at t1 and t0 be p1 and p0 respectively.
Let the energy readings at t1 and t0 be e1 and e0 respectively.
Since the energy readings are assumed to be cumulative, we know the energy consumed over the period, e_aggregate
e_aggregate = e1 - e0
These values are marked on the graph below. Since the area under the graph represents energy consumed, e_aggregate is represented by the combined blue and red areas.

Using these readings, we can accurately calculate the time instant at which the appliance changed state, tx. Let e_baseline and e_appliance be the energy consumed by the baseline appliances and the appliance of interest respectively.
We know that
e_aggregate = e_baseline + e_appliance
A simple substitution of energy = power * time for both e_baseline and e_appliance gives
e_aggregate = (t1 - t0) * p0 + (t1 - tx) * (p1 - p0)
The only unknown is tx, allowing us to calculate its value by rearranging this equation.

Thursday 3 February 2011

Meeting with Tassos

Today I met with Tassos, a researcher from the field of energy and buildings to discuss some specifics of AlertMe's data API. Their group is deploying a set of AlertMe kit to 200 houses, and mentioned that it might be possible for me to use some of the data they're collecting. The kit that is being deployed to each house consists of:
  • 1 Hub
  • 1 Meter - attached to household electricity entry - power + temperature
  • 1 Button - put in resident's car - car's location + temperature
  • 1 Motion sensor - living room - resident's motion + temperature
  • 1 Door magnet - on boiler - temperature
  • 1 Key ring - not used
  • 0/1 Smart plug - attached to an arbitrarily chosen appliance - power + temperature
The most interesting point to me was that they no plan for which appliance to attach the smart plug to. He mentioned that they'd be willing to attach it to a range of appliances, e.g. in 20 houses attach it to the fridge, in 20 other houses attach it to the dishwasher etc.

This would be really useful for me for three reasons:
  1. I would be able to analyse the readings collected from the smart plugs to investigate the range of features visible from each appliance's signature
  2. I would be able to compare signatures from similar appliances in different homes, e.g. investigate the extent to which signatures created by fridges vary
  3. I would be able to test my disaggregation algorithms on the household's aggregate data and measure their accuracy against the sub-metered data