3.5 KiB
loadwatch — Tier-0 NILM dataset builder
Non-intrusive load monitoring, working from a single Dutch DSMR-5 P1 stream at 1 Hz. Goal: build a labelled dataset of large-load switching events, so a model can later identify which big appliance turned on.
What this is
| Layer | What it does |
|---|---|
collector |
Subscribes to the SlimmeLezer+ ESPHome native API (1 Hz) and stores every sample |
detector |
Step-change detection → load episodes (on-step, duration, energy, peak) |
api + UI |
Live power, event list, one-click labelling, CSV export for training |
db |
TimescaleDB (Postgres 16) — 1-min continuous aggregate, compression after 7 days |
No Home Assistant involvement on the ingest path (HA stays the notification shell).
Architecture decision (2026-10-04)
Separate service on dev CT141, not inside HAOS:
- HA here is Core-in-Docker — custom Python services installed there die on every HA upgrade, and the recorder is not a dataset store (purges raw state ~10 days, writes on-change).
- Ingest goes straight to the meter (
192.168.178.213:6053, ESPHome API, no encryption) — no HA token, no secrets on this path. - Public URL via Coolify:
loadwatch.xm2561.duckdns.org(thexm2561apex belongs to Headscale — never take it).
Data model
samples(ts timestamptz, device text, key text, value double precision) -- hypertable, 1 day chunks
events(id, ts_start, ts_end, direction, delta_w, peak_w, baseline_w, duration_s,
energy_wh, appliance, label_source, labeled_at, notes)
appliances(name, notes, is_large)
samples is narrow (device, key) so additional sources — a smart plug, later a
second meter — need no schema change.
Run it (dev)
cd /root/projects/loadwatch
cp .env.example .env # set API_TOKEN
docker compose up -d --build
docker compose logs -f collector
UI: http://192.168.178.20:8099/ (token = API_TOKEN).
Collector entity keys
The collector matches ESPHome object_ids, discovered at connect time and logged:
| object_id | meaning |
|---|---|
power_consumed_phase_1 |
instantaneous import W |
power_produced_phase_1 |
instantaneous export W |
voltage_phase_1, current_phase_1 |
grid V / A |
energy_consumed_tariff_1/2 |
import kWh counters |
energy_produced_tariff_1/2 |
export kWh counters |
gas_consumed |
gas m³ (M-Bus, ~5 min) |
If a key is missing the collector logs every object_id it can see — that is the discovery mechanism, not a guess.
Adding the smart plug (ground truth)
Add its host to METER_HOSTS (comma-separated) — samples land with device = <host>.
A plug on one appliance gives labelled ground truth for exactly that appliance, which is
what a supervised model needs; the P1 event stream covers everything else.
Export for training
GET /api/export.csv?days=30 → one row per labelled event plus its raw power window
(GET /api/events/{id}/window returns the ±60 s raw trace for that event).
Detection defaults
EVENT_THRESHOLD_W=80 (step size), EVENT_MIN_DURATION_S=3, close back within 40 W held
3 s. Night baseload is ~50–200 W here; a kettle/oven/washer step is 1000–2500 W.
Documented limitation: two loads stepping at the same instant inside one open episode
are seen as one event.
Roadmap
- ✅ capture + detect + label (this repo)
- plug lane → labelled ground truth
- HA REST sensor → "big load started" notifications
- classifier trained on the labelled events (
trainer/)