diff --git a/.cspell/custom-dictionary-workspace.txt b/.cspell/custom-dictionary-workspace.txt index a6716bc84..4aa094c85 100644 --- a/.cspell/custom-dictionary-workspace.txt +++ b/.cspell/custom-dictionary-workspace.txt @@ -33,6 +33,7 @@ backprop Backpropagate backpropagation backsteps +backtest Basepath Batpred battemperature @@ -164,6 +165,7 @@ heatingactive heatingtemp heatpump heatsink +holdout homeassistant houseb htmlcov @@ -280,22 +282,23 @@ pdata pdetails perc percnt -PeterHaban +peterhaban photovoltaics pitem pkwh -Plenticore +plenticore postdata powercontrolmode powerline Powerwall ppdetails ppkwh -Pred -Predai -Predbat +pred +predai +predbat predbatt -Predheat +predheat +prevs psum pvbat pvenergytotal diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..66fb8eb7e --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,127 @@ +# AGENTS.md + +This file provides guidance to Codex (Codex.ai/code) when working with code in this repository. + +## Project Overview + +Predbat is a Home Assistant addon (app) that predicts and optimizes home battery charging/discharging based on electricity rates, solar forecasts, and historical load data. It supports inverters from GivEnergy, Solis, Huawei, SolarEdge, and Sofar, and integrates with energy providers like Octopus Energy, Kraken (EDF/E.ON), and Axle Energy VPP. + +It also supports Predbat.com which is a cloud based product that does not use Home Assistant and can run in a Docker environment. + +## Running Tests + +Tests take time to run, _always_ save the test output to a file and then grep the file afterwards. +Never just pipe the output to grep as if you search for the wrong thing you will have to re-run it again. + +Tests live in `apps/predbat/tests/` and are run from the `coverage/` directory: + +```bash +# First-time setup (creates venv and installs deps): +cd coverage +source setup.csh + +# Run all tests: +./run_all + +# Skip slow tests (used by CI): +./run_all --quick + +# Run a specific test by name: +./run_all --test basic_rates + +# Run multiple specific tests: +./run_all --test basic_rates --test units + +# Run tests matching a keyword: +./run_all -k octopus_ + +# List all available test names: +./run_all --list + +# Coverage analysis: +./run_cov --quick +# Then open htmlcov/index.html +``` + +The `run_all` script is a thin wrapper; you can run `unit_test.py` directly from the `coverage/` directory (it needs to be the working directory so relative paths resolve). + +## Code Quality + +All checks are enforced via pre-commit and must pass before merging: + +```bash +./run_pre_commit +``` + +Key constraints: + +- **Line length**: 256 chars (Black), 250 chars (Flake8) +- **Docstrings**: 100% coverage required (`interrogate`) for all functions and classes +- **Spell checking**: British English (`en-gb`) via CSpell; add valid unknown words to `.cspell/custom-dictionary-workspace.txt` (file is auto-sorted alphabetically on commit, so re-stage after running pre-commit) +- **Variable naming**: `lower_case_with_underscores` +- pre-commit.ci will auto-commit fixable issues (trailing whitespace, etc.) back to your PR branch — run `git pull` after pushing to avoid divergence + +## Architecture + +### Orchestrator Pattern + +`PredBat` in `predbat.py` is the main class and uses **multiple inheritance** to compose its behaviour: + +```python +class PredBat(hass.Hass, Octopus, Energidataservice, Fetch, Plan, Execute, Output, UserInterface): +``` + +The main loop (`update_pred()`) runs every 5 minutes: fetch data → run optimization → execute plan → publish results. + +### Core Modules + +| Module | Role | +|--------|------| +| `plan.py` | Optimization engine — multi-threaded search across thousands of charge/discharge window scenarios | +| `predict.py` / `prediction.py` | Battery SOC prediction models, PV generation, load forecasting | +| `fetch.py` | Pulls PV forecasts, historical load, rate data, and inverter state | +| `execute.py` | Sends charge/discharge/reserve commands to inverters | +| `output.py` | Creates and updates Home Assistant sensors, switches, selects | +| `inverter.py` | Multi-inverter abstraction layer (GivEnergy, Solis, Huawei, SolarEdge, Sofar) | +| `config.py` | Defines `CONFIG_ITEMS` (all user settings) and `APPS_SCHEMA` (YAML validation) | +| `ha.py` | WebSocket + REST communication with Home Assistant | +| `userinterface.py` | Manages HA input entities (switches, selects, input_numbers) | +| `components.py` | Plugin registry and component lifecycle management | +| `component_base.py` | Abstract base class for all pluggable components | + +### Component/Plugin System + +`components.py` defines a registry of 18 pluggable components (DB, HA, Web, MCP, GECloud, Octopus, Fox, Solax, Solis, Axle, Ohme, Kraken, etc.). Each component: + +- Inherits from `ComponentBase` +- Has `api_start()` / `api_stop()` lifecycle methods +- Can be independently enabled/disabled +- Has health monitoring with exponential backoff +- Routes HA events via entity prefix filtering + +### Key Data Flow + +1. `Fetch` retrieves rates (Octopus/Kraken API), solar forecasts (Solcast), historical load (HA history), and live inverter state +2. `Plan` runs a search algorithm to find the optimal set of charge/discharge windows over a 48-hour horizon +3. `Execute` sends the resulting commands to the inverter +4. `Output` publishes the plan and metrics as HA sensor states + +### Storage + +The Storage component provides an abstraction of saving/loading from a cache and must be used instead of direct file access. + +### Testing Infrastructure + +`unit_test.py` uses `TestHAInterface` (from `tests/test_infra.py`) to mock the Home Assistant connection. Tests call `create_predbat()` which builds a full `PredBat` instance against the mock. Individual test modules in `tests/` follow the naming convention `test_.py` with an exported `run__tests()` or `test_()` function registered in `TEST_REGISTRY` in `unit_test.py`. + +**IMPORTANT** Unit tests must be added for all new code. + +## Documentation + +Documentation source lives in `docs/` and is built with MkDocs: + +```bash +mkdocs serve # Live preview at http://localhost:8000 +``` + +When adding a new doc page, add it to `mkdocs.yml`. The published site at is built automatically from `main` via GitHub Actions. diff --git a/apps/predbat/config.py b/apps/predbat/config.py index ef12ca699..35d5e0652 100644 --- a/apps/predbat/config.py +++ b/apps/predbat/config.py @@ -2196,6 +2196,7 @@ "rates_export_override": {"type": "dict_list"}, "days_previous": {"type": "integer_list"}, "days_previous_weight": {"type": "float_list"}, + "days_previous_auto": {"type": "boolean"}, "forecast_hours": {"type": "integer"}, "notify_devices": {"type": "string_list"}, "battery_scaling": {"type": "sensor_list", "sensor_type": "float", "entries": "num_inverters", "modify": False}, diff --git a/apps/predbat/const.py b/apps/predbat/const.py index faccd55d6..87b386179 100644 --- a/apps/predbat/const.py +++ b/apps/predbat/const.py @@ -24,6 +24,7 @@ TIME_FORMAT_SOLIS = "%Y-%m-%d %H:%M:%S" PREDICT_STEP = 5 RUN_EVERY = 5 +LOAD_FORECAST_HISTORY_MAX_DAYS = 30 # Max days of history used by the weighted-bucket load forecast (days_previous_auto) CONFIG_ROOTS = ["/config", "/conf", "/homeassistant", "./"] TIME_FORMAT_HA = "%Y-%m-%dT%H:%M:%S%z" TIME_FORMAT_HA_TZ = "%Y-%m-%dT%H:%M:%S.%f%z" diff --git a/apps/predbat/fetch.py b/apps/predbat/fetch.py index b3d810018..fca7744c4 100644 --- a/apps/predbat/fetch.py +++ b/apps/predbat/fetch.py @@ -20,7 +20,7 @@ from datetime import datetime, timedelta from utils import minutes_to_time, str2time, dp1, dp2, dp3, dp4, time_string_to_stamp, minute_data, get_now_from_cumulative, MinuteArray -from const import MINUTE_WATT, PREDICT_STEP, TIME_FORMAT, PREDBAT_MODE_OPTIONS, PREDBAT_MODE_CONTROL_SOC, PREDBAT_MODE_CONTROL_CHARGEDISCHARGE, PREDBAT_MODE_CONTROL_CHARGE, PREDBAT_MODE_MONITOR +from const import MINUTE_WATT, PREDICT_STEP, TIME_FORMAT, PREDBAT_MODE_OPTIONS, PREDBAT_MODE_CONTROL_SOC, PREDBAT_MODE_CONTROL_CHARGEDISCHARGE, PREDBAT_MODE_CONTROL_CHARGE, PREDBAT_MODE_MONITOR, LOAD_FORECAST_HISTORY_MAX_DAYS from predbat_metrics import metrics from futurerate import FutureRate from axle import fetch_axle_sessions, load_axle_slot, fetch_axle_active @@ -172,45 +172,66 @@ def step_data_history( return values - def get_filtered_load_minute(self, data, minute_previous, historical, step=1): - """ - Gets a previous load minute after filtering for car charging - """ - load_yesterday_raw = 0 - - for offset in range(step): - if historical: - load_yesterday_raw += self.get_historical(data, minute_previous + offset) - else: - load_yesterday_raw += self.get_from_incrementing(data, minute_previous + offset) - - load_yesterday = load_yesterday_raw - - # Subtract car charging energy and iboost energy (if enabled) - subtract_energy = 0 - for offset in range(step): - if historical: - if self.car_charging_hold and self.car_charging_energy: - subtract_energy += self.get_historical(self.car_charging_energy, minute_previous + offset) - if self.iboost_energy_subtract and self.iboost_energy_today: - subtract_energy += self.get_historical(self.iboost_energy_today, minute_previous + offset) + def get_filtered_load_window(self, data, indices, step): + """ + Compute the car-charging/iBoost-filtered load for a single window given its list of backwards minute + indices. Returns (filtered_load, raw_load) for that one window (no base-load applied). + """ + raw = 0.0 + subtract_energy = 0.0 + for idx in indices: + raw += self.get_from_incrementing(data, idx) + if self.car_charging_hold and self.car_charging_energy: + subtract_energy += self.get_from_incrementing(self.car_charging_energy, idx) + if self.iboost_energy_subtract and self.iboost_energy_today: + subtract_energy += self.get_from_incrementing(self.iboost_energy_today, idx) + load = max(0.0, raw - subtract_energy) + if self.car_charging_hold and (not self.car_charging_energy) and (load >= (self.car_charging_threshold * step)): + # Car charging hold - ignore car charging in this window based on the threshold + load = max(load - (self.car_charging_rate[0] * step / 60.0), 0) + return load, raw + + def get_filtered_load_minute(self, data, minute_previous, historical, step=1, base_in_raw=True): + """ + Gets a previous load minute after filtering for car charging. + + For historical (days_previous) data the car-charging hold and car/iBoost subtraction are applied to + each previous day individually and only then weighted-averaged, so a single day's car charging is not + diluted below the hold threshold by averaging it with the other (lower-load) days first. + + base_in_raw controls whether the base-load top-up is reflected in the returned raw value. Set it to + False to keep the raw value at the true measured load so callers can detect genuine zero/gap buckets. + """ + if historical: + total = 0.0 + total_raw = 0.0 + total_weight = 0.0 + for point, days in enumerate(self.days_previous): + use_days = max(min(days, self.load_minutes_age), 1) + weight = self.days_previous_weight[point] + full_days = 24 * 60 * (use_days - 1) + indices = [24 * 60 - (minute_previous + offset) + full_days for offset in range(step)] + day_load, day_raw = self.get_filtered_load_window(data, indices, step) + total += day_load * weight + total_raw += day_raw * weight + total_weight += weight + if total_weight > 0: + load_yesterday = total / total_weight + load_yesterday_raw = total_raw / total_weight else: - if self.car_charging_hold and self.car_charging_energy: - subtract_energy += self.get_from_incrementing(self.car_charging_energy, minute_previous + offset) - if self.iboost_energy_subtract and self.iboost_energy_today: - subtract_energy += self.get_from_incrementing(self.iboost_energy_today, minute_previous + offset) - load_yesterday = max(0, load_yesterday - subtract_energy) - - if self.car_charging_hold and (not self.car_charging_energy) and (load_yesterday >= (self.car_charging_threshold * step)): - # Car charging hold - ignore car charging in computation based on threshold - load_yesterday = max(load_yesterday - (self.car_charging_rate[0] * step / 60.0), 0) + load_yesterday = 0.0 + load_yesterday_raw = 0.0 + else: + indices = [minute_previous + offset for offset in range(step)] + load_yesterday, load_yesterday_raw = self.get_filtered_load_window(data, indices, step) # Apply base load base_load = self.base_load * step / 60.0 if load_yesterday < base_load: add_to_base = base_load - load_yesterday load_yesterday += add_to_base - load_yesterday_raw += add_to_base + if base_in_raw: + load_yesterday_raw += add_to_base return load_yesterday, load_yesterday_raw @@ -1029,11 +1050,24 @@ def fetch_sensor_data(self, save=True): # Fetch PV forecast if enabled, today must be enabled, other days are optional self.pv_forecast_minute, self.pv_forecast_minute10 = self.fetch_pv_forecast() - if self.load_minutes and not self.load_forecast_only: - # Apply modal filter to historical data + if self.load_minutes and not self.load_forecast_only and not self.load_forecast_history: + # Apply modal filter to historical data. Skipped for days_previous_auto: the weighted-bucket + # forecast deliberately ignores missing-data buckets, so the gap-filling/padding the modal filter + # applies would mask exactly the gaps it is designed to exclude. self.previous_days_modal_filter(self.load_minutes) self.log("Historical days now {} weight {}".format(self.days_previous, self.days_previous_weight)) + # Weighted-bucket historical load forecast (days_previous_auto). Built here as it needs load_minutes + # after the power fill. load_ml takes precedence if it already set the forecast-only load source. + if self.load_forecast_history and self.load_minutes and self.load_minutes_age >= 1 and not self.load_forecast_only: + hist_forecast = self.compute_load_forecast_history(self.now_utc) + if hist_forecast: + self.load_forecast_only = True + for minute, value in hist_forecast.items(): + self.load_forecast[minute] = self.load_forecast.get(minute, 0) + value + self.load_forecast_array.append(hist_forecast) + self.log("Using weighted-bucket historical load forecast over {} days".format(min(self.load_minutes_age, self.max_days_previous - 1))) + # Load today vs actual if self.load_minutes: self.load_inday_adjustment = self.load_today_comparison(self.load_minutes, self.load_forecast, self.car_charging_energy, self.import_today, self.minutes_now, save=save) @@ -1972,6 +2006,119 @@ def fetch_ml_load_forecast(self, now_utc): return load_forecast return {} + def get_holiday_minutes(self, now_utc, num_days): + """ + Build a per-minute history of the holiday_days_left value (indexed by minutes-ago) from the recorded + entity history using minute_data, which holds each state forward until the next change. + + Returns the minute_data dict, or None when no usable history is available. + """ + item = self.config_index.get("holiday_days_left", None) if getattr(self, "config_index", None) else None + entity_id = item.get("entity") if item else ("input_number." + self.prefix + "_holiday_days_left") + if not entity_id: + return None + + try: + history = self.get_history_wrapper(entity_id=entity_id, days=num_days, required=False) + except (ValueError, TypeError): + history = None + + if not history or not isinstance(history, list) or not history[0]: + return None + + holiday_minutes, _ = minute_data(history[0], num_days, now_utc, "state", "last_updated", backwards=True) + return holiday_minutes or None + + def compute_load_forecast_history(self, now_utc): + """ + Build a forward load estimate from all available history (up to LOAD_FORECAST_HISTORY_MAX_DAYS days) + using per-5-minute weighted-bucket averaging. + + For each forward 5-minute slot the historical sample at the same time-of-day is gathered from each + available past day and combined as a weighted average, ignoring zero (missing-data) buckets entirely. + Per-sample weight = weekday_factor * holiday_factor * age_factor: + - weekday_factor: 1.0 if the historical day is the same weekday as today; else 0.7 if both are + weekend or both are weekday; else 0.5 (one weekday, one weekend). + - holiday_factor: 1.0 if the holiday state when that individual 5-minute sample was recorded matches + today's holiday state; else 0.5. This is matched per sample (not per whole day) so a mid-day change + of holiday mode is handled correctly. + - age_factor: 0.9 for yesterday, reducing by 0.03 per day down to a floor of 0.1. + + Returns a cumulative-from-midnight kWh dict (same format as the ML load forecast), or {} if no data. + """ + if not self.load_minutes or self.load_minutes_age < 1: + return {} + # Search window comes from max_days_previous (derived from days_previous), bounded by available history + num_days = min(self.load_minutes_age, self.max_days_previous - 1) + if num_days < 1: + return {} + + today_dow = now_utc.weekday() + today_holiday = self.holiday_days_left > 0 + holiday_minutes = self.get_holiday_minutes(now_utc, num_days) + max_holiday_index = num_days * 24 * 60 - 1 + + # Precompute the static per-day weight (weekday * age); the holiday factor is applied per 5-minute bucket + day_static_weight = {} + for d in range(1, num_days + 1): + hist_dow = (now_utc - timedelta(days=d)).weekday() + if hist_dow == today_dow: + weekday_factor = 1.0 + elif (hist_dow >= 5) == (today_dow >= 5): + # Both weekend or both weekday + weekday_factor = 0.7 + else: + weekday_factor = 0.5 + # Age: 0.9 for yesterday (d=1), reducing by 0.03 per day down to a floor of 0.1 + age_factor = max(0.1, 0.9 - (d - 1) * 0.03) + day_static_weight[d] = weekday_factor * age_factor + + # Build the per-step (5-minute) weighted-average estimate, keyed by minute-from-midnight, across the + # whole horizon from midnight today through the end of the plan so the resulting array is genuinely + # cumulative-from-midnight (consumers such as load_today_comparison read minutes before minutes_now too). + horizon_end = self.minutes_now + self.forecast_minutes + self.plan_interval_minutes + per_step = {} + for minute_absolute in range(0, horizon_end, PREDICT_STEP): + tod = minute_absolute % (24 * 60) # time of day of this slot + total = 0.0 + total_weight = 0.0 + for d in range(1, num_days + 1): + # Sample d whole days ago at this slot's time of day. Counting in whole days from today keeps + # each day distinct and handles midnight crossings (the slot may be tomorrow or later). + minute_previous = (self.minutes_now - tod) + d * 24 * 60 + # Skip zero (missing-data) buckets entirely so gaps in the history do not drag the estimate down. + # base_in_raw=False keeps raw at the true measured load so genuine gaps are detected even when + # a base load is configured (the filtered sample still has the base load applied). + sample, raw = self.get_filtered_load_minute(self.load_minutes, minute_previous, historical=False, step=PREDICT_STEP, base_in_raw=False) + if raw <= 0: + continue + # Match the holiday state at the moment this individual sample was recorded (per bucket). If + # the sample is older than the holiday history we have, treat it as matching today (neutral) + # rather than reusing the oldest known state. + if holiday_minutes is None or minute_previous > max_holiday_index: + holiday_active = today_holiday + else: + holiday_active = holiday_minutes.get(minute_previous, 0) > 0 + holiday_factor = 1.0 if (holiday_active == today_holiday) else 0.5 + weight = day_static_weight[d] * holiday_factor + total += sample * weight + total_weight += weight + per_step[minute_absolute] = (total / total_weight) if total_weight > 0 else 0.0 + + # Convert per-step kWh buckets into a cumulative-from-midnight dict, filling every minute by linear + # interpolation across each 5-minute span so get_from_incrementing(..., backwards=False) reproduces + # the per-step energy exactly (mirroring the dense output of minute_data with smoothing). + load_forecast = {} + cumulative = 0.0 + for minute_absolute in range(0, horizon_end, PREDICT_STEP): + step_energy = per_step[minute_absolute] + for offset in range(PREDICT_STEP): + load_forecast[minute_absolute + offset] = dp4(cumulative + step_energy * offset / PREDICT_STEP) + cumulative += step_energy + # Final boundary so the last span's increment is well defined + load_forecast[horizon_end] = dp4(cumulative) + return load_forecast + def fetch_extra_load_forecast(self, now_utc, ml_forecast=None): """ Fetch extra load forecast, this is future load data @@ -2124,10 +2271,21 @@ def fetch_config_options(self): if len(self.days_previous) > len(self.days_previous_weight): # Extend weights with 1 if required self.days_previous_weight += [1 for i in range(len(self.days_previous) - len(self.days_previous_weight))] - if self.holiday_days_left > 0: + + # days_previous_auto enables the weighted-bucket historical load forecast. The number of days of history + # it searches comes from max(days_previous) (or 7 when days_previous is not set), capped to the history + # Predbat can hold (LOAD_FORECAST_HISTORY_MAX_DAYS). + self.load_forecast_history = self.get_arg("days_previous_auto", False) + if self.load_forecast_history: + window_days = min(max(self.days_previous) if self.days_previous else 7, LOAD_FORECAST_HISTORY_MAX_DAYS) + self.log("days_previous_auto enabled - using weighted-bucket historical load forecast over up to {} days".format(window_days)) + self.max_days_previous = window_days + 1 + elif self.holiday_days_left > 0: self.days_previous = [1] self.log("Holiday mode is active, {} days remaining, setting days previous to 1".format(self.holiday_days_left)) - self.max_days_previous = max(self.days_previous) + 1 + self.max_days_previous = max(self.days_previous) + 1 + else: + self.max_days_previous = max(self.days_previous) + 1 self.forecast_days = int((forecast_hours + 23) / 24) self.forecast_minutes = forecast_hours * 60 diff --git a/apps/predbat/predbat.py b/apps/predbat/predbat.py index cb8922e4a..88296f798 100644 --- a/apps/predbat/predbat.py +++ b/apps/predbat/predbat.py @@ -35,7 +35,7 @@ import pytz import asyncio -THIS_VERSION = "v8.40.16" +THIS_VERSION = "v8.41.0" from download import predbat_update_move, predbat_update_download, check_install, DEFAULT_PREDBAT_REPOSITORY from const import MINUTE_WATT @@ -459,10 +459,10 @@ def reset(self): self.export_limits = [] self.export_limits_best = [] self.export_window_best = [] - self.battery_rate_max_charge = 0 + self.battery_rate_max_charge = 0.0333 self.battery_rate_max_charge_dc = 0 - self.battery_rate_max_discharge = 0 - self.battery_rate_max_export = 0 + self.battery_rate_max_discharge = 0.0333 + self.battery_rate_max_export = 0.0333 self.battery_rate_min = 0 self.battery_rate_max_scaling = 1.0 self.battery_rate_max_scaling_discharge = 1.0 @@ -591,6 +591,7 @@ def reset(self): self.inverter_data_last_fetch = None self.octopus_url_cache_loaded = False self.github_url_cache_loaded = False + self.load_forecast_history = False for root in CONFIG_ROOTS: if os.path.exists(root): diff --git a/apps/predbat/tests/test_filtered_load_minute.py b/apps/predbat/tests/test_filtered_load_minute.py new file mode 100644 index 000000000..f370bd060 --- /dev/null +++ b/apps/predbat/tests/test_filtered_load_minute.py @@ -0,0 +1,127 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long +# pylint: disable=attribute-defined-outside-init + +from utils import MinuteArray + + +def build_cumulative(increments, size): + """Build a backwards cumulative MinuteArray so get_from_incrementing(data, i) == increments[i].""" + data = {} + data[size - 1] = 0.0 + for i in range(size - 2, -1, -1): + data[i] = data[i + 1] + increments.get(i, 0.0) + return MinuteArray(data, size) + + +def reset(my_predbat): + """Reset the load-filtering related attributes to a clean (no filtering) baseline.""" + my_predbat.car_charging_hold = False + my_predbat.car_charging_energy = None + my_predbat.iboost_energy_subtract = False + my_predbat.iboost_energy_today = None + my_predbat.car_charging_threshold = 0.1 + my_predbat.car_charging_rate = [6.0] + my_predbat.base_load = 0.0 + + +def check(name, got, expected, tol=1e-9): + """Compare a value to its expectation, printing and returning failure state.""" + if abs(got - expected) > tol: + print("ERROR: {} expected {:.5f} got {:.5f}".format(name, expected, got)) + return True + print("OK: {} = {:.5f}".format(name, got)) + return False + + +def test_filtered_load_minute(my_predbat): + """ + Test get_filtered_load_window and get_filtered_load_minute (raw, car/iBoost filtering, averaging, base load). + """ + print("**** Running filtered_load_minute tests ****") + failed = False + + # --------------------------------------------------------------- + # get_filtered_load_window + # --------------------------------------------------------------- + reset(my_predbat) + data = build_cumulative({10: 0.1, 11: 0.2, 12: 0.3}, 40) + + # Raw, no filtering -> load == raw == sum of the window + load, raw = my_predbat.get_filtered_load_window(data, [10, 11, 12], 3) + failed |= check("window raw load", load, 0.6) + failed |= check("window raw value", raw, 0.6) + + # Car charging energy subtraction (per measured car energy) + reset(my_predbat) + my_predbat.car_charging_hold = True + my_predbat.car_charging_energy = build_cumulative({10: 0.05, 11: 0.05}, 40) + load, raw = my_predbat.get_filtered_load_window(data, [10, 11, 12], 3) + failed |= check("window minus car energy", load, 0.5) + failed |= check("window raw unchanged", raw, 0.6) + + # iBoost subtraction + reset(my_predbat) + my_predbat.iboost_energy_subtract = True + my_predbat.iboost_energy_today = build_cumulative({12: 0.2}, 40) + load, _ = my_predbat.get_filtered_load_window(data, [10, 11, 12], 3) + failed |= check("window minus iboost", load, 0.4) + + # Car charging hold threshold (no measured car energy): subtract the car rate when above the threshold + reset(my_predbat) + my_predbat.car_charging_hold = True + my_predbat.car_charging_threshold = 0.1 # per minute -> window threshold 0.1 * step = 0.3 + my_predbat.car_charging_rate = [6.0] # 6 kW -> 6 * 3 / 60 = 0.3 kWh removed + load, _ = my_predbat.get_filtered_load_window(data, [10, 11, 12], 3) # raw 0.6 >= 0.3 + failed |= check("window car hold threshold", load, 0.3) + + # Below threshold -> untouched + small = build_cumulative({10: 0.05, 11: 0.05}, 40) + load, _ = my_predbat.get_filtered_load_window(small, [10, 11, 12], 3) # raw 0.1 < 0.3 + failed |= check("window below threshold kept", load, 0.1) + + # --------------------------------------------------------------- + # get_filtered_load_minute - non-historical (single window) + # --------------------------------------------------------------- + reset(my_predbat) + data = build_cumulative({5: 0.2, 6: 0.2}, 40) + load, _ = my_predbat.get_filtered_load_minute(data, 5, historical=False, step=2) + failed |= check("non-historical single window", load, 0.4) + + # --------------------------------------------------------------- + # get_filtered_load_minute - historical weighted average across days + # --------------------------------------------------------------- + reset(my_predbat) + my_predbat.days_previous = [1, 2] + my_predbat.days_previous_weight = [1.0, 3.0] + my_predbat.load_minutes_age = 2 + # At minute 0, step 2: day1 indices [1440, 1439], day2 indices [2880, 2879] + hist = build_cumulative({1440: 0.2, 1439: 0.2, 2880: 0.4, 2879: 0.4}, 2 * 1440 + 10) + # day1 window = 0.4, day2 window = 0.8 -> weighted (0.4*1 + 0.8*3) / 4 = 0.7 + load, _ = my_predbat.get_filtered_load_minute(hist, 0, historical=True, step=2) + failed |= check("historical weighted average", load, 0.7) + + # --------------------------------------------------------------- + # Base load floor and base_in_raw flag + # --------------------------------------------------------------- + reset(my_predbat) + my_predbat.base_load = 0.1 # kW -> floor 0.1 * step / 60 + floor = 0.1 * 2 / 60.0 + low = build_cumulative({5: 0.0005, 6: 0.0005}, 40) # window 0.001 < floor + load, raw = my_predbat.get_filtered_load_minute(low, 5, historical=False, step=2, base_in_raw=True) + failed |= check("base load floor", load, floor) + failed |= check("base in raw on", raw, floor) + load, raw = my_predbat.get_filtered_load_minute(low, 5, historical=False, step=2, base_in_raw=False) + failed |= check("base load floor (raw kept)", load, floor) + failed |= check("base in raw off keeps measured", raw, 0.001) + + # Restore the shared predbat config so this test does not leak load-filtering state into later tests + my_predbat.fetch_config_options() + + return failed diff --git a/apps/predbat/tests/test_load_forecast_history.py b/apps/predbat/tests/test_load_forecast_history.py new file mode 100644 index 000000000..bea0377f6 --- /dev/null +++ b/apps/predbat/tests/test_load_forecast_history.py @@ -0,0 +1,416 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long +# pylint: disable=attribute-defined-outside-init + +from datetime import datetime, timezone, timedelta + +from utils import MinuteArray +from const import PREDICT_STEP + + +def build_load_minutes(day_rates, extra_minutes=10): + """ + Build a backwards-indexed cumulative load MinuteArray from per-day constant rates. + + day_rates maps a 1-based day number (1 = yesterday) to a constant kWh-per-minute rate for that day. + The returned array satisfies get_from_incrementing(data, i) == rate of the day that index i belongs to. + """ + num_days = max(day_rates.keys()) + size = num_days * 24 * 60 + extra_minutes + + def inc(i): + day = ((max(i, 1) - 1) // (24 * 60)) + 1 + return day_rates.get(day, 0.0) + + data = {} + data[size - 1] = 0.0 + for i in range(size - 2, -1, -1): + data[i] = data[i + 1] + inc(i) + return MinuteArray(data, size) + + +def expected_day_weight(now_utc, day, holiday_factor=1.0): + """Re-implement the static weekday*age weighting (holiday applied separately) to validate the forecast.""" + today_dow = now_utc.weekday() + hist_dow = (now_utc - timedelta(days=day)).weekday() + if hist_dow == today_dow: + weekday_factor = 1.0 + elif (hist_dow >= 5) == (today_dow >= 5): + weekday_factor = 0.7 + else: + weekday_factor = 0.5 + age_factor = max(0.1, 0.9 - (day - 1) * 0.03) + return weekday_factor * holiday_factor * age_factor + + +def step_energy_at(load_forecast, minute_absolute): + """Reproduce the consumer's read of a 5-minute bucket from the cumulative forecast dict.""" + total = 0.0 + for offset in range(PREDICT_STEP): + idx = minute_absolute + offset + total += max(load_forecast.get(idx + 1, 0) - load_forecast.get(idx, 0), 0) + return total + + +def setup_predbat(my_predbat, now_utc): + """Configure my_predbat for a clean deterministic forecast computation.""" + my_predbat.now_utc = now_utc + my_predbat.minutes_now = 0 + my_predbat.forecast_minutes = 120 + my_predbat.plan_interval_minutes = 30 + my_predbat.holiday_days_left = 0 + my_predbat.base_load = 0.0 + my_predbat.car_charging_hold = False + my_predbat.car_charging_energy = None + my_predbat.iboost_energy_subtract = False + my_predbat.iboost_energy_today = None + my_predbat.max_days_previous = 31 # search window cap; num_days = min(load_minutes_age, max_days_previous - 1) + + +def test_load_forecast_history(my_predbat): + """ + Test the weighted-bucket historical load forecast (days_previous: all). + """ + print("**** Running load_forecast_history tests ****") + failed = False + + now_utc = datetime(2026, 6, 17, 0, 0, 0, tzinfo=timezone.utc) # Wednesday + original_get_holiday_minutes = my_predbat.get_holiday_minutes + original_args = dict(my_predbat.args) if hasattr(my_predbat, "args") else {} + + # --------------------------------------------------------------- + # Test 1: combined weekday + age weighting end-to-end (holiday neutral) + # --------------------------------------------------------------- + print("Test 1: weekday/age weighting") + setup_predbat(my_predbat, now_utc) + num_days = 12 + day_rates = {d: 0.001 * d for d in range(1, num_days + 1)} # distinct constant rates + my_predbat.load_minutes = build_load_minutes(day_rates) + my_predbat.load_minutes_age = num_days + + my_predbat.get_holiday_minutes = lambda now, n: None # no holiday history -> neutral holiday factor (1.0) + + forecast = my_predbat.compute_load_forecast_history(now_utc) + + # Expected per-step value (constant across slots since each day's rate is constant) + num = 0.0 + den = 0.0 + for d in range(1, num_days + 1): + w = expected_day_weight(now_utc, d) + sample = 5 * day_rates[d] # 5-minute energy, base_load = 0, no subtraction + num += sample * w + den += w + expected = num / den + + # Tolerance allows for dp4 quantization of the cumulative forecast (the consumer reads 4-dp values) + for m in (5, 30, 60, 120): + actual = step_energy_at(forecast, m) + if abs(actual - expected) > 2e-4: + print("ERROR: slot {} expected {} got {}".format(m, expected, actual)) + failed = True + if not failed: + print("Weighting end-to-end correct: {} kWh per 5 min".format(round(expected, 6))) + + # --------------------------------------------------------------- + # Test 2: zero buckets excluded from numerator AND denominator + # --------------------------------------------------------------- + print("Test 2: zero-bucket exclusion") + setup_predbat(my_predbat, now_utc) + day_rates = {1: 0.002, 2: 0.0, 3: 0.004} # day 2 entirely missing/zero + my_predbat.load_minutes = build_load_minutes(day_rates) + my_predbat.load_minutes_age = 3 + my_predbat.get_holiday_minutes = lambda now, n: None # neutral holiday factor + + forecast = my_predbat.compute_load_forecast_history(now_utc) + + num = 0.0 + den = 0.0 + for d in (1, 3): # day 2 excluded + w = expected_day_weight(now_utc, d) + num += 5 * day_rates[d] * w + den += w + expected = num / den + actual = step_energy_at(forecast, 30) + if abs(actual - expected) > 2e-4: + print("ERROR: zero-bucket exclusion expected {} got {}".format(expected, actual)) + failed = True + else: + print("Zero buckets correctly excluded from both numerator and denominator") + + # Same case but with a non-zero base load: the gap (day 2) must still be excluded rather than + # filled with the base load. Rates are above the base-load floor so days 1/3 are unaffected. + setup_predbat(my_predbat, now_utc) + my_predbat.base_load = 0.05 # kW; floor = 0.05 * 5 / 60 ~= 0.0042 kWh per 5 min + my_predbat.load_minutes = build_load_minutes(day_rates) + my_predbat.load_minutes_age = 3 + my_predbat.get_holiday_minutes = lambda now, n: None + forecast = my_predbat.compute_load_forecast_history(now_utc) + actual = step_energy_at(forecast, 30) + if abs(actual - expected) > 2e-4: + print("ERROR: zero-bucket exclusion with base load expected {} got {}".format(expected, actual)) + failed = True + else: + print("Zero buckets still excluded (not filled with base load) when a base load is configured") + + # All-zero history -> forecast all zeros + setup_predbat(my_predbat, now_utc) + my_predbat.load_minutes = build_load_minutes({1: 0.0}) + my_predbat.load_minutes_age = 1 + my_predbat.get_holiday_minutes = lambda now, n: None + forecast = my_predbat.compute_load_forecast_history(now_utc) + if any(abs(step_energy_at(forecast, m)) > 1e-9 for m in (5, 30, 60)): + print("ERROR: all-zero history should produce a zero forecast") + failed = True + else: + print("All-zero history produced a zero forecast") + + # --------------------------------------------------------------- + # Test 3: age weighting clamps and nearer days dominate + # --------------------------------------------------------------- + print("Test 3: age weighting") + age_d1 = max(0.1, 0.9 - (1 - 1) * 0.03) + age_d10 = max(0.1, 0.9 - (10 - 1) * 0.03) + age_d28 = max(0.1, 0.9 - (28 - 1) * 0.03) + age_d40 = max(0.1, 0.9 - (40 - 1) * 0.03) + if abs(age_d1 - 0.9) > 1e-9 or abs(age_d10 - 0.63) > 1e-9 or abs(age_d28 - 0.1) > 1e-9 or abs(age_d40 - 0.1) > 1e-9: + print("ERROR: age factors wrong d1={} d10={} d28={} d40={}".format(age_d1, age_d10, age_d28, age_d40)) + failed = True + else: + print("Age factors correct (d1=0.9, -0.03/day, d10=0.63, floor 0.1 reached by ~d28)") + + # --------------------------------------------------------------- + # Test 4: short history / gaps - only available days used, no padding + # --------------------------------------------------------------- + print("Test 4: short history") + setup_predbat(my_predbat, now_utc) + # Provide many days of data but limit the reported age to 3 days + day_rates = {d: 0.001 * d for d in range(1, 13)} + my_predbat.load_minutes = build_load_minutes(day_rates) + my_predbat.load_minutes_age = 3 + my_predbat.get_holiday_minutes = lambda now, n: None + + forecast = my_predbat.compute_load_forecast_history(now_utc) + num = 0.0 + den = 0.0 + for d in (1, 2, 3): + w = expected_day_weight(now_utc, d) + num += 5 * day_rates[d] * w + den += w + expected = num / den + actual = step_energy_at(forecast, 30) + if abs(actual - expected) > 2e-4: + print("ERROR: short history expected only 3 days {} got {}".format(expected, actual)) + failed = True + else: + print("Short history correctly used only {} available days".format(my_predbat.load_minutes_age)) + + # Zero age -> empty forecast + my_predbat.load_minutes_age = 0 + if my_predbat.compute_load_forecast_history(now_utc) != {}: + print("ERROR: zero age should return empty forecast") + failed = True + else: + print("Zero age returned empty forecast") + + # --------------------------------------------------------------- + # Test 5: get_holiday_minutes reconstructs state-at-time via minute_data + # --------------------------------------------------------------- + print("Test 5: get_holiday_minutes") + my_predbat.get_holiday_minutes = original_get_holiday_minutes + my_predbat.holiday_days_left = 0 + holiday_now = datetime(2026, 6, 17, 12, 0, 0, tzinfo=timezone.utc) + + records = [ + {"state": "3", "last_updated": datetime(2026, 6, 10, 0, 0, 0, tzinfo=timezone.utc).isoformat()}, + {"state": "0", "last_updated": datetime(2026, 6, 14, 0, 0, 0, tzinfo=timezone.utc).isoformat()}, + ] + original_history = my_predbat.get_history_wrapper + my_predbat.get_history_wrapper = lambda entity_id, days=30, required=True, tracked=True: [records] + + holiday_minutes = my_predbat.get_holiday_minutes(holiday_now, 7) + # holiday on from 06-10 00:00 to 06-14 00:00; sample the state at noon of each historical day + expected_state = {1: False, 2: False, 3: False, 4: True, 5: True, 6: True} + holiday_map = {d: (holiday_minutes.get(d * 24 * 60, 0) > 0) for d in expected_state} + if holiday_map != expected_state: + print("ERROR: holiday map {} != expected {}".format(holiday_map, expected_state)) + failed = True + else: + print("Holiday history reconstructed correctly: {}".format(holiday_map)) + + # Empty history -> None so the holiday factor becomes neutral + my_predbat.get_history_wrapper = lambda entity_id, days=30, required=True, tracked=True: None + if my_predbat.get_holiday_minutes(holiday_now, 5) is not None: + print("ERROR: empty history should return None") + failed = True + else: + print("Empty history returns None (neutral holiday factor)") + my_predbat.get_history_wrapper = original_history + my_predbat.holiday_days_left = 0 + + # --------------------------------------------------------------- + # Test 5b: holiday weighting is matched per 5-minute bucket (mid-day toggle) + # --------------------------------------------------------------- + print("Test 5b: per-bucket holiday weighting") + + # Per-minute holiday history (minutes-ago -> holiday_days_left). Holiday active only for samples + # in yesterday's afternoon/evening (minutes-ago 1..720 == 06-16 12:00 to 06-17 00:00). + toggle_minutes = {ma: (3.0 if 1 <= ma <= 720 else 0.0) for ma in range(3 * 24 * 60)} + + # End-to-end: yesterday (d=1) had a holiday toggle mid-day; today is not on holiday. + # Morning buckets match today's state (factor 1.0), afternoon buckets do not (factor 0.5), + # so a morning slot and an afternoon slot must produce different weighted averages. + setup_predbat(my_predbat, now_utc) + my_predbat.forecast_minutes = 24 * 60 # cover a full day so afternoon slots map within yesterday + day_rates = {1: 0.002, 2: 0.004} # distinct so the holiday weight shift is observable + my_predbat.load_minutes = build_load_minutes(day_rates) + my_predbat.load_minutes_age = 2 + my_predbat.get_holiday_minutes = lambda now, n: toggle_minutes + + forecast = my_predbat.compute_load_forecast_history(now_utc) + + def expected_with_holiday(slot_minute): + num = 0.0 + den = 0.0 + for d in (1, 2): + minute_previous = 24 * 60 - slot_minute + 24 * 60 * (d - 1) + holiday_active = toggle_minutes.get(minute_previous, 0) > 0 + holiday_factor = 1.0 if (holiday_active == False) else 0.5 # today_holiday is False + w = expected_day_weight(now_utc, d, holiday_factor=holiday_factor) + num += 5 * day_rates[d] * w + den += w + return num / den + + morning = step_energy_at(forecast, 300) # 05:00 yesterday -> before toggle + afternoon = step_energy_at(forecast, 900) # 15:00 yesterday -> after toggle + if abs(morning - expected_with_holiday(300)) > 2e-4: + print("ERROR: morning bucket expected {} got {}".format(expected_with_holiday(300), morning)) + failed = True + if abs(afternoon - expected_with_holiday(900)) > 2e-4: + print("ERROR: afternoon bucket expected {} got {}".format(expected_with_holiday(900), afternoon)) + failed = True + if abs(morning - afternoon) < 1e-6: + print("ERROR: per-bucket holiday weighting had no effect (morning == afternoon)") + failed = True + if not failed: + print("Per-bucket holiday weighting applied correctly (morning {} != afternoon {})".format(round(morning, 6), round(afternoon, 6))) + + my_predbat.get_holiday_minutes = original_get_holiday_minutes + + # --------------------------------------------------------------- + # Test 6: activation via days_previous_auto + # --------------------------------------------------------------- + print("Test 6: activation via days_previous_auto") + my_predbat.args["days_previous"] = [14] + my_predbat.args["days_previous_auto"] = True + my_predbat.fetch_config_options() + if not getattr(my_predbat, "load_forecast_history", False): + print("ERROR: days_previous_auto did not enable load_forecast_history") + failed = True + elif my_predbat.max_days_previous != 15: + print("ERROR: window from max(days_previous) wrong, max_days_previous {} != 15".format(my_predbat.max_days_previous)) + failed = True + else: + print("days_previous_auto enabled forecast mode, window from max(days_previous), max_days_previous={}".format(my_predbat.max_days_previous)) + + # Window defaults to 7 when days_previous is not set + my_predbat.args["days_previous"] = [7] + my_predbat.fetch_config_options() + if my_predbat.max_days_previous != 8: + print("ERROR: default window expected max_days_previous 8 got {}".format(my_predbat.max_days_previous)) + failed = True + else: + print("Default days_previous gives a 7-day window") + + # Switch off leaves the flag off + my_predbat.args["days_previous_auto"] = False + my_predbat.fetch_config_options() + if getattr(my_predbat, "load_forecast_history", False): + print("ERROR: days_previous_auto off should not enable forecast mode") + failed = True + else: + print("days_previous_auto off leaves forecast mode off") + + # --------------------------------------------------------------- + # Test 7: car-charging hold is applied per day before averaging (legacy days_previous) + # --------------------------------------------------------------- + print("Test 7: per-day car-charging hold in legacy averaging") + setup_predbat(my_predbat, now_utc) + my_predbat.days_previous = [1, 2] + my_predbat.days_previous_weight = [1.0, 1.0] + my_predbat.load_minutes_age = 2 + my_predbat.car_charging_hold = True + my_predbat.car_charging_energy = None + my_predbat.car_charging_threshold = 0.1 # stored as kWh/min (i.e. 6 kW); window threshold = 0.1 * step + my_predbat.car_charging_rate = [7.0] + # Day 1 has a 7 kW (EV charge) constant load, day 2 a normal 0.5 kW load + my_predbat.load_minutes = build_load_minutes({1: 7.0 / 60.0, 2: 0.5 / 60.0}) + + load, _ = my_predbat.get_filtered_load_minute(my_predbat.load_minutes, 300, historical=True, step=PREDICT_STEP) + # Per-day: day1 5-min load 0.583 kWh >= 0.5 threshold -> car removed -> ~0; day2 0.0417 kWh kept. + # Correct (per-day) average = (0 + 0.0417) / 2 = 0.0208. The old (average-first) bug gave ~0.3125. + day2_window = 5 * (0.5 / 60.0) + expected = (0.0 + day2_window) / 2.0 + buggy = (5 * (7.0 / 60.0) + day2_window) / 2.0 + if abs(load - expected) > 2e-3: + print("ERROR: per-day car hold expected {:.4f} got {:.4f} (old buggy value was {:.4f})".format(expected, load, buggy)) + failed = True + else: + print("Car-charging hold correctly applied per day before averaging ({:.4f} kWh, not the diluted {:.4f})".format(load, buggy)) + + # --------------------------------------------------------------- + # Test 8: cumulative-from-midnight and midnight-crossing + # --------------------------------------------------------------- + print("Test 8: cumulative-from-midnight + midnight crossing") + setup_predbat(my_predbat, now_utc) + my_predbat.minutes_now = 600 + my_predbat.forecast_minutes = 1200 # horizon 600 + 1200 + 30 = 1830 crosses midnight + my_predbat.load_minutes = build_load_minutes({d: 0.002 for d in range(1, 8)}) + my_predbat.load_minutes_age = 6 + my_predbat.get_holiday_minutes = lambda now, n: None + forecast = my_predbat.compute_load_forecast_history(now_utc) + + # The array is genuinely cumulative from midnight: zero at minute 0 and populated before minutes_now + if abs(forecast.get(0, 0.0)) > 1e-9: + print("ERROR: cumulative should start at 0 at midnight, got {}".format(forecast.get(0))) + failed = True + elif forecast.get(my_predbat.minutes_now, 0) <= 0: + print("ERROR: earlier-today portion not populated (cumulative at minutes_now is 0)") + failed = True + else: + print("Cumulative-from-midnight: starts at 0, populated through minutes_now ({:.4f} kWh)".format(forecast.get(my_predbat.minutes_now))) + + # Midnight crossing: a tomorrow slot samples whole, distinct days at the slot's time of day + mn = my_predbat.minutes_now + minute_absolute = 1500 # tomorrow 01:00 (time of day 60) + tod = minute_absolute % (24 * 60) + prevs = [(mn - tod) + d * 24 * 60 for d in range(1, 5)] + if any(prevs[i + 1] - prevs[i] != 24 * 60 for i in range(len(prevs) - 1)): + print("ERROR: midnight-crossing samples not distinct whole days apart: {}".format(prevs)) + failed = True + else: + print("Midnight-crossing samples distinct days {} (1440 apart)".format(prevs)) + + # Same time-of-day today and tomorrow sample the same history, so (constant rate) give the same energy + energy_tomorrow = step_energy_at(forecast, 1500) # tomorrow 01:00 + energy_today = step_energy_at(forecast, 60) # today 01:00 (earlier today) + if energy_tomorrow <= 0 or abs(energy_tomorrow - energy_today) > 2e-4: + print("ERROR: tomorrow slot energy {} != same-time-of-day today {}".format(energy_tomorrow, energy_today)) + failed = True + else: + print("Tomorrow slot matches same-time-of-day today (constant rate): {:.5f}".format(energy_tomorrow)) + + # --------------------------------------------------------------- + # Restore mocks/state + # --------------------------------------------------------------- + my_predbat.get_holiday_minutes = original_get_holiday_minutes + my_predbat.args.clear() + my_predbat.args.update(original_args) + my_predbat.fetch_config_options() + + return failed diff --git a/apps/predbat/tests/test_single_debug.py b/apps/predbat/tests/test_single_debug.py index 184f9f4cc..902be7ecb 100644 --- a/apps/predbat/tests/test_single_debug.py +++ b/apps/predbat/tests/test_single_debug.py @@ -14,16 +14,38 @@ from tests.test_infra import reset_inverter -def run_single_debug(test_name, my_predbat, debug_file, expected_file=None, compare=False, debug=False): +def _dump_state_before_plan(my_predbat, filename): + """Dump a JSON-comparable snapshot of predbat's member variables for cross-run-context comparison. + + Only cleanly JSON-serialisable values are recorded as-is; everything else is recorded as its type name + (without memory addresses) so two snapshots can be diffed without noise. + """ + state = {} + for key in sorted(my_predbat.__dict__.keys()): + if key.startswith("__"): + continue + value = my_predbat.__dict__[key] + if callable(value): + continue + try: + json.dumps(value) + state[key] = value + except (TypeError, ValueError, OverflowError): + try: + state[key] = "<{} len={}>".format(type(value).__name__, len(value)) + except (TypeError, AttributeError): + state[key] = "<{}>".format(type(value).__name__) + with open(filename, "w") as handle: + json.dump(state, handle, indent=2, sort_keys=True) + + +def run_single_debug(test_name, my_predbat, debug_file, expected_file=None, compare=False, debug=False, redo=False): print("**** Running debug test {} ****\n".format(debug_file)) - if not expected_file: - re_do_rates = True - reset_load_model = True - reload_octopus_slots = True - else: - reset_load_model = False - re_do_rates = False - reload_octopus_slots = False + # Will recompute the rates, load model and octopus slots if redo is True. This is useful for debugging a single test case, but the + # debug_cases regression suite exercise the same code path and produce the same result. + re_do_rates = redo + reset_load_model = redo + reload_octopus_slots = redo load_override = 1.0 my_predbat.load_user_config() failed = False @@ -196,6 +218,9 @@ def run_single_debug(test_name, my_predbat, debug_file, expected_file=None, comp ## Calculate the plan my_predbat.plan_valid = False + # Dump the full predbat state just before calculate_plan so the same case can be compared across run + # contexts (full ./run_all suite vs standalone) to find any leaked/uninitialised state. + # _dump_state_before_plan(my_predbat, test_name + ".state.json") print("Re-calculate plan") my_predbat.calculate_plan(recompute=True, debug_mode=debug) print("Plan calculated") @@ -224,8 +249,10 @@ def run_single_debug(test_name, my_predbat, debug_file, expected_file=None, comp print("ERROR: Expected file {} does not exist".format(expected_file)) else: expected_data = json.loads(open(expected_file).read()) - expected_json = json.dumps(expected_data) - if actual_json != expected_json: + # Compare the parsed contents by value rather than the raw JSON strings. The plan values are + # numpy floats whose json repr differs from a plain Python float's, so a byte-for-byte string + # compare reports spurious mismatches even when the numbers are identical. + if json.loads(actual_json) != expected_data: print("ERROR: Actual plan does not match expected plan") failed = True # Write actual plan diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index 3d0ebfeeb..af2eb3bd6 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -73,6 +73,8 @@ from tests.test_override_time import test_get_override_time_from_string from tests.test_units import run_test_units from tests.test_previous_days_modal import test_previous_days_modal_filter +from tests.test_load_forecast_history import test_load_forecast_history +from tests.test_filtered_load_minute import test_filtered_load_minute from tests.test_fill_load_from_power import run_all_tests as test_fill_load_from_power from tests.test_fetch_pv_forecast import run_all_tests as test_fetch_pv_forecast from tests.test_octopus_free import test_octopus_free @@ -204,6 +206,8 @@ def main(): ("format_time_ago", test_format_time_ago, "Format time ago tests", False), ("override_time", test_get_override_time_from_string, "Override time from string tests", False), ("previous_days_modal", test_previous_days_modal_filter, "Previous days modal filter tests", False), + ("load_forecast_history", test_load_forecast_history, "Weighted historical load forecast tests", False), + ("filtered_load_minute", test_filtered_load_minute, "Filtered load minute / window tests", False), ("fill_load_from_power", test_fill_load_from_power, "Fill load from power sensor tests", False), ("fetch_pv_forecast", test_fetch_pv_forecast, "Fetch PV forecast with relative_time offset tests", False), # Octopus Energy URL/API tests @@ -329,6 +333,7 @@ def main(): parser = argparse.ArgumentParser(description="Predbat unit tests") parser.add_argument("--debug_file", action="store", help="Enable debug output") parser.add_argument("--full_debug", action="store_true", help="Enable full debug output") + parser.add_argument("--redo", action="store_true", help="Redo rates, load model and octopus slots for debug test") parser.add_argument("--compare", action="store_true", help="Run compare") parser.add_argument("--test", "-t", action="append", help="Run specific test(s) by name (can be used multiple times, use --list to see available tests)") parser.add_argument("--keyword", "-k", action="store", help="Run tests matching keyword pattern (e.g., -k carbon_ runs all carbon tests)") @@ -407,7 +412,7 @@ def main(): sys.exit(0) if args.debug_file: - run_single_debug(args.debug_file, my_predbat, args.debug_file, compare=args.compare, debug=args.full_debug) + run_single_debug(args.debug_file, my_predbat, args.debug_file, compare=args.compare, debug=args.full_debug, redo=args.redo) sys.exit(0) # Collect tests to run based on arguments diff --git a/coverage/cases/predbat_debug_pre_saving1.yaml.expected.json b/coverage/cases/predbat_debug_pre_saving1.yaml.expected.json index 1ef4f29eb..e3892f6c5 100644 --- a/coverage/cases/predbat_debug_pre_saving1.yaml.expected.json +++ b/coverage/cases/predbat_debug_pre_saving1.yaml.expected.json @@ -1 +1 @@ -{"charge_limit_best": [0.38, 0.38, 0.77, 9.52, 6.02, 9.52, 9.52, 9.52, 9.02, 0.38, 0.38, 0.38, 9.52, 9.52], "charge_window_best": [{"start": 1020, "end": 1080, "average": 25.58, "target": 0.38}, {"start": 1140, "end": 1320, "average": 25.58, "target": 0.38}, {"start": 1320, "end": 1380, "average": 25.58, "target": 0.77}, {"start": 1410, "end": 1530, "average": 7.0, "target": 5.424}, {"start": 1530, "end": 1560, "average": 7.0, "target": 6.02}, {"start": 1560, "end": 1630, "average": 7.0, "target": 8.91}, {"start": 1650, "end": 1720, "average": 7.0, "target": 9.509}, {"start": 1740, "end": 1770, "average": 7.0, "target": 9.332}, {"start": 1770, "end": 1800, "average": 25.58, "target": 9.02}, {"start": 1830, "end": 2100, "average": 25.58, "target": 0.38}, {"start": 2130, "end": 2160, "average": 25.58, "target": 0.38}, {"start": 2280, "end": 2430, "average": 25.58, "target": 0.38}, {"start": 2850, "end": 3210, "average": 7.0, "target": 9.52}, {"start": 3210, "end": 3900, "average": 25.58, "target": 9.52}], "export_window_best": [{"average": 75.0, "end": 1140, "start": 1080, "set": 69.8, "start_orig": 1080, "target": 9}, {"average": 15.0, "end": 1650, "start": 1630, "set": 14.0, "target": 84}, {"average": 15.0, "end": 1740, "start": 1720, "set": 14.0, "start_orig": 1710, "target": 90}], "export_limits_best": [4.0, 80.0, 86.0]} +{"charge_limit_best": [3.02, 0.38, 0.38, 9.52, 9.52, 9.52, 0.38, 0.38, 0.38, 7.02, 0.38, 9.52, 9.52], "charge_window_best": [{"start": 1020, "end": 1050, "average": 25.58, "target": 3.02}, {"start": 1050, "end": 1080, "average": 25.58, "target": 0.38}, {"start": 1170, "end": 1380, "average": 25.58, "target": 0.38}, {"start": 1410, "end": 1660, "average": 7.0, "target": 9.478}, {"start": 1680, "end": 1735, "average": 7.0, "target": 9.519}, {"start": 1740, "end": 1770, "average": 7.0, "target": 9.52}, {"start": 1830, "end": 2100, "average": 25.58, "target": 0.38}, {"start": 2160, "end": 2250, "average": 25.58, "target": 0.38}, {"start": 2310, "end": 2340, "average": 25.58, "target": 0.38}, {"start": 2340, "end": 2370, "average": 25.58, "target": 7.02}, {"start": 2370, "end": 2430, "average": 25.58, "target": 0.38}, {"start": 2850, "end": 3210, "average": 7.0, "target": 9.52}, {"start": 3210, "end": 3900, "average": 25.58, "target": 9.52}], "export_window_best": [{"average": 75.0, "end": 1140, "start": 1080, "set": 69.8, "start_orig": 1080, "target": 12}, {"average": 15.0, "end": 1680, "start": 1660, "set": 14.0, "target": 92}, {"average": 15.0, "end": 1740, "start": 1735, "set": 14.0, "start_orig": 1710, "target": 98}], "export_limits_best": [7.0, 88.0, 94.0]} diff --git a/docs/apps-yaml.md b/docs/apps-yaml.md index d48f88dc7..4de2b5a34 100644 --- a/docs/apps-yaml.md +++ b/docs/apps-yaml.md @@ -355,6 +355,37 @@ Or if you want Predbat to take the average of the same day for the last two week Further details and worked examples of [how days_previous works](#understanding-how-days_previous-works) are covered at the end of this document. +#### days_previous_auto (weighted historical load forecast) + +Setting **days_previous_auto** to `True` in `apps.yaml` switches house-load prediction from the fixed +list/weighting approach above to a weighted-bucket forecast: + +```yaml + days_previous_auto: True +``` + +In this mode Predbat ignores the fixed averaging and instead builds a forward load forecast from **all** of +the load history within the search window (without padding when fewer days exist). The window is taken from +`max(days_previous)`, or 7 days when `days_previous` is not set, capped at 30 days. This is more robust when +there are gaps in your history or when your usage pattern has changed (for example when returning from +holiday), because it no longer depends on a small number of specific days all being present and +representative. + +Each historical 5-minute sample is combined into a weighted average for the matching time-of-day, where the +weight of every sample is the product of three factors: + +- **Weekday** - 1.0 if the historical day is the same day of the week as today; 0.7 if it is a different day + but both are weekdays or both are weekend days; 0.5 if one is a weekday and the other a weekend day. +- **Holiday** - reduced by 50% if that historical day's [holiday mode](customisation.md#holiday-mode) state + does not match today's holiday mode state. The historical holiday state is reconstructed from the recorded + history of `holiday_days_left`. +- **Age** - 0.9 for yesterday, reducing by 0.03 per day down to a floor of 0.1 (reached after about a + month), so recent days count for more. + +Buckets with no recorded data (zero) are ignored entirely so gaps in the history do not drag the estimate +down. As with Load ML, this replaces the normal days_previous averaging; if [Load ML](load-ml.md) is enabled +it takes precedence over `days_previous_auto`. + Do keep in mind that Home Assistant only keeps 10 days of history by default, so if you want to access more than this for Predbat you might need to increase the number of days of history kept in HA before it is purged by editing and adding the following to the `/homeassistant/configuration.yaml` configuration file and restarting Home Assistant afterwards: