Make daily traffic scale to dozens of screens: incremental sync, ping keepalive, one action per message (0.2.39)

App 0.2.39 / firmware 0.2.33, items 1, 2, 4, 5 and 6 of the 30-screen scaling check.

- Only the tiles (and top bar) of changed entities are rebuilt; the screen list is
  cached until the registry or a screen diagnostic changes; the SSE stream follows.
- Sensor history comes from one recorder/statistics_during_period call per window in
  a background task, with the REST history as fallback per sensor without statistics.
- Firmware 0.2.33 answers a keepalive ping with its layout revision; a mismatch or a
  restart asks for the layout again, everything is repeated hourly as a safety net.
- The screen_message API action takes one whole message; the text inbox chunks stay
  as fallback for older firmware. A repeated unchanged layout keeps the inbox state.
- Quiet diagnostics: Opgestart (uptime as timestamp) replaces Uptime, heap every
  5 min, setting numbers publish only on change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

screen_manager/CHANGELOG.md

@@ -1,3 +1,14 @@
+## 0.2.39 (firmware 0.2.33)
+
+Zuiniger in dagelijks gebruik, ook met tientallen schermen. Niets verandert aan wat je ziet of instelt.
+
+- **Alleen doorrekenen wat veranderde.** Bij een statuswijziging bouwt de app alleen de tegels (en de bovenbalk) opnieuw waar die entiteit op staat; de andere berichten blijven zoals ze verstuurd zijn. De schermlijst wordt niet meer bij elke wijziging uit het hele Home Assistant-register opgebouwd, maar bewaard tot het register of een schermdiagnose verandert. De live-stream van de beheerpagina volgt dezelfde regel.
+- **Historie gebundeld, buiten de lus.** De 24 staafjes van sensortegels komen uit één `recorder/statistics_during_period`-vraag voor alle sensoren tegelijk (uurgemiddelden voor 24 uur, 5-minuutgemiddelden voor 1 en 6 uur), in een achtergrondtaak die elke vijf minuten ververst. Sensoren zonder statistiek (geen state class) houden de oude REST-historie, ook op de achtergrond. De verzendlus wacht nergens meer op de database; een gewijzigde grafiek stuurt alleen die tegel.
+- **Keepalive als ping.** Schermen met firmware 0.2.33 krijgen elke twee minuten alleen een klein bericht met de revisie van hun indeling. Komt die niet overeen (herstart, demo-indeling, verloren bericht), dan meldt het scherm `Indeling opnieuw nodig` en stuurt de app alles opnieuw; als vangnet gaat eens per uur toch alles. De inbox-status wisselt niet meer per ronde tussen "Indeling ontvangen" en "Gesynchroniseerd". Oudere firmware houdt de volledige herhaling per twee minuten.
+- **Eén actie per bericht.** Firmware 0.2.33 heeft de actie `esphome.<apparaatnaam>_screen_message` die een heel bericht in één keer aanneemt; de app gebruikt die zodra het scherm die firmware meldt. De base64-blokjes van 200 tekens in de tekstentiteit blijven de terugval voor oudere firmware.
+- **Rustige diagnostiek.** De diagnostische sensor `Uptime` (elke 15 s een nieuwe waarde) is vervangen door `Opgestart`, een tijdstempel met één waarde per opstart; Home Assistant ruimt de oude entiteit zelf op. De heap-sensoren melden per vijf minuten in plaats van per 30 s, en de vier instellingsgetallen (helderheid, standby) publiceren alleen nog een gewijzigde waarde. Dat scheelt tot honderdduizenden recorder-rijen per dag bij dertig schermen.
+- Firmware 0.2.33 voor beide borden; de app biedt de update aan. Alles werkt ook met de vorige firmware, alleen zonder ping en actie.
+
 ## 0.2.38 (firmware 0.2.32)
 
 - **Bovenbalk per scherm.** Bovenaan elk scherm staat links de naam en rechts wat jij kiest, tot zes onderdelen: de **tijd**, een **analoge klok**, de **datum**, of een **entiteit met icoon** uit Home Assistant, zoals temperatuur, luchtvochtigheid, verbruik, een deur of raam (open/dicht), het alarmpaneel, een slot, wie er thuis is (`zone.home`) of een telefoon (`device_tracker`). Zonder aanpassing blijft het zoals het was: naam en tijd.

screen_manager/app/core.py

@@ -1,6 +1,7 @@
 """Pure validation, firmware generation and bounded display protocol."""
 import base64
 from datetime import datetime, timedelta, timezone
+import hashlib
 import json
 import math
 import re
@@ -16,7 +17,7 @@ WEEKDAYS = ['ma', 'di', 'wo', 'do', 'vr', 'za', 'zo']
 REPO = 'https://github.com/MaxGramser/homeassistant_espscreen'
 REFS = {'cyd': 'main', 'guition': 'main'}
 # Firmware shipped with this app release; screens below it get an update offer.
-FIRMWARE_VERSION = '0.2.32'
+FIRMWARE_VERSION = '0.2.33'
 ATTRS = frozenset('brightness percentage current_position current_temperature temperature current_humidity min_temp max_temp target_temp_step supported_color_modes hvac_modes hvac_action hs_color color_temp_kelvin min_color_temp_kelvin max_color_temp_kelvin fan_speed_list unit_of_measurement battery_level fan_speed volume_level is_volume_muted media_title options min max step temperature_unit supported_features device_class next_rising next_setting finishes_at duration remaining humidity wind_speed wind_speed_unit apparent_temperature fan_modes swing_modes fan_mode swing_mode'.split())
 # Attributes whose boolean value the screen needs; every other bool stays behind.
 BOOL_ATTRS = frozenset(['is_volume_muted'])
@@ -109,6 +110,9 @@ def resolve_controls(tile):
     choice = options.get('controls', CONTROLS[domain][0][0])
     return None if choice == 'none' else choice
 
+# Diagnostic entities every ESP Screens firmware exposes; the manager watches them for screens.
+SCREEN_ENTITY_NAMES = frozenset({'Tegelinstellingen', 'Schermfirmware', 'Guition schermtype', 'Apparaatnaam', 'IP-adres'})
+
 # Additive schema 1 extension. An absent object retains old firmware/YAML defaults.
 SETTING_RULES = {
     'standby_enabled': (True, None, None),
@@ -152,6 +156,11 @@ def entity_id(value):
 # ----- Top bar (firmware 0.2.32+): the screen name on the left, up to six items on the right.
 # Without a `header` the screen keeps its name and the clock of `show_clock`. -----
 HEADER_MIN_FIRMWARE = (0, 2, 32)
+# Firmware 0.2.33+ takes a whole message in one API action (esphome.<node>_screen_message)
+# and answers a keepalive ping with its layout revision; older firmware gets base64
+# chunks in the text inbox and a full repeat every keepalive.
+TRANSPORT_MIN_FIRMWARE = (0, 2, 33)
+MESSAGE_ACTION = 'screen_message'
 HEADER_MAX_ITEMS = 6
 # Items the screen draws on its own clock, without Home Assistant.
 HEADER_BUILTIN = {'clock': 'Tijd', 'analog': 'Analoge klok', 'date': 'Datum'}
@@ -438,15 +447,28 @@ def state_message(index, tile, states, extra=None):
             'state': short(state.get('state', 'unavailable'), 160), 'a': bounded,
             **({'o': options} if options is not None else {}), **({'x': extra} if extra else {})}
 
-def packets(message, token=None):
-    raw = json.dumps(message, ensure_ascii=False, separators=(',', ':'), allow_nan=False).encode()
-    if len(raw) > 4096:
+def encode(message):
+    """The message as the firmware parses it: compact JSON, at most 4096 bytes."""
+    raw = json.dumps(message, ensure_ascii=False, separators=(',', ':'), allow_nan=False)
+    if len(raw.encode()) > 4096:
         raise ValueError('Het schermbericht is te groot.')
-    encoded = base64.b64encode(raw).decode('ascii')
+    return raw
+
+def revision(message):
+    """Short fingerprint of a layout message; the screen echoes it back on every ping."""
+    return hashlib.sha256(json.dumps(message, sort_keys=True, separators=(',', ':')).encode()).hexdigest()[:12]
+
+def packets(message, token=None):
+    """The message as base64 chunks for the 255-character text inbox (firmware before 0.2.33)."""
+    encoded = base64.b64encode(encode(message).encode()).decode('ascii')
     token = token or secrets.token_hex(8)
     chunks = [encoded[i:i+200] for i in range(0, len(encoded), 200)]
     return [f'{token}|{i}|{int(i == len(chunks)-1)}|{part}' for i, part in enumerate(chunks)]
 
+def message_action(node):
+    """Home Assistant action that hands a screen one whole message, or None without a node name."""
+    return alert_service(node, MESSAGE_ACTION)
+
 # ----- Alerts (firmware 0.2.31+): the reference the cheatsheet shows. tests/test_alerts_reference.py
 # keeps every value here equal to what the board profiles compile. -----
 ALERT_MIN_FIRMWARE = '0.2.31'
@@ -485,7 +507,12 @@ def alert_reference():
             'suggested_icons': [{'name': name, 'cp': tile_icons.GLYPHS[name]} for name in ALERT_SUGGESTED_ICONS],
             'extra_icons': [{'name': name, 'cp': cp} for name, cp in tile_icons.FIXED]}
 
-def discover(registry, states, devices, areas):
+def screen_items(registry):
+    """The ESPHome entities that describe a screen; the subset `discover_screens` needs."""
+    return [item for item in registry if item.get('platform') == 'esphome' and item.get('original_name') in SCREEN_ENTITY_NAMES]
+
+def discover_screens(registry, states, devices, areas):
+    """Paired screens: every enabled ESPHome inbox with the diagnostics of its device."""
     device_map = {d['id']: d for d in devices}
     area_map = {a['area_id']: a['name'] for a in areas}
     versions = {item.get('device_id'): states.get(item['entity_id'], {}).get('state', 'onbekend') for item in registry
@@ -502,23 +529,37 @@ def discover(registry, states, devices, areas):
         return found
     nodes = diagnostic('Apparaatnaam', r'[a-z0-9][a-z0-9-]{0,30}')
     addresses = diagnostic('IP-adres', r'\d{1,3}(\.\d{1,3}){3}')
-    screens, entities = [], []
+    screens = []
     for item in registry:
         eid = item['entity_id']
+        if not (item.get('platform') == 'esphome' and eid.startswith('text.') and item.get('original_name') == 'Tegelinstellingen' and not item.get('disabled_by')):
+            continue
         device = device_map.get(item.get('device_id'), {})
         state = states.get(eid, {})
         area = area_map.get(item.get('area_id') or device.get('area_id'), '')
-        if item.get('platform') == 'esphome' and eid.startswith('text.') and item.get('original_name') == 'Tegelinstellingen' and not item.get('disabled_by'):
-            screens.append({'id': eid, 'name': device.get('name_by_user') or device.get('name') or eid,
-                            'firmware': versions.get(item.get('device_id'), 'onbekend'),
-                            'board': boards.get(item.get('device_id'), 'unknown'),
-                            'node': nodes.get(item.get('device_id')), 'ip': addresses.get(item.get('device_id')),
-                            'device': device.get('name') or '',
-                            'area': area, 'online': state.get('state') not in (None, 'unknown', 'unavailable'),
-                            'status': state.get('state', 'Niet verbonden')})
+        screens.append({'id': eid, 'name': device.get('name_by_user') or device.get('name') or eid,
+                        'firmware': versions.get(item.get('device_id'), 'onbekend'),
+                        'board': boards.get(item.get('device_id'), 'unknown'),
+                        'node': nodes.get(item.get('device_id')), 'ip': addresses.get(item.get('device_id')),
+                        'device': device.get('name') or '',
+                        'area': area, 'online': state.get('state') not in (None, 'unknown', 'unavailable'),
+                        'status': state.get('state', 'Niet verbonden')})
+    return screens
+
+def discover(registry, states, devices, areas):
+    """(screens, entities): the paired screens and every entity a tile or the top bar can show."""
+    device_map = {d['id']: d for d in devices}
+    area_map = {a['area_id']: a['name'] for a in areas}
+    screens = discover_screens(registry, states, devices, areas)
+    entities = []
+    for item in registry:
+        eid = item['entity_id']
         tile = entity_id(eid)
         if not (tile or header_entity(eid)) or item.get('disabled_by'):
             continue
+        device = device_map.get(item.get('device_id'), {})
+        state = states.get(eid, {})
+        area = area_map.get(item.get('area_id') or device.get('area_id'), '')
         entities.append({'id': eid, 'name': state.get('attributes', {}).get('friendly_name') or item.get('name') or item.get('original_name') or eid,
                          'device': device.get('name_by_user') or device.get('name') or '', 'area': area,
                          'state': state.get('state', 'unavailable'), 'icon': tile_icons.ha_icon(state.get('attributes')),

screen_manager/app/server.py

@@ -1,4 +1,4 @@
-"""HA Ingress app. HA writes: text.set_value on discovered inboxes and one persistent notification when a nightly update stops."""
+"""HA Ingress app. HA writes: text.set_value on discovered inboxes (or the screen_message action on firmware 0.2.33+) and one persistent notification when a nightly update stops."""
 import asyncio
 import contextlib
 import json
@@ -14,17 +14,45 @@ import tile_icons
 from updates import Updater
 
 from aiohttp import ClientSession, ClientTimeout, WSMsgType, web
-from core import BUILTIN, HEADER_MIN_FIRMWARE, TILE_BACKGROUNDS, alert_reference, alert_service, controls_catalogue, discover, extras, header_items, min_firmware, pack_slots, packets, state_message, validate_header, validate_layout, validate_settings
+from core import BUILTIN, HEADER_MIN_FIRMWARE, TILE_BACKGROUNDS, TRANSPORT_MIN_FIRMWARE, alert_reference, alert_service, controls_catalogue, discover, discover_screens, encode, extras, header_items, message_action, min_firmware, pack_slots, packets, revision, screen_items, state_message, validate_header, validate_layout, validate_settings
 import header_bar
 from zoneinfo import ZoneInfo
 
 LOG = logging.getLogger('screen_manager')
 
 REGISTRY_EVENTS = ('entity_registry_updated', 'device_registry_updated', 'area_registry_updated')
-SCREEN_ENTITY_NAMES = {'Tegelinstellingen', 'Schermfirmware', 'Guition schermtype', 'Apparaatnaam', 'IP-adres'}
-# Full repeat of a screen's layout and states. Every layout message declares it, so the
-# firmware sizes its feed watchdog from this one constant (firmware 0.2.22+).
+# Keepalive cadence. Firmware 0.2.33+ gets a small ping carrying the layout revision; every
+# layout message declares this number, so the firmware sizes its feed watchdog from it
+# (firmware 0.2.22+). Firmware before 0.2.33 still gets the whole layout every round.
 KEEPALIVE_SECONDS = 120
+# Safety net for screens that answer every ping: everything again once an hour.
+FULL_REPEAT_SECONDS = 3600
+# Sensor history is refreshed in a background task, never inside the sync loop.
+HISTORY_SECONDS = 300
+# Inbox states after which a screen needs the whole layout again: a restart, a ping that
+# did not match, or a tile state that never arrived. Guarded so a slow batch cannot loop.
+RESEND_STATES = frozenset({'Klaar voor tegelconfiguratie', 'Indeling opnieuw nodig', 'Tegels laden'})
+RESEND_GUARD_SECONDS = 120
+FORECAST_SECONDS = 1800
+
+def samples(events, begin, span):
+    """24 values, one per bucket: the last known value at the end of each bucket (None until the first)."""
+    events.sort(key=lambda pair: pair[0])
+    position, value, out = 0, None, []
+    for bucket in range(24):
+        boundary = begin + (bucket + 1) * span / 24
+        while position < len(events) and events[position][0] <= boundary:
+            value = events[position][1]
+            position += 1
+        out.append(value)
+    return out
+
+def rounded(value):
+    try:
+        value = float(value)
+    except (TypeError, ValueError):
+        return None
+    return round(value, 3) if math.isfinite(value) else None
 
 class HomeAssistant:
     registry_interval = 600
@@ -40,6 +68,8 @@ class HomeAssistant:
         self.changed = asyncio.Event()
         # Entity ids the manager cares about; None wakes it for every state change.
         self.relevant = None
+        # Entity ids whose state changed since the manager last looked; it only rebuilds those tiles.
+        self.dirty = set()
         self.registry_changed = asyncio.Event()
         self.setting_events = []
         self.time_zone = None
@@ -84,6 +114,7 @@ class HomeAssistant:
                     else:
                         self.states.pop(eid, None)
                     if self.relevant is None or eid in self.relevant:
+                        self.dirty.add(eid)
                         self.changed.set()
                 elif event.get('event_type') in REGISTRY_EVENTS:
                     self.registry_changed.set()
@@ -168,7 +199,12 @@ class HomeAssistant:
                         future.set_exception(ConnectionError('Verbinding verbroken.'))
             await asyncio.sleep(5)
 
-    async def send(self, inbox, message):
+    async def send(self, inbox, message, action=None):
+        """One message to a screen: as one ESPHome action call (firmware 0.2.33+), else as text chunks."""
+        if action:
+            domain, service = action.split('.', 1)
+            await self.request('call_service', domain=domain, service=service, service_data={'message': encode(message)})
+            return
         for packet in packets(message):
             await self.request('call_service', domain='text', service='set_value',
                                service_data={'entity_id': inbox, 'value': packet})
@@ -180,6 +216,30 @@ class HomeAssistant:
         forecast = ((result or {}).get('response') or {}).get(entity, {}).get('forecast', [])
         return forecast if isinstance(forecast, list) else []
 
+    async def statistics(self, entities, hours):
+        """24 samples per entity from the recorder's statistics, all entities in one request.
+
+        Hourly means for a day, five-minute means below that; a sum-only sensor (energy) gives its
+        state. Entities without statistics (no state class) are absent and fall back to `history`."""
+        period = 'hour' if hours >= 24 else '5minute'
+        start = datetime.now(timezone.utc) - timedelta(hours=hours)
+        result = await self.request('recorder/statistics_during_period', start_time=start.isoformat(),
+                                    statistic_ids=sorted(entities), period=period, types=['mean', 'state'])
+        begin, span, found = start.timestamp(), hours * 3600, {}
+        for entity, rows in (result or {}).items():
+            if entity not in entities or not isinstance(rows, list):
+                continue
+            events = []
+            for row in rows:
+                moment = row.get('start') if isinstance(row, dict) else None
+                if not isinstance(moment, (int, float)):
+                    continue
+                if moment > 1e11:  # milliseconds since HA 2023.9
+                    moment /= 1000
+                events.append((moment, rounded(row.get('mean') if row.get('mean') is not None else row.get('state'))))
+            found[entity] = samples(events, begin, span)
+        return found
+
     async def history(self, entity, hours):
         start = (datetime.now(timezone.utc)-timedelta(hours=hours)).isoformat()
         async with self.session.get(self.base+'/history/period/'+start,
@@ -191,34 +251,33 @@ class HomeAssistant:
                 raw.extend(chunk)
                 if len(raw)>2*1024*1024: raise ValueError('Geschiedenis te groot.')
             rows=json.loads(raw)
-        samples=[]; events=[]
+        events=[]
         begin=datetime.fromisoformat(start).timestamp(); span=hours*3600
         for row in rows[0] if rows else []:
             try:
                 timestamp=datetime.fromisoformat(row.get('last_changed',row.get('last_updated','')).replace('Z','+00:00')).timestamp()
             except (ValueError,TypeError): continue
-            try:
-                value=float(row['state']); value=round(value,3) if math.isfinite(value) else None
-            except (ValueError,TypeError,KeyError): value=None
-            events.append((timestamp,value))
-        events.sort(key=lambda pair:pair[0]);position=0;value=None
-        for bucket in range(24):
-            boundary=begin+(bucket+1)*span/24
-            while position<len(events) and events[position][0]<=boundary:
-                value=events[position][1];position+=1
-            samples.append(value)
-        return samples
+            events.append((timestamp,rounded(row.get('state'))))
+        return samples(events, begin, span)
 
 class Manager:
     def __init__(self, ha, path):
         self.ha, self.path = ha, Path(path)
-        self.layouts, self.sent, self.status, self.last = {}, {}, {}, {}
+        # sent: per inbox what the screen holds ({'layout', 'header', 'states', 'rev'}); last: the
+        # last full send; pinged: the last keepalive ping.
+        self.layouts, self.sent, self.status, self.last, self.pinged = {}, {}, {}, {}, {}
+        # (entity, hours) -> (monotonic, 24 samples); filled by history_loop, read by sync_one.
         self.histories = {}
+        self.history_wake = asyncio.Event()
         self.forecasts = {}
         self.listeners = set()  # asyncio.Event per open /api/events stream
         self.firmware = Firmware(os.environ.get("ESPHOME_CONFIG", "/homeassistant/esphome"), self.path.parent)
         self.updates = Updater(self, self.path.parent / 'updates.json')
         self._registry_source, self._registry_index = None, {}
+        self._items_source, self._items = None, []
+        self._screens_key, self._screens = None, []
+        self._watched_key, self._watched = None, set()
+        self._seen_registry = None
         if self.path.exists():
             raw = json.loads(self.path.read_text())
             # Versioned persistent data. Never silently overwrite an unknown schema.
@@ -227,11 +286,34 @@ class Manager:
             self.layouts = {key: validate_layout(value) for key, value in raw['screens'].items()}
 
     def inventory(self):
+        """(screens, every tile/top-bar entity): the full walk over the registry, for the editor and saving."""
         return discover(self.ha.registry, self.ha.states, self.ha.devices, self.ha.areas)
 
+    def screen_registry(self):
+        """The few registry entries that describe screens; rebuilt only when HA delivers a new registry."""
+        registry = getattr(self.ha, 'registry', [])
+        if self._items_source is not registry:
+            self._items_source, self._items = registry, screen_items(registry)
+        return self._items
+
+    def screens(self):
+        """The paired screens, cached until the registry or one of the screens' own diagnostics changes.
+
+        Costs a handful of dictionary lookups per screen instead of a walk over every entity in
+        Home Assistant, so the sync loop can run it on every wake."""
+        ha = self.ha
+        items = self.screen_registry()
+        key = (id(ha.registry), id(ha.devices), id(ha.areas), tuple(ha.states.get(item['entity_id'], {}).get('state') for item in items))
+        if key != self._screens_key:
+            self._screens_key, self._screens = key, discover_screens(items, ha.states, ha.devices, ha.areas)
+        return [dict(screen) for screen in self._screens]
+
+    def screen(self, inbox):
+        return next((s for s in self.screens() if s['id'] == inbox), None)
+
     def firmware_version(self, inbox, screen=None):
         if screen is None:
-            screen=next((s for s in self.inventory()[0] if s['id']==inbox), {})
+            screen = self.screen(inbox) or {}
         try:
             version=tuple(int(part) for part in screen.get('firmware','').split('.'))
             return version if len(version)==3 else None
@@ -245,6 +327,16 @@ class Manager:
     def supports_header(self, inbox, screen=None):
         return (self.firmware_version(inbox, screen) or (0, 0, 0)) >= HEADER_MIN_FIRMWARE
 
+    def supports_ping(self, inbox, screen=None):
+        """Firmware 0.2.33+ answers a keepalive ping with its layout revision."""
+        return (self.firmware_version(inbox, screen) or (0, 0, 0)) >= TRANSPORT_MIN_FIRMWARE
+
+    def transport(self, inbox, screen=None):
+        """The ESPHome action that takes a whole message (firmware 0.2.33+), or None for the text inbox."""
+        if screen is None:
+            screen = self.screen(inbox) or {}
+        return message_action(screen.get('node')) if self.supports_ping(inbox, screen) else None
+
     def registry_index(self):
         """Entity registry by id (display precision, entity category); rebuilt only when HA delivers a new registry."""
         registry = getattr(self.ha, 'registry', [])
@@ -316,6 +408,7 @@ class Manager:
         self.layouts = updated
         self.sent.pop(inbox, None)
         self.status[inbox] = 'Opgeslagen; wacht op synchronisatie'
+        self.history_wake.set()
         self.ha.changed.set()
         self.notify()
 
@@ -337,13 +430,16 @@ class Manager:
                 if meta.get('screen') and meta.get('node') not in nodes and (meta.get('friendly') or None) not in devices]
 
     def watched_entities(self):
-        """Entities whose state changes matter: tiles on any layout plus the screens' own diagnostics."""
-        watched = {tile['entity'] for layout in self.layouts.values() for tile in layout['tiles']}
-        watched |= {item['entity'] for layout in self.layouts.values() for item in header_items(layout) if item['type'] == 'entity'}
-        for item in getattr(self.ha, 'registry', []):
-            if item.get('platform') == 'esphome' and item.get('original_name') in SCREEN_ENTITY_NAMES:
-                watched.add(item['entity_id'])
-        return watched
+        """Entities whose state changes matter: tiles on any layout plus the screens' own diagnostics.
+
+        Cached per (registry, layouts): both are replaced as whole objects when they change."""
+        key = (id(getattr(self.ha, 'registry', [])), id(self.layouts))
+        if key != self._watched_key:
+            watched = {tile['entity'] for layout in self.layouts.values() for tile in layout['tiles']}
+            watched |= {item['entity'] for layout in self.layouts.values() for item in header_items(layout) if item['type'] == 'entity'}
+            watched |= {item['entity_id'] for item in self.screen_registry()}
+            self._watched_key, self._watched = key, watched
+        return set(self._watched)
 
     async def cached(self, store, key, ttl, fetch):
         entry=store.get(key)
@@ -353,51 +449,120 @@ class Manager:
             entry=(time.monotonic(),value);store[key]=entry
         return entry[1]
 
-    async def sync_one(self, inbox, layout, force=False, screen=None):
+    def forecast_due(self, entity):
+        entry = self.forecasts.get(entity)
+        hourly = self.forecasts.get((entity, 'hourly'))
+        return not entry or not hourly or time.monotonic() - min(entry[0], hourly[0]) > FORECAST_SECONDS
+
+    async def tile_message(self, index, tile):
+        """The state message of one tile: state, options, extras, and the history the background task holds."""
+        forecast=hourly=None
+        if tile['entity'].startswith('weather.') and hasattr(self.ha,'forecast'):
+            forecast=await self.cached(self.forecasts, tile['entity'], FORECAST_SECONDS, lambda: self.ha.forecast(tile['entity']))
+            # Hourly forecasts feed the weather card's next-hours strip (0.2.23+); refreshed every half hour.
+            hourly=await self.cached(self.forecasts, (tile['entity'],'hourly'), FORECAST_SECONDS, lambda: self.ha.forecast(tile['entity'],'hourly'))
+        message=state_message(index,tile,self.ha.states,extras(tile,self.ha.states,forecast,getattr(self.ha,'time_zone',None),hourly))
+        if tile['entity'].startswith('sensor.') and hasattr(self.ha,'history'):
+            hours=tile.get('options',{}).get('history_hours',24)
+            entry=self.histories.get((tile['entity'],hours))
+            if entry is not None:
+                message['history']={'hours':hours,'values':entry[1]}
+            else:
+                self.history_wake.set()
+        return message
+
+    def layout_message(self, inbox, layout, screen):
+        tiles = layout['tiles']
+        # Grid positions (firmware 0.2.26+; older firmware ignores them and packs the entities in order).
+        message = {'v': 1, 'op': 'layout', 'inbox': inbox, 'title': layout['title'], 'entities': [t['entity'] for t in tiles],
+                   'slots': [t['slot'] for t in tiles], 'keepalive': KEEPALIVE_SECONDS}
+        if 'pages' in layout:
+            message['pages'] = layout['pages']
+        if 'settings' in layout:
+            message['settings'] = {k:v for k,v in layout['settings'].items() if k not in ('swipe_pages','rotation')}
+            message['swipe_pages'] = layout['settings'].get('swipe_pages',False)
+            if screen.get('board')=='guition':
+                message['rotation'] = layout['settings'].get('rotation',0)
+        # The revision the screen echoes on every ping (firmware 0.2.33+; older firmware ignores it).
+        message['rev'] = revision(message)
+        return message
+
+    async def sync_one(self, inbox, layout, force=False, screen=None, dirty=None):
+        """Send the screen what it lacks; True when something went out.
+
+        `dirty` is the set of entity ids whose state changed since the last pass: only their tiles
+        (and the top bar, when it shows one of them) are rebuilt, the other messages are reused as
+        sent. None rebuilds every message and sends the differences; `force` sends everything."""
         if screen is None:
-            screen=next((s for s in self.inventory()[0] if s['id']==inbox), {})
+            screen = self.screen(inbox) or {}
         needed = self.needs_firmware(inbox, layout, screen)
         if needed:
             self.status[inbox]=f"Indeling bewaard; firmware {needed}+ nodig voor deze tegels"
-            return
+            return False
         tiles = layout['tiles']
-        # Grid positions (firmware 0.2.26+; older firmware ignores them and packs the entities in order).
-        messages = [{'v': 1, 'op': 'layout', 'inbox': inbox, 'title': layout['title'], 'entities': [t['entity'] for t in tiles],
-                     'slots': [t['slot'] for t in tiles], 'keepalive': KEEPALIVE_SECONDS}]
-        if 'pages' in layout:
-            messages[0]['pages'] = layout['pages']
-        if 'settings' in layout:
-            messages[0]['settings'] = {k:v for k,v in layout['settings'].items() if k not in ('swipe_pages','rotation')}
-            messages[0]['swipe_pages'] = layout['settings'].get('swipe_pages',False)
-            if screen.get('board')=='guition':
-                messages[0]['rotation'] = layout['settings'].get('rotation',0)
+        layout_msg = self.layout_message(inbox, layout, screen)
+        previous = self.sent.get(inbox)
+        full = force or not previous or previous['layout'] != layout_msg or dirty is None
         # The top bar right after the layout (firmware 0.2.32+; older firmware draws the clock of show_clock).
+        header_msg = None
         if self.supports_header(inbox, screen):
-            messages.append(self.header_message(layout))
-        for i,tile in enumerate(layout['tiles']):
-            forecast=hourly=None
-            if tile['entity'].startswith('weather.') and hasattr(self.ha,'forecast'):
-                forecast=await self.cached(self.forecasts, tile['entity'], 1800, lambda: self.ha.forecast(tile['entity']))
-                # Hourly forecasts feed the weather card's next-hours strip (0.2.23+); refreshed every half hour.
-                hourly=await self.cached(self.forecasts, (tile['entity'],'hourly'), 1800, lambda: self.ha.forecast(tile['entity'],'hourly'))
-            message=state_message(i,tile,self.ha.states,extras(tile,self.ha.states,forecast,getattr(self.ha,'time_zone',None),hourly))
-            if tile['entity'].startswith('sensor.') and hasattr(self.ha,'history'):
-                hours=tile.get('options',{}).get('history_hours',24)
-                key=(tile['entity'],hours)
-                message['history']={'hours':hours,'values':await self.cached(self.histories, key, 300, lambda: self.ha.history(*key))}
-            messages.append(message)
-        previous = self.sent.get(inbox, [])
-        force = force or not previous or messages[0] != previous[0]
-        for i, message in enumerate(messages):
+            bar = {item['entity'] for item in header_items(layout) if item['type'] == 'entity'}
+            if full or previous['header'] is None or bar & dirty:
+                header_msg = self.header_message(layout)
+            else:
+                header_msg = previous['header']
+        states = []
+        for i, tile in enumerate(tiles):
+            reuse = not full and i < len(previous['states']) and tile['entity'] not in dirty
+            if reuse and tile['entity'].startswith('weather.') and self.forecast_due(tile['entity']):
+                reuse = False
+            states.append(previous['states'][i] if reuse else await self.tile_message(i, tile))
+        outgoing = []
+        if force or not previous or layout_msg != previous['layout']:
+            outgoing.append(layout_msg)
+        if header_msg is not None and (force or not previous or header_msg != previous['header']):
+            outgoing.append(header_msg)
+        for i, message in enumerate(states):
+            if force or not previous or i >= len(previous['states']) or message != previous['states'][i]:
+                outgoing.append(message)
+        action = self.transport(inbox, screen)
+        for message in outgoing:
             # Save during transmission aborts the old batch, then starts a full new layout.
             if self.layouts.get(inbox) != layout:
                 self.sent.pop(inbox, None)
-                return
-            if force or i >= len(previous) or message != previous[i]:
-                await self.ha.send(inbox, message)
-        self.sent[inbox] = messages
-        self.last[inbox] = time.monotonic()
+                return True
+            await self.ha.send(inbox, message, action)
+        self.sent[inbox] = {'layout': layout_msg, 'header': header_msg, 'states': states, 'rev': layout_msg['rev']}
+        # The hourly timer follows a real full transmission; any message refreshes the screen's
+        # feed window, so the ping timer follows whatever went out (a rebuild without differences,
+        # after a registry refresh, must not postpone the ping).
+        if force or not previous:
+            self.last[inbox] = time.monotonic()
+        if outgoing:
+            self.pinged[inbox] = time.monotonic()
         self.status[inbox] = 'Verzonden naar Home Assistant'
+        return bool(outgoing)
+
+    async def ping(self, inbox, screen):
+        """Keepalive for firmware 0.2.33+: the layout revision only; the screen asks for the rest itself."""
+        sent = self.sent.get(inbox)
+        if not sent:
+            return
+        await self.ha.send(inbox, {'v': 1, 'op': 'ping', 'rev': sent['rev'], 'keepalive': KEEPALIVE_SECONDS}, self.transport(inbox, screen))
+        self.pinged[inbox] = time.monotonic()
+
+    def take_dirty(self):
+        """Entity ids that changed since the last pass, or None when everything must be rebuilt."""
+        dirty = getattr(self.ha, 'dirty', None)
+        if dirty is None:
+            return None
+        self.ha.dirty = set()
+        registry = getattr(self.ha, 'registry', None)
+        if registry is not self._seen_registry:
+            # A new registry can change names and display precision: rebuild everything once.
+            self._seen_registry = registry
+            return None
+        return dirty
 
     async def run(self):
         while True:
@@ -424,29 +589,95 @@ class Manager:
                         layout['settings']=validate_settings(settings);self.save(inbox,layout)
                 except (ValueError,TypeError): pass
             if hasattr(self.ha,'setting_events'): self.ha.setting_events.clear()
-            screens = self.inventory()[0]
+            dirty = self.take_dirty()
+            screens_key = self._screens_key
+            screens = self.screens()
+            changed = self._screens_key != screens_key
             self.ha.relevant = self.watched_entities()
             for screen in screens:
                 inbox = screen['id']
+                before = self.status.get(inbox)
                 if not screen['online']:
                     self.sent.pop(inbox, None)
                     self.status[inbox] = 'Scherm offline; wijzigingen bewaard'
+                    changed = changed or self.status[inbox] != before
                     continue
                 if inbox not in self.layouts:
                     continue
                 try:
-                    force = time.monotonic() - self.last.get(inbox, 0) >= KEEPALIVE_SECONDS
-                    # last records full keepalive, not unrelated HA state events.
-                    before = self.last.get(inbox, 0)
-                    await self.sync_one(inbox, self.layouts[inbox], force, screen)
-                    if not force:
-                        self.last[inbox] = before
+                    now = time.monotonic()
+                    pings = self.supports_ping(inbox, screen)
+                    # The screen says it lost the layout (restart, mismatched ping, a state that never came).
+                    if inbox in self.sent and screen['status'] in RESEND_STATES and now - self.last.get(inbox, 0) >= RESEND_GUARD_SECONDS:
+                        self.sent.pop(inbox)
+                    force = now - self.last.get(inbox, 0) >= (FULL_REPEAT_SECONDS if pings else KEEPALIVE_SECONDS)
+                    if await self.sync_one(inbox, self.layouts[inbox], force, screen, dirty):
+                        changed = True
+                    if pings and inbox in self.sent and now - self.pinged.get(inbox, 0) >= KEEPALIVE_SECONDS:
+                        await self.ping(inbox, screen)
                 except Exception as error:
                     self.sent.pop(inbox, None)
                     self.status[inbox] = 'Verzenden mislukt; automatisch opnieuw proberen'
                     LOG.warning('Schermsynchronisatie opnieuw proberen (%s)', type(error).__name__)
                     await asyncio.sleep(1)
-            self.notify()
+                changed = changed or self.status.get(inbox) != before
+            if changed:
+                self.notify()
+
+    def history_keys(self):
+        """(entity, hours) of every sensor tile on any screen."""
+        return {(tile['entity'], tile.get('options', {}).get('history_hours', 24))
+                for layout in self.layouts.values() for tile in layout['tiles'] if tile['entity'].startswith('sensor.')}
+
+    async def refresh_histories(self):
+        """Fetch the histories that are missing or older than HISTORY_SECONDS: one statistics request per
+        window for all sensors at once, the REST history only for sensors without statistics. Changed
+        values mark their entity dirty so the sync loop resends just those tiles."""
+        wanted = self.history_keys()
+        for key in [key for key in self.histories if key not in wanted]:
+            del self.histories[key]
+        now = time.monotonic()
+        due = [key for key in wanted if key not in self.histories or now - self.histories[key][0] >= HISTORY_SECONDS]
+        if not due:
+            return
+        found = {}
+        if hasattr(self.ha, 'statistics'):
+            for hours in sorted({hours for _, hours in due}):
+                entities = {entity for entity, h in due if h == hours}
+                try:
+                    found.update({(entity, hours): values for entity, values in (await self.ha.statistics(entities, hours)).items()})
+                except Exception as error:
+                    LOG.warning('Statistieken ophalen mislukt (%s); geschiedenis per sensor', type(error).__name__)
+        changed = set()
+        for key in due:
+            values = found.get(key)
+            if values is None:
+                try:
+                    values = await self.ha.history(*key)
+                except Exception:
+                    values = []
+            old = self.histories.get(key)
+            self.histories[key] = (time.monotonic(), values)
+            if old is None or old[1] != values:
+                changed.add(key[0])
+        if changed:
+            dirty = getattr(self.ha, 'dirty', None)
+            if dirty is not None:
+                dirty.update(changed)
+            self.ha.changed.set()
+
+    async def history_loop(self):
+        """Background refresh of sensor histories, so the sync loop never waits for Home Assistant's database."""
+        while True:
+            with contextlib.suppress(TimeoutError):
+                await asyncio.wait_for(self.history_wake.wait(), 30)
+            self.history_wake.clear()
+            if not self.ha.online or not hasattr(self.ha, 'history'):
+                continue
+            try:
+                await self.refresh_histories()
+            except Exception as error:
+                LOG.warning('Geschiedenis vernieuwen mislukt (%s)', type(error).__name__)
 
 def create_app(manager, development=False):
     csrf = secrets.token_urlsafe(32)
@@ -475,9 +706,10 @@ def create_app(manager, development=False):
 
     async def index(request):
         return web.FileResponse(static / 'index.html')
-    def light_payload():
+    def light_payload(screens=None):
         """Screens and update status: everything that changes while the page is open."""
-        screens, entities = manager.inventory()
+        if screens is None:
+            screens = manager.screens()
         profiles = manager.firmware.profile_names()
         for screen in screens:
             screen['layout'] = manager.layouts.get(screen['id'], {'title': 'Thuis', 'tiles': []})
@@ -487,19 +719,21 @@ def create_app(manager, development=False):
             screen['dismiss_action'] = alert_service(screen.get('node'), 'dismiss_alert')
         return {'csrf': csrf, 'connected': manager.ha.online, 'screens': screens,
                 'pending': manager.pending_profiles(screens, profiles),
-                'updates': manager.updates.summary(screens, profiles)}, entities
+                'updates': manager.updates.summary(screens, profiles)}
     async def inventory(request):
-        payload, entities = light_payload()
-        if request.query.get('light') != '1':
+        if request.query.get('light') == '1':
             # The page polls the light form; entities, backgrounds and icons (~100 KB) only on demand.
-            payload['entities'] = entities
-            payload['backgrounds'] = TILE_BACKGROUNDS
-            payload['controls'] = controls_catalogue()
-            payload['icons'] = tile_icons.editor()
-            payload['alerts'] = alert_reference()
-            payload['builtin'] = [{'id': key, 'name': name, 'device': 'Ingebouwd op het scherm', 'area': '', 'state': 'ok'} for key, name in BUILTIN.items()]
-            payload['header'] = {**header_bar.catalogue(), 'suggestions': {
-                screen['id']: header_bar.suggestions(screen, entities, manager.ha.states, manager.registry_index()) for screen in payload['screens']}}
+            return web.json_response(light_payload())
+        screens, entities = manager.inventory()
+        payload = light_payload(screens)
+        payload['entities'] = entities
+        payload['backgrounds'] = TILE_BACKGROUNDS
+        payload['controls'] = controls_catalogue()
+        payload['icons'] = tile_icons.editor()
+        payload['alerts'] = alert_reference()
+        payload['builtin'] = [{'id': key, 'name': name, 'device': 'Ingebouwd op het scherm', 'area': '', 'state': 'ok'} for key, name in BUILTIN.items()]
+        payload['header'] = {**header_bar.catalogue(), 'suggestions': {
+            screen['id']: header_bar.suggestions(screen, entities, manager.ha.states, manager.registry_index()) for screen in payload['screens']}}
         return web.json_response(payload)
     async def header_preview(request):
         """The top bar as a screen would draw it right now, so the editor shows unsaved changes live."""
@@ -521,7 +755,7 @@ def create_app(manager, development=False):
                 with contextlib.suppress(TimeoutError):
                     await asyncio.wait_for(wake.wait(), 3)
                 wake.clear()
-                body = json.dumps(light_payload()[0], ensure_ascii=False)
+                body = json.dumps(light_payload(), ensure_ascii=False)
                 if body != sent:
                     await response.write(f'data: {body}\n\n'.encode())
                     sent = body
@@ -545,9 +779,10 @@ def create_app(manager, development=False):
         return web.json_response(manager.updates.summary())
     async def inspector(request):
         inbox=request.match_info['inbox']
-        if inbox not in {s['id'] for s in manager.inventory()[0]}: raise ValueError('Onbekend scherm.')
+        screen=manager.screen(inbox)
+        if screen is None: raise ValueError('Onbekend scherm.')
         layout=manager.layouts.get(inbox,{'tiles':[]})
-        return web.json_response({'screen':next(s for s in manager.inventory()[0] if s['id']==inbox),
+        return web.json_response({'screen':screen,
             'delivery':manager.status.get(inbox), 'layout':layout,
             'tiles':[{'entity':t['entity'],'state':manager.ha.states.get(t['entity'],{}).get('state'),
                       'attributes':state_message(i,t,manager.ha.states)['a'],
@@ -592,7 +827,7 @@ async def main():
         await runner.setup()
         await web.TCPSite(runner, '127.0.0.1' if development else '0.0.0.0', 8099).start()
         try:
-            await asyncio.gather(ha.run(), manager.run(), manager.updates.run())
+            await asyncio.gather(ha.run(), manager.run(), manager.history_loop(), manager.updates.run())
         finally:
             await runner.cleanup()
 

screen_manager/app/updates.py

@@ -55,7 +55,7 @@ class Updater:
         return bool(self.task and not self.task.done())
 
     def screen(self, inbox):
-        return next((s for s in self.manager.inventory()[0] if s['id'] == inbox), None)
+        return self.manager.screen(inbox)
 
     def resolve(self, screen, profiles=None):
         """Profile file and OTA address for a screen; None when the add-on cannot tell."""
@@ -89,7 +89,7 @@ class Updater:
         # Callers that already hold the inventory and profile list pass them in; the inventory
         # handler otherwise re-reads every ESPHome profile several times per request.
         if screens is None:
-            screens = self.manager.inventory()[0]
+            screens = self.manager.screens()
         if profiles is None:
             profiles = self.manager.firmware.profile_names()
         return [s['id'] for s in screens

screen_manager/config.yaml

@@ -1,5 +1,5 @@
 name: ESP Screen Manager
-version: "0.2.38"
+version: "0.2.39"
 slug: esp_screen_manager
 description: Kies tegels voor je CYD en Guition touchscreen, zonder opnieuw flashen.
 url: https://github.com/MaxGramser/homeassistant_espscreen