Concept + definition: Live Interactivity (Lumens) — Tier-1-fast, agent-generated, safe#31
Merged
Merged
Conversation
…neration; add kiosk Lumen mockup
…isual-spec, implementation-plan, README
10 tasks
iret77
changed the title
Local Codex review (adversarial, vs origin/main) flagged six issues across the Live-Interactivity additions; tractable points fixed, two architectural points resolved with the maintainer: - Additivity made fail-closed: 1.1 content is negotiation-gated (a 1.0 client is never sent it; unknown types are hard-rejected by the whitelist, not silently "ignored"). Aligns lumens-spec §0/§12 with protocol/1.0 §2. - Capability authority: the agent owns capability *requests*, never *grants* (a grant is Tier-2 policy + user consent; no self-grant via patch). §0.5. - Bounded wakeups: `timer` is capped like `tick` via a combined per-Lumen wakeup budget, rejected at validation when exceeded. §0.2/§4/§11. - Capability-broker egress bounds (rate/quota/max-in-flight/idempotency/ backpressure) + state/DataRef-derived fetch/writeData classified external-effect unless pre-approved. §6/§11 + open-question §13.8. - Shared/preset assets travel by content id; Tier 2 re-mints DataRef tokens scoped to the recipient, inaccessible assets render inert. §9 / concept §7. - Cross-element reads: ambient-by-default selection replaced by a declared `expose` read-only interface (ambient-by-declaration) — un-exposed state stays private; imported Lumens see no ambient neighbour state. §7/§11/§9.1. Plus: LX versioned with the protocol (LX/1.1, handshake lxVersion "1.1"); mockup references visual-spec v0.5 §4.13. Version headers bumped (interactivity-concept 0.6; lumens-spec draft 1.1 rev 2). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jun 16, 2026
Merged
Weegy
added a commit
to byte5ai/omadia
that referenced
this pull request
Jun 16, 2026
… (server) (#315) * feat(canvas-core): Lumens (Live Interactivity) protocol 1.1 — schemas, LX interpreter, capability policy omadia-canvas-protocol/1.1, additive over 1.0 (PR byte5ai/omadia-ui#31, issue #34). - L0: lx-ast/scene/ports-wires/capability-manifest/lumen JSON schemas (+ var read node, get projection node) with accept/reject conformance fixtures. - L1: deterministic, gas+depth+value-size-bounded, no-eval LX interpreter with seeded random/now and a static semantic validator. Prototype-pollution hardened; all 10 findings from an adversarial (Forge/GPT-5.4) review fixed. - L5/L7/L8 Tier-2 policy (pure, consumed by the orchestrator when built): effect classification, broker egress bounds (rate/quota/in-flight/idempotency/ backpressure), content-addressed never-stale asset cache, import consent, content-addressed presets + resolve-then-generate + fork lineage, share token re-mint (assets travel by id, never the author token). - validateLumenFull combined gate. 172 tests, tsc clean. Land this before the omadia-ui UI PR (UI syncs these schemas). * feat(ui-orchestrator): Lumen producer tool + 1.1 tree validation (Tier-2) Makes Lumens reachable end-to-end over a live server (the producer half of #34): - canvas_publish_lumen — a canvas-output producer that instantiates a vetted reference Lumen (variant: arcade game · interactive map · defrag animation) as a surface_snapshot. Authorised in the canvas-output allow-set AND the deterministic-action allow-set, so a canvas action of that type dispatches LLM-free (a click renders the Lumen with no model turn). - referenceLumens.ts — the three vetted reference Lumens (declarative LX data), each exercising a different slice: tick simulation + events, interactive selection/zoom via get/if, and a map(range) tick animation. - treeValidator: extended additively to omadia-canvas-protocol/1.1 — the 1.0 canvas-tree with scene + lumen added to the primitive oneOf, so a snapshot carrying a Lumen validates server-side. Depends on the canvas-core 1.1 schemas in this PR. * feat(ui-orchestrator): agent-authored Lumens (dynamic LLM generation) canvas_publish_lumen now PREFERS an agent-authored `lumen` (the real thesis: the LLM generates declarative LX data; the host validates it). The tool validates the authored Lumen structurally (validateLumenNode against the 1.1 lumen schema) and, on failure, returns a path-pointed error so the agent self-corrects — this is what makes LLM-generated interactivity safe (the model proposes data, the host proves it bounded/total/deterministic before it runs). The tool description carries the compact LX grammar + a worked example. The `variant` reference presets remain as a canned-demo fallback (resolve-then- generate). Semantic bounds beyond the schema are enforced Tier-1 by the interpreter (halts a bad Lumen with surface_error, never the canvas). * feat(ui-orchestrator): data-bound Lumens from real privacy-shielded datasets (L5 loadData) canvas_publish_lumen now also takes a privacy-shield `datasetId` (+ optional labelField/valueField). The real rows are resolved SERVER-SIDE via the same privacy provider canvas_publish_rows uses — the unmasked data never reaches the LLM — and an interactive, tappable data Lumen is built deterministically in datasetLumen.ts (one selectable row per record; tap highlights via a state-driven conditional). This is the privacy-safe answer to 'visualise the user's real data as a live artifact': the model only passes the dataset handle, the host resolves + constructs. Locally verified (validateLumen ok; view + tap transition evaluate correctly). * feat(ui-orchestrator): accept inline data rows for data-bound Lumens Adds a `data` param to canvas_publish_lumen: when the agent already HOLDS the rows (an unshielded fetch — e.g. dynamics tools in bypass mode emit real rows directly, no datasetId interned), it passes them as `data` and the server builds the interactive data Lumen via buildDatasetLumen — no LX authoring needed. Complements `datasetId` (masked/shielded path, resolved server-side). Tool description now steers: visible rows → `data`, masked → `datasetId`. * test(ui-orchestrator): include canvas_publish_lumen in producer-tool registration assertions The producer now registers three canvas-output tools (rows + choice + lumen); update the registration/dispose assertions accordingly (3 registered, 3 disposed).
…ls, transaction patterns, reliability net
Stress-tested by hand-tracing board-game-class Lumens (Tetris/Pacman) in actual
LX-AST. Surfaced three normative gaps the prose hid and two practice-fit gaps:
- §2.2: map/filter/fold binder node forms + computed-index at/setAt — without
them no iteration or board/cell mutation is expressible (the load-bearing
hole; pure LX could name map but not express the per-element body).
- §2.6: native kernels — bounded, host-owned algorithms (sortBy, groupBy,
aggregate, scale, timeBucket, layoutGraph, treemap, geo, floodFill, pathfind)
exposed as pure {kernel:…} calls. The capability pattern applied to compute:
the agent calls but never writes one, so no-arbitrary-code / cannot-hang hold.
- §6.3: high-frequency/transactional patterns (local-first commit-once,
session-scoped consent, optimistic+reconcile) so kiosk/ordering flows fit
without a confirmation modal per tap; server stays authoritative.
- §2.7: declared invariants (silent-wrong → loud-error, post-transition
rollback) + §14 golden-trace authoring gate; preset/idiom assembly normative
as the primary path.
- §14: fifth (transactional) reference Lumen; hand-authoring the reference set
in real LX-AST is the acceptance gate before implementation budget.
Consistency: §11 security summary + §12 SDK deltas + §13 handshake (kernels,
kernelGasLimit) updated. Rationale synced in interactivity-concept.md (v0.7,
§13 items 9-10).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hand-writing the arcade tick in real LX-AST surfaced a blocker the rev-3 binder
addition inherited and did not close: the catalog defined `let` and the
map/filter/fold binders (`as`/`acc`) but never a node to *read* a bound local.
Without it no `let`/iteration body can reference what it binds — i.e. iteration
was still not actually expressible.
Add `{var:name}` / `{var:name, path:"f.g"}` (§2.2): read a let local or a
map/filter/fold binder, with optional dotted sub-path into a record/list. Split
the old combined "var / let" row which only ever gave the `let` form.
Found by the §14 acceptance test (hand-author the reference set before
implementation budget) doing exactly its job.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… binder, flatten
Resolves the B/C/D findings from the hand-author test:
- B (table explosion): add an immutable `const` section (§1.2) + `{const}` read
node (§2.2). Typed/bounded like state but agent-owned, read-only, NOT
serialised (no state-cap cost, no persist, no undo/replay). Static tables
(tetromino shapes, mazes, colour maps) are declared once and referenced, not
re-inlined into every transition; it is the unit the idiom/preset library
ships. Large blobs still go via DataRef.
- C (grid vs list<list>): `at`/`setAt` 2-D `[x,y]` now indexes a `list<list>`
too (as coll[y][x]), with a spatial convention (x horizontal, y vertical) for
both backings. grid for fixed dims, list<list> for add/remove rows — both
index identically.
- D (render ergonomics): optional `idx` binder on map/filter/fold (position-
dependent iteration without map(range)+at), and `flatten` in the std-lib
(nested per-row node lists → one scene draw-list).
Consistency: validity rule, §2.2 footer convention, §2.3, §12 SDK deltas, and a
rev-3.1 changelog note updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…user content) A user-authored kiosk / branded-ordering / product Lumen must be able to use any colour, not only the Lume palette. The spec previously hard-clipped scene colours to theme tokens (§3) and mandated Lume-only (§10), which forbade exactly that. Add a declared, scoped colour-authority model (§3.1): - colorMode: 'theme' (default, Lume tokens, re-tintable) | 'brand' (declared bounded palette) | 'free' (arbitrary sRGB/hex), + a `palette` field on the Lumen. - Scope = the Lumen's OWN subtree (its scene draw-list + emitted primitives). Omadia chrome (header, action panel, Beam, canvas frame) and sibling elements always stay Lume — no host white-label in v1 (deliberate identity boundary). - Brand colour may ride the Lume material (glow/luminosity) or render flat. - In brand/free the normaliser no longer clips and enforces NO contrast floor — accessibility of free-colour content is the author's responsibility. 44 pt hit-targets (§4) + reduced-motion (§5) still apply (interaction-safety, not colour). Product decisions taken with the user: content-only scope (no host white-label), no contrast enforcement. Touches: §1 type (colorMode/palette), §3 colour bullet, new §3.1, §10 visual treatment, §12 SDK deltas, rev-3.2 note; visual-spec §4.13; interactivity-concept §4 + v0.8 changelog. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…, not a preference Per user: theme is the default ONLY because the agent assumes (absent any colour direction) that the Lumen embeds in an existing Lume UI. That assumption is not universal — a standalone kiosk, branded-ordering or product surface is a first-class case where the agent reads the intent and picks brand/free DIRECTLY. The user must never have to "fight an opt-out", and theme is not "safer" or "more correct" than brand/free. Removes the value-laden "safe default / opt-out-explicit" framing; states the colorMode is derived from request + embedding context (UI Skill carries the heuristic). Mechanism unchanged (§3.1 theme/brand/free, chrome stays Lume). Touches: lumens-spec §3.1 + §10 + rev-3.2 note; visual-spec §4.13; interactivity-concept §4 + v0.8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the Live Interactivity (Lumens) extension — the Omadia, on-canvas answer to sandbox-style "live artifacts", deliberately re-aimed to be more capable and structurally safer. Docs only (concept + normative definition + integrations + reference mockup). Implementation is tracked in #34.
Thesis: most of what arbitrary-code sandboxes block is blocked by missing capabilities, not missing compute. So Lumens constrain computation (a declarative, bounded, deterministic, interpreted behaviour model — no arbitrary code; the whitelist parser / CSP
default-src 'self'model extends to it) and open capabilities (real data, write-back, allowlisted network, generated assets) mediated through the existing Tier-2/3 orchestration and effect classification.Unblocks: small games, interactive data workflows, unusual visualisations (defrag-style), live maps, kiosk/iPad surfaces — agent-generated, run Tier-1-fast (60 fps, no per-frame server round-trip), shareable + presettable.
A Lumen
Declarative data, not code:
state + transitions + view + events + capabilities, plus a newsceneprimitive, declarativeports/wires, per-region render cadence, and a preset library (author once, reuse constantly).Guarantees: no arbitrary code (LX = validated JSON-AST walked by a shipped interpreter, no
eval) · bounded & total (gas + frame ceiling → can't hang the host) · deterministic (replay/undo/share/multiplayer) · default-deny brokered capabilities · content-addressed never-stale assets (generation lives in omadia-core connectors) · fully additive overomadia-canvas-protocol/1.0.Files
docs/interactivity-concept.md— concept / rationale (v0.5)docs/lumens-spec.md— normative definition (omadia-canvas-protocol/1.1draft): LX,scene, events/touch, cadence, capabilities, ports/wires, lifecycle/presets, sharing, securitydocs/visual-spec.md§4.13 — Lume rendering of Lumens/scene(light-as-material, not glass)docs/implementation-plan.md§10 — phased implementation outline (L0–L9)docs/mockups/kiosk-lumen-aura.html— reference kiosk Lumen mockup (Lume / Lagoon)CONCEPT.md— extension pointer +Lumenlight-vocabulary entry;README.md— docs index + feature bulletHighlights addressed in review iterations
static/reactive/{tick}, reactive-by-default (~0 % CPU at rest), declarative GPU animation distinct from LX tickstap/longPress/drag/pinch/swipe, 44 pt hit-targets, input-modality handshakevisual-spec.md§1.3)id = kind-sha256) ⇒ never-stale cache; generation = omadia-core LLM connectorsTracking
Implementation requested from maintainers in #34. Codex review requested in-thread.
https://claude.ai/code/session_01AUuCvw7SQkFSk2pGpgMwBD