eclipse-iofog · emirhandurmus · Jul 6, 2026 · Jul 3, 2026 · Jul 6, 2026
diff --git a/Dockerfile b/Dockerfile
@@ -5,7 +5,7 @@
 #     --build-arg VERSION=v0.0.0-test \
 #     -t ghcr.io/eclipse-iofog/edgelet:test .
 
-FROM golang:1.26.4-trixie@sha256:0dcba0d95dbfb072e9917a106b9e07d7cc298097dc83e9307056ef1889de654d AS builder
+FROM golang:1.26.4-trixie@sha256:68b7145ec43d1820b9a56704554b53d1520aa2a15cb5233e374188a31b2a1bce AS builder
 
 ARG BUILDARCH
 ARG TARGETARCH
@@ -42,7 +42,7 @@ RUN set -eux; \
     test -f "build/out/data-linux-${ARCH}.tar.zst"
 
 # Unpack embed bundle + minimal rootfs.
-FROM alpine:3.22@sha256:310c62b5e7ca5b08167e4384c68db0fd2905dd9c7493756d356e893909057601 AS base
+FROM alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce AS base
 RUN apk add --no-cache ca-certificates zstd tzdata
 ARG TARGETARCH
 ARG VERSION=dev

diff --git a/build/Dockerfile.embedded b/build/Dockerfile.embedded
@@ -2,7 +2,7 @@
 # macOS dev:
 #   docker build -f build/Dockerfile.embedded -t edgelet-embed-ci .
 #   docker run --rm -v "$(pwd)":/src -w /src edgelet-embed-ci ./scripts/ci
-FROM golang:1.26.4-trixie@sha256:0dcba0d95dbfb072e9917a106b9e07d7cc298097dc83e9307056ef1889de654d
+FROM golang:1.26.4-trixie@sha256:68b7145ec43d1820b9a56704554b53d1520aa2a15cb5233e374188a31b2a1bce
 
 COPY scripts/install-embed-build-deps /usr/local/bin/install-embed-build-deps
 RUN chmod +x /usr/local/bin/install-embed-build-deps \

diff --git a/docs/edgelet/README.md b/docs/edgelet/README.md
@@ -22,6 +22,7 @@ Operator and developer documentation for the Edgelet edge agent.
 | [dns.md](dns.md) | Bridge DNS, embedded resolver, docker/podman aliases and ExtraHosts |
 | [workload-metadata.md](workload-metadata.md) | Container labels and `EDGELET_*` env contract |
 | [workload-continuity.md](workload-continuity.md) | Reconcile behavior across restarts and engine changes |
+| [volumes.md](volumes.md) | Volume types, delete vs prune behavior, disk layout |
 | [edgeguard.md](edgeguard.md) | Hardware attestation (`edgeGuardFrequency`) |
 | [control-plane.md](control-plane.md) | Local Datasance Controller deployment |
 | [exec-sessions.md](exec-sessions.md) | Multi-session exec (local CLI and controller-initiated) |

diff --git a/docs/edgelet/container-engine.md b/docs/edgelet/container-engine.md
@@ -47,7 +47,7 @@ Isolated from host Docker/Podman installations:
 | `/var/lib/edgelet/data/current/bin/edgelet` | **Fat** runtime ELF (linux): daemon + in-process containerd |
 | `/var/lib/edgelet/data/current/bin/` | Shim (`containerd-shim-runc-v2`), `crun`, CNI multicall + symlinks |
 | `/var/lib/edgelet/data/current` / `previous` | Symlinks to active / prior extracted bundle directories |
-| `/var/lib/edgelet/` | User data (`diskDirectory`) |
+| `/var/lib/edgelet/` | User data (`diskDirectory`); workload volume data under `volumes/data/` — see [volumes.md](volumes.md) |
 | `/var/lib/edgelet-containerd/` | Containerd images, snapshots, CNI state |
 | `/run/edgelet/containerd.sock` | Containerd API socket |
 | `/run/edgelet/edgelet.sock` | EdgeletAPI Unix socket |

diff --git a/docs/edgelet/manifest-reference.md b/docs/edgelet/manifest-reference.md
@@ -78,7 +78,7 @@ spec:
 | `env` | `{key,value}[]` | User env (`EDGELET_*` reserved) |
 | `extraHosts` | `{name,address}[]` or legacy strings | `/etc/hosts` + docker ExtraHosts |
 | `ports` | `{internal,external,protocol}[]` | Port mappings |
-| `volumes` | `{hostDestination,containerDestination,accessMode,type}[]` | Bind mounts / volumes |
+| `volumes` | `{hostDestination,containerDestination,accessMode,type}[]` | `BIND`, `VOLUME`, or controller `VOLUME_MOUNT`. **Delete does not remove `VOLUME` data** on the embedded engine — see [volumes.md](volumes.md). |
 | `commands` | []string | Container command override |
 | `cpuSetCpus` | string | cpuset |
 | `memoryLimit` | int64 | Memory limit bytes |

diff --git a/docs/edgelet/modules/README.md b/docs/edgelet/modules/README.md
@@ -123,6 +123,7 @@ Edge Guard, Pruning, Volume Mount, and SSH Proxy are not in this fixed array; th
 | EdgeletAPI usage | [../edgelet-api-v1.md](../edgelet-api-v1.md) |
 | Controller sync / CP deploy | [../control-plane.md](../control-plane.md) |
 | SQLite backup / wipe | [../persistence.md](../persistence.md) |
+| Workload volumes | [../volumes.md](../volumes.md) |
 | Container engines | [../container-engine.md](../container-engine.md) |
 | DNS | [../dns.md](../dns.md) |
 | Workload labels/env | [../workload-metadata.md](../workload-metadata.md) |

diff --git a/docs/edgelet/modules/volumemount.md b/docs/edgelet/modules/volumemount.md
@@ -67,7 +67,7 @@ Triggered from Field Agent sync path (`fieldagent/sync.go`).
 
 ### Cleanup
 
-`CleanupMicroserviceVolumes(microserviceUUID)` when container removed — preserves shared secret/configmap data dirs where appropriate (see tests).
+`CleanupMicroserviceVolumes(microserviceUUID)` when container removed — preserves shared secret/configmap data dirs where appropriate (see tests). Operator guide: [../volumes.md](../volumes.md).
 
 ## Volume types
 

diff --git a/docs/edgelet/persistence.md b/docs/edgelet/persistence.md
@@ -42,6 +42,8 @@ Tables are grouped by **source prefix** (v1 schema):
 
 Image layers and containerd state live **outside** `diskDirectory` (for example `/var/lib/edgelet-containerd/` on the embedded engine). Backing up `edgelet.db` does **not** back up pulled images or running container filesystems.
 
+Stateful **`VOLUME`** mappings persist files under `{diskDirectory}/volumes/data/` outside SQLite. Include that tree in backups for stateful workloads. See [volumes.md](volumes.md).
+
 ---
 
 ## Backup runbook (R85)
@@ -289,3 +291,4 @@ go test ./internal/store/... ./internal/fieldagent/... ./internal/processmanager
 | [troubleshooting.md](troubleshooting.md) | Daemon won't start (includes disk space under `/var/lib/edgelet`) |
 | [control-plane.md](control-plane.md) | ControlPlane redeploy after DB wipe |
 | [container-engine.md](container-engine.md) | `/var/lib/edgelet` vs `edgelet-containerd` data paths |
+| [volumes.md](volumes.md) | `volumes/data/` lifecycle and backup scope |
diff --git a/docs/edgelet/troubleshooting.md b/docs/edgelet/troubleshooting.md
@@ -37,6 +37,8 @@ Common issues when running Edgelet on edge nodes.
    df -h /var/lib/edgelet /var/lib/edgelet-containerd
    ```
 
+   If usage grows after deleting microservices, orphaned `VOLUME` data may remain under `/var/lib/edgelet/volumes/data/` until prune runs — see [volumes.md](volumes.md).
+
 ---
 
 ## Cgroup delegation (embedded edgelet engine)

diff --git a/docs/edgelet/volumes.md b/docs/edgelet/volumes.md
@@ -0,0 +1,179 @@
+# Workload volumes and lifecycle
+
+How Edgelet stores microservice volume data on disk, what is removed when a workload is deleted, and how operators reclaim space.
+
+Applies to all container engines; **embedded `edgelet` engine** behavior is called out where it differs from Docker/Podman.
+
+---
+
+## Why this matters
+
+On the embedded `edgelet` engine, a `VOLUME` mapping does **not** use Docker's volume subsystem. Edgelet creates a persistent directory under `{diskDirectory}/volumes/data/` and bind-mounts it into the container.
+
+**Deleting a microservice removes the container, but usually not the volume data.** Orphaned data is reclaimed by **volume prune** (scheduled or manual), not at delete time.
+
+Operators who need immediate cleanup must run prune or delete the directories manually.
+
+---
+
+## On-disk layout
+
+Base path: `{diskDirectory}` (default `/var/lib/edgelet/`).
+
+```
+volumes/
+  data/{microservice-uuid}/{volume-name}/     # VOLUME-type persistent data (edgelet engine)
+  microservices/{microservice-uuid}/          # Per-MS mount staging (secrets/configmaps)
+  secrets/{volume-name}/                      # Controller secret payloads (shared)
+  configMaps/{volume-name}/                   # Controller configmap payloads (shared)
+```
+
+Image layers and container filesystems live under `/var/lib/edgelet-containerd/` (embedded engine), **not** under `volumes/data/`. Backing up `edgelet.db` alone does **not** back up `VOLUME` data — see [persistence.md](persistence.md).
+
+---
+
+## Volume mapping types
+
+| Type | `hostDestination` | Edgelet behavior |
+|------|-------------------|------------------|
+| **`VOLUME`** | Relative name, e.g. `mydata` | Creates `{diskDirectory}/volumes/data/{ms-uuid}/mydata/` and bind-mounts it |
+| **`VOLUME_MOUNT`** | Controller volume name, e.g. `my-secret` or `my-secret/key` | Materializes under `volumes/microservices/{ms-uuid}/` from shared `secrets/` or `configMaps/` trees |
+| **`BIND`** | Absolute host path, e.g. `/opt/data` | Bind-mounts that path; Edgelet never deletes it |
+
+Local deploy manifests (`edgelet deploy -f`) support **`BIND`** and **`VOLUME`** only. Controller workloads may also use **`VOLUME_MOUNT`** for secrets and configmaps synced from Pot.
+
+---
+
+## What happens when a microservice is deleted
+
+Deletion paths:
+
+- **Controller:** microservice marked `delete: true` in Pot snapshot → process manager reconciles removal
+- **Local:** `edgelet ms rm <id>` or EdgeletAPI delete of a local workload
+
+In all cases Edgelet:
+
+1. Stops and removes the container
+2. Runs volume-mount cleanup for that microservice UUID
+
+### Always removed
+
+| Path | When |
+|------|------|
+| Container / CRI sandbox | On delete |
+| `volumes/microservices/{uuid}/` | On delete (per-MS secret/configmap symlinks) |
+
+### Not removed on delete
+
+| Path | Reason |
+|------|--------|
+| `volumes/data/{uuid}/` | **VOLUME** data is intentionally persistent until prune |
+| `volumes/secrets/`, `volumes/configMaps/` | Shared controller artifacts; other workloads may still reference them |
+| **`BIND`** host paths | Operator-managed; Edgelet does not touch them |
+
+---
+
+## `deleteWithCleanup` (controller workloads)
+
+When Pot sets `deleteWithCleanup: true` on a microservice:
+
+| Engine | Effect |
+|--------|--------|
+| **docker / podman** | Container remove may pass `RemoveVolumes` to the engine; also removes the image when cleanup is requested |
+| **edgelet (embedded)** | **`deleteWithCleanup` does not delete `volumes/data/`** — the engine ignores the cleanup flag for volumes. The main extra effect is **image removal** when cleanup is requested |
+
+Local `edgelet ms rm` always uses non-cleanup removal (container only; no automatic image or volume data delete).
+
+---
+
+## When volume data is actually deleted
+
+### Volume prune (embedded engine)
+
+`PruneVolumes` removes orphaned directories under:
+
+- `volumes/data/{uuid}/`
+- `volumes/microservices/{uuid}/`
+
+for any microservice UUID that has **no running container**.
+
+Triggered by:
+
+- Scheduled prune (`pruningFrequency` / `frequencyInterval` in config) — runs containers → volumes → images
+- Manual prune:
+
+```bash
+edgelet system prune volumes
+edgelet system prune all
+```
+
+### Full deprovision (`scope=all`)
+
+Agent deprovision stops all workloads, then runs container and volume prune hooks. This is the broadest automatic cleanup short of wiping `diskDirectory`.
+
+### Controller artifact clear (`scope=local` deprovision)
+
+Clears `secrets/` and `configMaps/` and SQLite volume-mount rows, but **preserves** `volumes/data/` (local workload data).
+
+### Manual cleanup
+
+If you must reclaim space immediately after deleting a microservice:
+
+```bash
+# After confirming the MS UUID is gone and data is not needed
+sudo rm -rf /var/lib/edgelet/volumes/data/<microservice-uuid>
+```
+
+Or run `edgelet system prune volumes` to remove all orphaned volume trees.
+
+---
+
+## Operator checklist
+
+| Goal | Action |
+|------|--------|
+| Delete workload, keep data for redeploy | `edgelet ms rm` or controller delete — data remains under `volumes/data/{uuid}/` |
+| Delete workload and reclaim disk soon | `edgelet system prune volumes` or wait for scheduled prune |
+| Backup stateful workloads | Copy `edgelet.db` **and** `volumes/data/` (and BIND paths if used) |
+| Wipe node completely | Stop services, remove `diskDirectory` (see [persistence.md](persistence.md)) |
+| Secret/configmap only used by one MS | Shared trees are not auto-deleted on MS delete; controller deprovision or manual removal |
+
+---
+
+## Troubleshooting
+
+### Disk usage grows after microservices are deleted
+
+1. List remaining volume data:
+
+   ```bash
+   sudo du -sh /var/lib/edgelet/volumes/data/*
+   ```
+
+2. Confirm no container still references the UUID:
+
+   ```bash
+   edgelet ms ls
+   ```
+
+3. Prune orphans:
+
+   ```bash
+   edgelet system prune volumes
+   ```
+
+### Redeployed microservice has empty volume
+
+`VOLUME` data is keyed by **microservice UUID**. A new deployment with a new UUID gets a **new** directory under `volumes/data/`. To preserve data across UUID changes, use **`BIND`** to a fixed host path you control.
+
+---
+
+## Related documentation
+
+| Document | Topic |
+|----------|--------|
+| [manifest-reference.md](manifest-reference.md) | `spec.container.volumes` YAML fields |
+| [persistence.md](persistence.md) | SQLite backup; include `volumes/data/` for stateful apps |
+| [container-engine.md](container-engine.md) | `diskDirectory` vs containerd paths |
+| [modules/volumemount.md](modules/volumemount.md) | Implementation details (Volume Mount Manager) |
+| [modules/pruning.md](modules/pruning.md) | Pruning manager and scheduled prune order |
-Original file line number
+Diff line change
@@ Expand Up / @@ -37,6 +37,8 @@ Common issues when running Edgelet on edge nodes. @@
        df -h /var/lib/edgelet /var/lib/edgelet-containerd
        ```
+       If usage grows after deleting microservices, orphaned `VOLUME` data may remain under `/var/lib/edgelet/volumes/data/` until prune runs — see [volumes.md](volumes.md).
     ---
     ## Cgroup delegation (embedded edgelet engine)
@@ Expand Down @@