oci: Add varlink APIs using "splitdirfdstream"

[cgwalters]

First, I discovered that actually fd-passing with varlink generally works well, and I was misguided in thinking we needed jsonrpc-fdpass.

Almost: one issue is that varlink doesn't have good support for passing a lot of file descriptors (which jsonrpc-fdpass was designed to handle).

But upon some reflection, I realized we don't need to pass a file descriptor per file, all use cases here are fine with a directory fd plus filename.

So here a new data stream format "splitdirfdstream" is implemented.

We first now use that internally when we're doing a direct pull from containers-storage for reflinking/hardlinkling.

But better: let's expose that data concept over varlink, where a varlink client can both pull or push container image layers that way.

This paves the way to a very clear mechanism for us to integrate with containers-storage or other storage stacks (like containerd) in an agnostic way.

We also now support cfsctl oci copy to copy across composefs repositories which is also implemented this way.

Generated-by: OpenCode (Claude Opus 4.8)

[giuseppe]

how will this work with podman-container-tools/container-libs#651 ?

I don't think the container-tools are going to add again varlink as a dependency for this use case only

[cgwalters]

One thing to bear in mind is there's two levels to this, the splitdirfdstream (replacing splitfdstream) and the higher level IPC mechanism by which that is passed around as representations of OCI layers. Only the latter involves varlink, and we could choose to keep using jsonrpc-fdpass for the latter in container-tools if you prefer.

Sorry about the proposed strategy change, but as I dug in more it just feels right - and I somehow again just missed the fd passing support in the varlink ecosystem. I guess one thing that changed is zlink is relatively new, and is well maintained and good code. (The varlink/rust project went through a messy time)

Would varlink be heavier than jsonrpc-fdpass? Hmm, let me see...I spent some tokens on this

varlink/go vs jsonrpc-fdpass-go: binary size comparison

🤖 Assisted-by: OpenCode (Claude Sonnet 4.6)

Measured against the containers-storage binary built from origin/main with
-ldflags="-s -w" -tags exclude_graphdriver_btrfs. Each branch adds an
equivalent stub service (listen → accept → receive → reply) in
cmd/containers-storage/rpc_bench_stub.go, guarded by a package-level var so
the linker cannot dead-code-eliminate it. Branches: bench/varlink,
bench/jsonrpc-fdpass.

Dependency	Size (KiB)	Δ KiB	Δ %	Δ packages
baseline	8,003.8	—	—	244
+varlink/go	8,071.9	+68.1	+0.851%	246
+jsonrpc-fdpass-go	8,083.9	+80.1	+1.001%	245

With a realistic service stub, both libraries are similar in weight (~68–80 KiB).
varlink/go is slightly smaller despite pulling in 2 extra packages vs 1,
likely because jsonrpc-fdpass-go depends on golang.org/x/sys for
syscall.RawConn fd passing whereas varlink/go has no external dependencies.

---

That said, varlink/go#43 needs doing.

[cgwalters]

Only the latter involves varlink, and we could choose to keep using jsonrpc-fdpass for the latter in container-tools if you prefer.

Or, it'd probably work to use gRPC for most metadata/controlplane but pass fds over a separate negotiated socket. I don't have a really strong opinion.

---

One other thing I'd say here that I think is a big cleanup is that our parsing of containers-storage: layouts now always goes through splitdirfdstream(+varlink) as an intermediary - we've added an abstraction layer there that we're now always testing. This helps pave the way much more clearly to plugging in an external binary.

Also the converse is now true - it should now be straightforward for external tooling to push content into composefs-rs in an efficient way.

[giuseppe]

That said, varlink/go#43 needs doing.

This is a blocker for the containers/storage integration. I have no preference whether it is varlink or jsonrpc, but should we hold this until we know we can use it in containers/storage too?

[cgwalters]

This is a blocker for the containers/storage integration. I have no preference whether it is varlink or jsonrpc, but should we hold this until we know we can use it in containers/storage too?

I generated a PR in varlink/go#44

I think we don't need to block this strictly speaking, it seems in the end better for us to temporarily use a patched varlink/go than to use the custom jsonrpc-fdpass right? Also, if we can take this track to finally replace the current experimental-image-proxy protocol with one thing it'd be overall a large win.

[giuseppe]

I think we don't need to block this strictly speaking, it seems in the end better for us to temporarily use a patched varlink/go than to use the custom jsonrpc-fdpass right? Also, if we can take this track to finally replace the current experimental-image-proxy protocol with one thing it'd be overall a large win.

@mheon are you fine with that?

[mheon]

Sorry for not following this one closely. Are we talking about doing a hard dependency on Varlink in order to use composefs with c/storage? Do we know what that's going to do to our binary size?

[giuseppe]

Sorry for not following this one closely. Are we talking about doing a hard dependency on Varlink in order to use composefs with c/storage? Do we know what that's going to do to our binary size?

yes, to add it as the RPC to communicate between Go and Rust.

@cgwalters made a comparison here:

#309 (comment)

[cgwalters]

OK, this one I think is ready for review/merge.

[mheon]

I don't think we have a fundamental disagreement with Varlink as an IPC protocol, though I am worried about protocol design. What kind of long term stability are we expecting with this? Are we going to have to do a hard pinning of composefs-rs against c/storage versions to ensure both ends are talking the same version of the protocol?

[cgwalters]

Are we going to have to do a hard pinning of composefs-rs against c/storage

Right, this is a confusing topic. At the current time, containers-storage uses /usr/bin/mkcomposefs from https://github.com/composefs/composefs - the C implementation.

We have an effort to fully replace that project with this one, including a new Rust mkcomposefs that has landed here that aims to be 100% compatible.

The trajectory is then to ship the composefs package with this project.

But that's not (directly) related to this PR or the PR Giuseppe is working on, which aim to expose varlink APIs on both ends.

I think the trajectory that would help the most is to replace skopeo --experimental-image-proxy (that's used by bootc/ostree) with a varlink-based interface, that's what's going on in podman-container-tools/container-libs#651 - but that needs updating to match this.

In that flow, it's more the other way around again: bootc/composefs(-rs) would be calling into skopeo ➡️ container-libs via varlink.

[cgwalters]

OK! I've gone over and cleaned this up more. I think it's ready for a wider review - I've trimmed out some garbage etc. One thing I had my agent do was inject random "dummy" fds into the producer stream to ensure that the consumer side was correctly reading the indices (and not just e.g. hardcoding 0 to access the dirfd), that shook out some bugs.

[alexlarsson]

Can you split out the oci copy cli support into a separate commit?

[alexlarsson]

Is this really the natural approach? I would personally expect --repo to be the destination repo.

[alexlarsson]

This says the algorithm must be the same, but later comments (in the implementation) seems to say it supports conversion.

[alexlarsson]

Shouldn't this read storage.conf and get the graphroot and the additional image dirs from there. And also respect the CONTAINERS_STROAGE_CONF env var.

[cgwalters]

Yes. That said, this code is basically a "placeholder", the goal is to back it with podman/skopeo.

[alexlarsson]

I did a highlevel pass through this code, but man, there is a lot of code. Also, I think @giuseppe need to review the storage related parts.