Tests fail on ZFS-backed storage

[obbardc]

There has been a Debian bug report that desync's tests fail on ZFS-backed storage when building in a Debian environment.
Upstream: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1137623

Bug report:

I perform most of my Debian development work in unprivileged Incus
containers with ZFS for their underlying storage. In such setups,
several of desync's tests fail in a similar way:

=== RUN TestExtractWithNonStaticSeeds
assemble_test.go:442:
Error Trace: /build/desync-1.0.1/debian/_build/src/github.com/folbricht/desync/assemble_test.go:442
Error: Received unexpected error:
invalid argument
failure cloning blocks from testdata/blob2_corrupted to /tmp/TestExtractWithNonStaticSeeds1126043460/001/out
github.com/folbricht/desync.CloneRange
/build/desync-1.0.1/debian/_build/src/github.com/folbricht/desync/ioctl_linux.go:58
github.com/folbricht/desync.(*fileSeedSegment).clone
/build/desync-1.0.1/debian/_build/src/github.com/folbricht/desync/fileseed.go:240
github.com/folbricht/desync.(*fileSeedSegment).WriteInto
/build/desync-1.0.1/debian/_build/src/github.com/folbricht/desync/fileseed.go:178
github.com/folbricht/desync.AssembleFile.func1
/build/desync-1.0.1/debian/_build/src/github.com/folbricht/desync/assemble.go:178
golang.org/x/sync/errgroup.(*Group).Go.func1
/build/desync-1.0.1/debian/_build/src/golang.org/x/sync/errgroup/errgroup.go:93
runtime.goexit
/usr/lib/go-1.26/src/runtime/asm_amd64.s:1771
Test: TestExtractWithNonStaticSeeds
--- FAIL: TestExtractWithNonStaticSeeds (0.00s)

=== RUN TestExtractCommand/extract_with_corrupted_seed
extract_test.go:94:
Error Trace: /build/desync-1.0.1/debian/_build/src/github.com/folbricht/desync/cmd/desync/extract_test.go:94
Error: Received unexpected error:
resource temporarily unavailable
failure cloning blocks from /tmp/TestExtractCommand1197972145/001/.tmp-block2462010531 to /tmp/TestExtractCommand1197972145/001/.out1.3473831054
github.com/folbricht/desync.CloneRange
/build/desync-1.0.1/debian/_build/src/github.com/folbricht/desync/ioctl_linux.go:58
github.com/folbricht/desync.(*nullChunkSection).clone
/build/desync-1.0.1/debian/_build/src/github.com/folbricht/desync/nullseed.go:152
github.com/folbricht/desync.(*nullChunkSection).WriteInto
/build/desync-1.0.1/debian/_build/src/github.com/folbricht/desync/nullseed.go:120
github.com/folbricht/desync.AssembleFile.func1
/build/desync-1.0.1/debian/_build/src/github.com/folbricht/desync/assemble.go:178
golang.org/x/sync/errgroup.(*Group).Go.func1
/build/desync-1.0.1/debian/_build/src/golang.org/x/sync/errgroup/errgroup.go:93
runtime.goexit
/usr/lib/go-1.26/src/runtime/asm_amd64.s:1771
Test: TestExtractCommand/extract_with_corrupted_seed

I'm unsure if the desync binary is also similarly broken when run in
an environment using ZFS storage. Performing the build in an identical
setup with ext4-backed storage works, so I'm not reporting this as an
RC bug.

Do you have any ideas? I guess the next step would be to figure out how to reproduce this.

[folbricht]

Thanks for the report, this turned out to be more interesting (and more serious) than expected. I was able to reproduce it on a ZFS pool and there are two things going on:

1. ZFS reflink semantics. OpenZFS 2.2+ implements FICLONERANGE (block cloning), so desync's CanClone probe — a 0-byte clone between two empty temp files — now succeeds on ZFS and desync takes the reflink path. But actual clones on ZFS are much stricter than on btrfs/XFS: offsets have to be aligned to the recordsize (128k by default), and cloning from a file whose data hasn't been committed to disk yet returns EAGAIN (with the default zfs_bclone_wait_dirty=0). That's exactly the two errors in your report: invalid argument from misaligned seed clones, and resource temporarily unavailable from the null-seed cloning its freshly written zero blockfile. On older ZFS the ioctl didn't exist, the probe failed, and desync quietly copied instead — which is why this only started appearing.

2. A latent bug exposed by 128k blocks. While verifying the fix I found the clone logic assumed every seed segment contains at least one full filesystem block. That's always true with 4k blocks, but with ZFS reporting st_blksize of 128k, segments smaller than a block cause an unsigned underflow of the clone length, copies that land outside the segment, and in one case a zero-length FICLONERANGE call — which the kernel defines as "clone to the end of the source file". Where ZFS accepted desync's clone requests, this produced silently corrupted output rather than an error. So to answer your question about whether the binary is also affected: yes — on a ZFS system with zfs_bclone_enabled=1, desync extract with seeds could either fail with these errors or, worse, corrupt the extracted file.

Both are fixed in #356: ranges without a full aligned block are copied instead of cloned, and any clone failure now falls back to copying that range (the same approach coreutils cp takes) instead of failing the extraction. Verified against a real ZFS pool (zfs 2.2.2, 128k recordsize): the previously failing tests pass, and properly aligned ranges still get reflinked, so ZFS keeps the dedup benefit.

For the Debian build environment specifically: reproducing requires zfs_bclone_enabled=1. It defaults to 0 on zfs 2.2.x and to 1 on 2.3+, which is presumably why this shows up on some ZFS systems and not others.

[obbardc]

@folbricht thanks a lot for the detail. I am actually amazed that you managed to fix that bug in the time it took me to setup a VM with ZFS and fail to reproduce the issue ;-).

[folbricht]

Well, I can't take all the credit on that, Claude did most of the hard work :)

The change is small, simple and very low risk so I cut a v1.0.2 release from the latest master. That should take care of the bug.

[obbardc]

Thanks! I tested the new release on ZFS and it works! I uploaded it into Debian too. Thanks for sorting so quickly.