Demonstrate spfs clean can corrupt repository#1282

Closed

jrray wants to merge 1 commit intomainfrom

clean-repo-corruption

Collaborator

jrray commented Oct 21, 2025

Clean can end up violating the invariant that all child objects of a parent must exist. This test demonstrates this with a manifest and a layer but the problem applies to other relationships as well.


          Demonstrate spfs clean can corrupt repository

9c85604

Clean can end up violating the invariant that all child objects of a
parent must exist. This test demonstrates this with a manifest and a
layer but the problem applies to other relationships as well.

Signed-off-by: J Robert Ray <jrray@jrray.org>

jrray added the bug label

jrray self-assigned this

jrray added a commit that referenced this pull request


          WIP: Rework clean and iter_objects for topological traversal

b313368

To address the problem demonstrated in #1282, clean needs to delete
things in a top-level order, and skip cleaning children of anything
deemed uncleanable.

The plan is to make iter_objects (now iter_items) return more detailed
information including what the item's parent(s) are, so that the
consumer of the stream can build up a graph of items. Since iter_items
can be called over RPC and used as a stream, it isn't practical to
compute the entire graph and return it in a single response. But the
stream items can be used to build up the graph incrementally.

Signed-off-by: J Robert Ray <jrray@jrray.org>

jrray mentioned this pull request

Rework clean and iter_objects for topological traversal #1288

Open

jrray added a commit that referenced this pull request


          Rework clean and iter_objects for topological traversal

74f07c7

To address the problem demonstrated in #1282, clean needs to delete
things in a top-level order, and skip cleaning children of anything
deemed uncleanable.

The plan is to make iter_objects (now iter_items) return more detailed
information including what the item's parent(s) are, so that the
consumer of the stream can build up a graph of items. Since iter_items
can be called over RPC and used as a stream, it isn't practical to
compute the entire graph and return it in a single response. But the
stream items can be used to build up the graph incrementally.

Signed-off-by: J Robert Ray <jrray@jrray.org>

jrray added a commit that referenced this pull request


          Add test that clean does not leave dangling pointers

25d9c79

This is only true when there are no concurrent writers!

This is the test from #1282 but passes as of the changes in this PR.

Signed-off-by: J Robert Ray <jrray@jrray.org>

Collaborator Author

jrray commented Oct 24, 2025

Closing now that #1288 can pass this test.

jrray closed this

jrray added a commit that referenced this pull request


          Rework clean and iter_objects for topological traversal

183da48

To address the problem demonstrated in #1282, clean needs to delete
things in a top-level order, and skip cleaning children of anything
deemed uncleanable.

The plan is to make iter_objects (now iter_items) return more detailed
information including what the item's parent(s) are, so that the
consumer of the stream can build up a graph of items. Since iter_items
can be called over RPC and used as a stream, it isn't practical to
compute the entire graph and return it in a single response. But the
stream items can be used to build up the graph incrementally.

Signed-off-by: J Robert Ray <jrray@jrray.org>

jrray added a commit that referenced this pull request


          Add test that clean does not leave dangling pointers

50b5ce6

This is only true when there are no concurrent writers!

This is the test from #1282 but passes as of the changes in this PR.

Signed-off-by: J Robert Ray <jrray@jrray.org>

jrray added a commit that referenced this pull request


          Rework clean and iter_objects for topological traversal

a9e0a7c

To address the problem demonstrated in #1282, clean needs to delete
things in a top-level order, and skip cleaning children of anything
deemed uncleanable.

The plan is to make iter_objects (now iter_items) return more detailed
information including what the item's parent(s) are, so that the
consumer of the stream can build up a graph of items. Since iter_items
can be called over RPC and used as a stream, it isn't practical to
compute the entire graph and return it in a single response. But the
stream items can be used to build up the graph incrementally.

Signed-off-by: J Robert Ray <jrray@jrray.org>

jrray added a commit that referenced this pull request


          Add test that clean does not leave dangling pointers

b405511

This is only true when there are no concurrent writers!

This is the test from #1282 but passes as of the changes in this PR.

Signed-off-by: J Robert Ray <jrray@jrray.org>

jrray added a commit that referenced this pull request


          Rework clean and iter_objects for topological traversal

e3cb34a

To address the problem demonstrated in #1282, clean needs to delete
things in a top-level order, and skip cleaning children of anything
deemed uncleanable.

The plan is to make iter_objects (now iter_items) return more detailed
information including what the item's parent(s) are, so that the
consumer of the stream can build up a graph of items. Since iter_items
can be called over RPC and used as a stream, it isn't practical to
compute the entire graph and return it in a single response. But the
stream items can be used to build up the graph incrementally.

Signed-off-by: J Robert Ray <jrray@jrray.org>

jrray added a commit that referenced this pull request


          Add test that clean does not leave dangling pointers

ba586aa

This is only true when there are no concurrent writers!

This is the test from #1282 but passes as of the changes in this PR.

Signed-off-by: J Robert Ray <jrray@jrray.org>

jrray added a commit that referenced this pull request


          Rework clean and iter_objects for topological traversal

28c2143

To address the problem demonstrated in #1282, clean needs to delete
things in a top-level order, and skip cleaning children of anything
deemed uncleanable.

The plan is to make iter_objects (now iter_items) return more detailed
information including what the item's parent(s) are, so that the
consumer of the stream can build up a graph of items. Since iter_items
can be called over RPC and used as a stream, it isn't practical to
compute the entire graph and return it in a single response. But the
stream items can be used to build up the graph incrementally.

Signed-off-by: J Robert Ray <jrray@jrray.org>

jrray added a commit that referenced this pull request


          Add test that clean does not leave dangling pointers

9aae863

This is only true when there are no concurrent writers!

This is the test from #1282 but passes as of the changes in this PR.

Signed-off-by: J Robert Ray <jrray@jrray.org>

jrray added a commit that referenced this pull request


          Rework clean and iter_objects for topological traversal

e5dc451

To address the problem demonstrated in #1282, clean needs to delete
things in a top-level order, and skip cleaning children of anything
deemed uncleanable.

The plan is to make iter_objects (now iter_items) return more detailed
information including what the item's parent(s) are, so that the
consumer of the stream can build up a graph of items. Since iter_items
can be called over RPC and used as a stream, it isn't practical to
compute the entire graph and return it in a single response. But the
stream items can be used to build up the graph incrementally.

Signed-off-by: J Robert Ray <jrray@jrray.org>

jrray added a commit that referenced this pull request


          Add test that clean does not leave dangling pointers

8fecf80

This is only true when there are no concurrent writers!

This is the test from #1282 but passes as of the changes in this PR.

Signed-off-by: J Robert Ray <jrray@jrray.org>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug