Skip to content

Indexes 4: Adds Indexes for SPK repositories#1339

Open
dcookspi wants to merge 10 commits intoindex-3-flatbuffer-and-solver-package-specfrom
index-4-indexed-repository-and-fbindex
Open

Indexes 4: Adds Indexes for SPK repositories#1339
dcookspi wants to merge 10 commits intoindex-3-flatbuffer-and-solver-package-specfrom
index-4-indexed-repository-and-fbindex

Conversation

@dcookspi
Copy link
Copy Markdown
Collaborator

@dcookspi dcookspi commented Mar 19, 2026 •
edited
Loading

Note: for info on benefits of indexing for spk solves see #1340 (5 of 5). Maybe start there and work back down to this PR if you prefer to review PRs top down.

This adds a new Indexed repository type, RepositoryIndex traits, a RepoIndex enum, and a flatbuffer index implementation in FlatBufferRepoIndex, along with some configuration. The indexing is designed to help the solvers. An index only contains information useful for solving. It does not contain enough information to do other things, such as build or test the packages.

The hierarchy of new types is roughly:

RepositoryHandle (existing spk repos)
   Indexed 
       RepoIndex
           FlatBufferRepoIndex  (makes SolverPackageSpecs, a v0 package alternate, see PR 3 of 5 below)
                flatbuffers schema (see PR 3 of 5 below)

This also updates the solver and repo based tests to include the new Indexed repo. This does not add command line tools or solver interaction for indexes. Those are in the last PR in this chain.

This is the change that adds indexes and index based packages to Spk repositories.

This is 4 of 5 chained PRs for adding indexes to spk solves:

  1. Indexes 1: Change Package and related traits to not return references to fields #1336
  2. Indexes 2: Add new_unchecked() constructors to spk schema objects #1337
  3. Indexes 3: Adds flatbuffers schema and SolverPackageSpec for indexes to spk #1338
  4. this PR
  5. Indexes 5: Adds spk repo index subcommand for index generation and updates #1340
  6. Indexes 6: Changes version_filter field in index schema #1344

@dcookspi dcookspi self-assigned this Mar 19, 2026
@dcookspi dcookspi added enhancement New feature or request SPI AOI Area of interest for SPI pr-chain This PR doesn't target the main branch, don't merge! labels Mar 19, 2026
dcookspi
dcookspi commented Mar 19, 2026
View reviewed changes
dcookspi
dcookspi commented Mar 19, 2026
View reviewed changes
dcookspi
dcookspi commented Mar 19, 2026
View reviewed changes
dcookspi
dcookspi commented Mar 19, 2026
View reviewed changes
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 19, 2026 •
edited
Loading

Codecov Report

❌ Patch coverage is 60.46948% with 421 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
crates/spk-storage/src/storage/flatbuffer_index.rs 56.41% 306 Missing ⚠️
crates/spk-storage/src/storage/indexed.rs 57.36% 55 Missing ⚠️
crates/spk-schema/src/v0/indexed_package.rs 0.00% 29 Missing ⚠️
...s/spk-storage/src/storage/flatbuffer_index_test.rs 83.72% 21 Missing ⚠️
crates/spk-storage/src/storage/repository_index.rs 85.71% 4 Missing ⚠️
crates/spk-solve/src/solvers/solver_test.rs 80.00% 2 Missing ⚠️
crates/spk-storage/src/storage/handle.rs 66.66% 2 Missing ⚠️
crates/spk-storage/src/storage/runtime.rs 0.00% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

dcookspi
dcookspi commented Mar 19, 2026
View reviewed changes
@dcookspi dcookspi force-pushed the index-4-indexed-repository-and-fbindex branch 2 times, most recently from 00757f2 to 8530adc Compare March 20, 2026 00:18
This was referenced Mar 20, 2026
Open
Open
Open
Open
@dcookspi
Copy link
Copy Markdown
Collaborator Author

dcookspi commented Mar 20, 2026

I can't replicate the test failures locally at the moment. I'll dig into them further tomororw.

@dcookspi dcookspi requested review from jrray and rydrman March 20, 2026 00:50
@dcookspi dcookspi force-pushed the index-3-flatbuffer-and-solver-package-spec branch from 47ea722 to af8dcc2 Compare March 20, 2026 19:29
@dcookspi dcookspi force-pushed the index-4-indexed-repository-and-fbindex branch 2 times, most recently from 68bb519 to 1e28bf4 Compare March 20, 2026 19:53
dcookspi
dcookspi commented Mar 20, 2026
View reviewed changes
@dcookspi dcookspi force-pushed the index-4-indexed-repository-and-fbindex branch from 1e28bf4 to 854446a Compare March 21, 2026 01:07
@dcookspi
Copy link
Copy Markdown
Collaborator Author

dcookspi commented Mar 21, 2026

I can't replicate the test failures locally at the moment. I'll dig into them further tomororw.

The issue was related to deprecated packages and the migration-to-components feature. I've put in a temporary fix so the tests pass for now. But it should not get merged as is. A direction needs to be decided with regard to deprecated packages, solves, and what to index in order to replace the temp fix with a better solution. See #1339 (comment) comment above.

@dcookspi dcookspi force-pushed the index-3-flatbuffer-and-solver-package-spec branch from af8dcc2 to 53f24d1 Compare March 25, 2026 01:07
@dcookspi dcookspi force-pushed the index-4-indexed-repository-and-fbindex branch 2 times, most recently from a58142a to b32582c Compare March 25, 2026 17:08
@dcookspi dcookspi force-pushed the index-3-flatbuffer-and-solver-package-spec branch from 53f24d1 to cb733cd Compare March 27, 2026 19:09
@dcookspi dcookspi force-pushed the index-4-indexed-repository-and-fbindex branch from b32582c to 1a43f64 Compare March 27, 2026 19:30
jrray
jrray requested changes Mar 28, 2026
View reviewed changes
@dcookspi
Copy link
Copy Markdown
Collaborator Author

dcookspi commented Mar 28, 2026

I can't replicate the test failures locally at the moment. I'll dig into them further tomororw.

These have been fixed now.

@dcookspi dcookspi force-pushed the index-4-indexed-repository-and-fbindex branch 3 times, most recently from 96ad9df to 31d99fa Compare April 1, 2026 16:26
@dcookspi dcookspi mentioned this pull request Apr 1, 2026
Open
@dcookspi dcookspi requested a review from jrray April 2, 2026 01:17
@dcookspi dcookspi mentioned this pull request Apr 2, 2026
Open
jrray
jrray requested changes Apr 2, 2026
View reviewed changes
crates/spk-storage/src/storage/flatbuffer_index.rs Outdated
let name = repo.name();

let filepath = Self::repo_index_location(repo).await?;
let temp_file = PathBuf::from(format!("{}_being_generated", filepath.display()));
Copy link
Copy Markdown
Collaborator

@jrray jrray Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use a unique filename via uuid/ulid/tempfile but you're on the right track. The temp file should be in the same directory as the final file.

Copy link
Copy Markdown
Collaborator Author

@dcookspi dcookspi Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a ulid to the temp file name.

crates/spk-storage/src/storage/flatbuffer_index.rs
temp_file.display().to_string(),
err,
));
} else {
Copy link
Copy Markdown
Collaborator

@jrray jrray Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the error handlers at this point forward should attempt to delete the temp file.

Copy link
Copy Markdown
Collaborator Author

@dcookspi dcookspi Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've pulled out the index file removal code into a function and used it for the temp files in each of the error handling points.

@dcookspi dcookspi force-pushed the index-3-flatbuffer-and-solver-package-spec branch from acf608d to d47df82 Compare April 7, 2026 20:23
dcookspi added 10 commits April 7, 2026 14:17
@dcookspi
Adds new Indexed repository type, related RepositoryIndex traits, and…
aace79d 
… flabuffers index and configuration

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
@dcookspi
Adds index schema version checks, and schema version number to index …
620ff3e 
…file name.

Also renames IndexPackage after rebasing earlier changes

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
@dcookspi
Adds temp file writing for index generation, adds read back and
2575099 
verification, and moving temp file to correct location once all that
is complete.

Refactors loading index from file to create a read_from_file method
that is used when loading an index and verifying one during the save
process.

Adds log output of the number of tables in a newly generated index
that is about to be saved to a file.

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
@dcookspi
Removed old comment
7285c9f 
Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
@dcookspi
Updates tests to use values for use_index rather than cases.
1e3fd97 
Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
@dcookspi
Updates repository handle enum to use Variantly.
d860412 
Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
@dcookspi
Removes spk repo name from the index filename.
e8fb6ad 
Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
@dcookspi
Renames flatbuffers specific config setting to verify_index_before_use
cce35ff 
so it is more general.

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
@dcookspi
Adds ulid to temp file name, and temp file removal during error handling
a844fe5 
Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
@dcookspi
Adds config sub-structure for Index and uses it in Solver and Reposit…
e5ee8b6 
…ory configs

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
@dcookspi dcookspi force-pushed the index-4-indexed-repository-and-fbindex branch from 34d1bbd to e5ee8b6 Compare April 8, 2026 00:37
@dcookspi dcookspi requested a review from jrray April 8, 2026 00:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rydrman rydrman Awaiting requested review from rydrman

@jrray jrray Awaiting requested review from jrray

Assignees

@dcookspi dcookspi

Labels

enhancement New feature or request pr-chain This PR doesn't target the main branch, don't merge! SPI AOI Area of interest for SPI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dcookspi @jrray