Add `SCCACHE_BASEDIRS` support #2521

Felixoid · 2025-12-18T21:08:45Z

This is an attempt to fix #35.

The ENV variable SCCACHE_BASEDIRS and configuration parameter basedirs are added.

As well as new tests to validate the behavior.

codecov-commenter · 2025-12-18T21:29:44Z

Codecov Report

❌ Patch coverage is 96.84729% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.90%. Comparing base (cd7dcd5) to head (7a966d4).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
src/cache/cache.rs	76.00%	18 Missing ⚠️
src/util.rs	97.54%	4 Missing ⚠️
src/config.rs	99.08%	3 Missing ⚠️
src/server.rs	78.57%	3 Missing ⚠️
src/compiler/preprocessor_cache.rs	97.64%	2 Missing ⚠️
tests/oauth.rs	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2521      +/-   ##
==========================================
+ Coverage   71.04%   71.90%   +0.86%     
==========================================
  Files          64       64              
  Lines       35369    36403    +1034     
==========================================
+ Hits        25128    26176    +1048     
+ Misses      10241    10227      -14

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

src/config.rs

src/util.rs

Felixoid · 2025-12-23T09:24:45Z

I got an idea, that basedirs should be added to the stats command as well

AJIOB · 2025-12-24T09:22:45Z

I got an idea, that basedirs should be added to the stats command as well

I think we can also provide the stat about base dir usage translation:

Number of base dir applied requests
Number of base dir skipped requests

In this case we can see, do we need to provide more/better base dirs or not

docs/Configuration.md

src/cache/cache.rs

src/config.rs

src/util.rs

Felixoid · 2025-12-24T14:58:45Z

Number of base dir applied requests

Number of base dir skipped requests

It's a tricky one. After taking a look, the number of successful substitutions is relatively easy to implement, although the counter should be threaded to the ServerStats.

But how to count the number of skipped directories? What is it? If a base_directory didn't match any of the output at all?

AJIOB · 2025-12-24T19:22:26Z

Number of base dir applied requests

Number of base dir skipped requests

It's a tricky one. After taking a look, the number of successful substitutions is relatively easy to implement, although the counter should be threaded to the ServerStats.

But how to count the number of skipped directories? What is it? If a base_directory didn't match any of the output at all?

No, just to check what number of cache requests has been converted using any of base dirs and what number was kept as an absolute one

AJIOB · 2025-12-24T19:23:42Z

About paths: we can try to normalize paths via https://docs.rs/normpath/latest/normpath/ or something similar instead of doing one more implementation.

Felixoid · 2025-12-29T12:52:06Z

About paths: we can try to normalize paths via https://docs.rs/normpath/latest/normpath/ or something similar instead of doing one more implementation.

It looks like https://doc.rust-lang.org/stable/std/path/fn.absolute.html is what I should use for both cases. It normalizes slashes, but keeps the cases as is.

Although see my comment regarding the preprocessor_output. normalize_path is used not only to needles from basedirs, but to a haystack as well. Both https://doc.rust-lang.org/stable/std/path/fn.absolute.html and https://docs.rs/normpath/latest/normpath/trait.PathExt.html#tymethod.normalize_virtually work on Path-like objects. They don't fit the purpose of the plaintext normalization, as far as I can tell.

Co-authored-by: whisperity <whisperity@gmail.com>

Felixoid · 2025-12-30T00:33:50Z

No, just to check what number of cache requests has been converted using any of base dirs and what number was kept as an absolute one

I am afraid it causes changes in too many places, including the signatures of hash_key and other significant components. I want to cut this task from the PR, if you don't mind; it's already quite huge.

src/config.rs

AJIOB · 2025-12-30T12:50:16Z

src/util.rs

+    for basedir_path in basedirs.iter() {
+        let basedir_str = basedir_path.to_string_lossy();
+        let basedir = basedir_str
+            .trim_end_matches('/')


nit: why we do this trims every time, not while loading it from config?

On one hand, it's a good point.

On the other hand, I feel it's more reliable than to hope, that the directories are cleaned in the config generation.

Probably, we can move this logic there AND test that strip_basedirs works on the output from Config.

src/util.rs

Co-authored-by: Alex Overchenko <aleksandr9809@gmail.com>

mathstuf · 2025-12-30T15:38:49Z

docs/Configuration.md


 * `SCCACHE_ALLOW_CORE_DUMPS` to enable core dumps by the server
 * `SCCACHE_CONF` configuration file path
+* `SCCACHE_BASEDIRS` base directory (or directories) to strip from paths for cache key computation. This is similar to ccache's `CCACHE_BASEDIR` and enables cache hits across different absolute paths when compiling the same source code. Multiple directories can be separated by `;` on Windows hosts and by `:` on any other. When multiple directories are specified, the longest matching prefix is used. Path matching is **case-insensitive** on Windows and **case-sensitive** on other operating systems. Environment variable takes precedence over file configuration. Only absolute paths are supported; relative paths will cause an error and prevent the server from start.


Suggested change

* `SCCACHE_BASEDIRS` base directory (or directories) to strip from paths for cache key computation. This is similar to ccache's `CCACHE_BASEDIR` and enables cache hits across different absolute paths when compiling the same source code. Multiple directories can be separated by `;` on Windows hosts and by `:` on any other. When multiple directories are specified, the longest matching prefix is used. Path matching is **case-insensitive** on Windows and **case-sensitive** on other operating systems. Environment variable takes precedence over file configuration. Only absolute paths are supported; relative paths will cause an error and prevent the server from start.

* `SCCACHE_BASEDIRS` base directory (or directories) to strip from paths for cache key computation. This is similar to ccache's `CCACHE_BASEDIR` and enables cache hits across different absolute paths when compiling the same source code. Multiple directories can be separated by `;` on Windows hosts and by `:` on any other operating system. When multiple directories are specified, the longest matching prefix is used. Path matching is **case-insensitive** on Windows and **case-sensitive** on other operating systems. Environment variable takes precedence over file configuration. Only absolute paths are supported; relative paths will cause an error and prevent the server from start.

mathstuf · 2025-12-30T15:39:17Z

README.md

+export SCCACHE_BASEDIRS=/home/user/project
+```
+
+You can also specify multiple base directories by separating them by `;` on Windows hosts and by `:` on any other. When multiple directories are provided, the longest matching prefix is used:


Suggested change

You can also specify multiple base directories by separating them by `;` on Windows hosts and by `:` on any other. When multiple directories are provided, the longest matching prefix is used:

You can also specify multiple base directories by separating them by `;` on Windows hosts and by `:` on any other operating system. When multiple directories are provided, the longest matching prefix is used:

mathstuf · 2025-12-30T15:40:24Z

docs/Configuration.md

+# When multiple paths are provided, the longest matching prefix
+# is applied.


@copilot

Suggested change

# When multiple paths are provided, the longest matching prefix

# is applied.

# When multiple matching paths are provided, the longest prefix

# is used.

mathstuf · 2025-12-30T15:42:11Z

src/cache/readonly.rs

+            .unwrap();
+
+        let tempdir = tempfile::Builder::new()
+            .prefix("sccache_test_readonly_basedirs")


Suggested change

.prefix("sccache_test_readonly_basedirs")

.prefix("readonly_storage_forwards_basedirs")

?

mathstuf · 2025-12-30T15:44:59Z

src/compiler/c.rs

+        let basedirs = [
+            PathBuf::from("/home/user1/project"),
+            PathBuf::from("/home/user2/project"),
+        ];


Can a case be added where the basedirs given to the two hash_key calls differ? I want to make sure that basedirs itself is not included in the hash.

mathstuf · 2025-12-30T16:01:14Z

src/util.rs

+/// in the format found in preprocessor output (e.g., `# 1 "/path/to/file"`).
+pub fn strip_basedirs(preprocessor_output: &[u8], basedirs: &[PathBuf]) -> Vec<u8> {
+    if basedirs.is_empty() || preprocessor_output.is_empty() {
+        return preprocessor_output.to_vec();


Why re-allocate instead of returning a Cow if the contents are unchanged?

mathstuf · 2025-12-30T16:01:39Z

src/util.rs

+    for basedir_path in basedirs.iter() {
+        let basedir_str = basedir_path.to_string_lossy();
+        let basedir = basedir_str
+            .trim_end_matches(|c| c == '/' || c == '\\')


\ should not be trimmed on non-Windows.

#2521 (comment)

so, stripping \|/ on windows, / on other OSes, and then adding the trailing /

src/util.rs

mathstuf · 2025-12-30T16:09:36Z

src/util.rs

+        // Copy everything before the match
+        result.extend_from_slice(&preprocessor_output[current_pos..match_pos]);
+        // Replace the basedir with "."
+        result.push(b'.');


Perhaps use __SCCACHE_BASEDIR__ so that it doesn't collide with a reference to a relative path that also lives under a basedir?

Imagine:

#line "./relative_path.h" #line "/a/basedir/match/relative_path.h"

I worry that this can cause false matches (though the contents are likely to be very different, it might be __FILE__ that is more likely to collide in practice).

# 0 "test.c" ||| # 0 "/tmp/test.c" # 0 "<built-in>" ||| # 0 "<built-in>" # 0 "<command-line>" ||| # 0 "<command-line>" # 1 "/usr/include/stdc-predef.h" 1 3 4 ||| # 1 "/usr/include/stdc-predef.h" 1 3 4 # 0 "<command-line>" 2 ||| # 0 "<command-line>" 2 # 1 "test.c" ||| # 1 "/tmp/test.c" # 1 "/usr/include/stdio.h" 1 3 4 ||| # 1 "/usr/include/stdio.h" 1 3 4 # 28 "/usr/include/stdio.h" 3 4 ||| # 28 "/usr/include/stdio.h" 3 4

I like the idea. But instead, I'd like to delete the match along with the following slash. And for that, we should actually normalize the basedirs by adding a trailing slash. And it's sound OK to have a current dir ./ there by default.

mathstuf · 2025-12-30T16:11:40Z

src/util.rs

+fn normalize_path(path: &[u8]) -> Vec<u8> {
+    path.iter()
+        .map(|&b| match b {
+            b'A'..=b'Z' => b + 'a' - 'A',


Suggested change

b'A'..=b'Z' => b + 'a' - 'A',

c @ b'A'..=b'Z' => c.to_ascii_lowercase(),

Maybe without any condition? Just do it for every char, not for every byte.

We can have non-english locale too

The question is how to make it universal and safe.

For example, if we are going to Turkish, where ı and i are different letters, that have İ and I as capital, it's already bad enough.

Cyrillic would work, I think, but it should be UTF, right?

At that point, you need to ask the platform for its APIs to do it. The codepage in use might not be Unicode at all.

Felixoid · 2025-12-30T22:49:27Z

src/config.rs

+                    bytes
+                }
+            };
+            basedirs.push(normalized);


The tests for trailing slashes must be implemented here for checking how it works with strip_basedirs

Add SCCACHE_BASEDIR support

dc30acc

Felixoid mentioned this pull request Dec 18, 2025

Implement an equivalent to CCACHE_BASEDIR #35

Open

Felixoid added 3 commits December 18, 2025 23:49

Fix clippy, introduce multiple basedirs logic

ace9c0e

Make codecov happy

dc589c0

Apply fmt to all files

9eb3241

Felixoid force-pushed the add-basedir-configuration branch from bfba6ec to 9eb3241 Compare December 18, 2025 23:45

Felixoid commented Dec 22, 2025

View reviewed changes

src/config.rs Outdated Show resolved Hide resolved

Felixoid commented Dec 22, 2025

View reviewed changes

src/config.rs Outdated Show resolved Hide resolved

Clean out basedir intermediate step, use basedirs

d2e6edd

Felixoid force-pushed the add-basedir-configuration branch from e818064 to d2e6edd Compare December 22, 2025 10:47

AJIOB reviewed Dec 22, 2025

View reviewed changes

src/util.rs Outdated Show resolved Hide resolved

Felixoid added 2 commits December 23, 2025 14:45

Cover Windows case for mixed slashes in paths

903bcba

Add Base directories to --show-stats output

6f47f36

Felixoid force-pushed the add-basedir-configuration branch from ca0a0db to 6f47f36 Compare December 23, 2025 13:57

Felixoid changed the title ~~Add SCCACHE_BASEDIR support~~ Add SCCACHE_BASEDIRS support Dec 23, 2025

Make basedirs match case insensitive on Windows

cf0b871

Felixoid force-pushed the add-basedir-configuration branch from b5a7d22 to cf0b871 Compare December 23, 2025 23:03

Add trace logs to strip_basedirs, fix remainings

99dde65

whisperity reviewed Dec 24, 2025

View reviewed changes

Remove outdated test

4a014de

Optimize strip_basedirs for big input

ada2650

Felixoid and others added 2 commits December 29, 2025 18:11

Apply suggestions from code review

2f4fa22

Co-authored-by: whisperity <whisperity@gmail.com>

Move basedir validation to config building

b0a22ed

Add basedirs for remote cache implementations

9ccabc3

Felixoid force-pushed the add-basedir-configuration branch from af7c5a2 to 50a4ceb Compare December 29, 2025 18:26

Felixoid added 2 commits December 30, 2025 00:13

Address review point for directories separator

fb5d0fb

Simplify strip_basedirs a bit more

bccc8f4

Felixoid force-pushed the add-basedir-configuration branch from a97218c to bccc8f4 Compare December 29, 2025 23:13

Felixoid requested review from AJIOB, mathstuf and whisperity December 30, 2025 00:36

Felixoid force-pushed the add-basedir-configuration branch from 61d0ed7 to 0887cea Compare December 30, 2025 12:09

Final cleanup of the code for leftovers

e42a1f6

Felixoid force-pushed the add-basedir-configuration branch from 0887cea to e42a1f6 Compare December 30, 2025 12:12

AJIOB reviewed Dec 30, 2025

View reviewed changes

src/config.rs Show resolved Hide resolved

AJIOB approved these changes Dec 30, 2025

View reviewed changes

Felixoid and others added 2 commits December 30, 2025 14:39

Review: less reallocations

96b8f95

Apply suggestions from code review

104221e

Co-authored-by: Alex Overchenko <aleksandr9809@gmail.com>

mathstuf suggested changes Dec 30, 2025

View reviewed changes

Move normalization to the config building, use bytes for basedirs

81e67e8

Felixoid force-pushed the add-basedir-configuration branch 3 times, most recently from fae67f8 to 50a0ea6 Compare December 30, 2025 22:20

Return unnormalized output, fix tests

7a966d4

Felixoid force-pushed the add-basedir-configuration branch from 50a0ea6 to 7a966d4 Compare December 30, 2025 22:28

Felixoid commented Dec 30, 2025

View reviewed changes

	You can also specify multiple base directories by separating them by `;` on Windows hosts and by `:` on any other. When multiple directories are provided, the longest matching prefix is used:
	You can also specify multiple base directories by separating them by `;` on Windows hosts and by `:` on any other operating system. When multiple directories are provided, the longest matching prefix is used:

		# When multiple paths are provided, the longest matching prefix
		# is applied.

	.prefix("sccache_test_readonly_basedirs")
	.prefix("readonly_storage_forwards_basedirs")

	b'A'..=b'Z' => b + 'a' - 'A',
	c @ b'A'..=b'Z' => c.to_ascii_lowercase(),

Add SCCACHE_BASEDIRS support #2521

Are you sure you want to change the base?

Add SCCACHE_BASEDIRS support #2521

Conversation

Felixoid commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Felixoid commented Dec 23, 2025

Uh oh!

AJIOB commented Dec 24, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Felixoid commented Dec 24, 2025

Uh oh!

AJIOB commented Dec 24, 2025

Uh oh!

AJIOB commented Dec 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Felixoid commented Dec 29, 2025

Uh oh!

Felixoid commented Dec 30, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Add `SCCACHE_BASEDIRS` support #2521

Add `SCCACHE_BASEDIRS` support #2521

Felixoid commented Dec 18, 2025 •

edited

Loading

codecov-commenter commented Dec 18, 2025 •

edited

Loading

AJIOB commented Dec 24, 2025 •

edited

Loading