feat (core): added ReplicaMap #741

ryuwd · 2026-01-21T15:30:44Z

Summary

Adds ReplicaMap, a Pydantic model for mapping Logical File Names (LFNs) to their physical replicas across distributed storage elements.

Description

ReplicaMap provides a structured, validated representation of file replica information intended to be stored in JSON format. It serves as a more user-friendly replacement for Pool XML Catalog.

Key features:

LFN/PFN validation: Automatic stripping of LFN: and PFN: prefixes, with path validation
Checksum support: Adler-32 (8 hex chars) and GUID (UUID format) checksums with format validation
Storage element tracking: Each replica is associated with a storage element identifier
Optional metadata: File size in bytes and checksum information

Example usage:

from diracx.core.models.replica_map import ReplicaMap

replica_map = ReplicaMap(root={
    "/lhcb/MC/2024/file.dst": {
        "replicas": [
            {"url": "https://storage1.cern.ch/file.dst", "se": "CERN-DST"},
            {"url": "https://storage2.in2p3.fr/file.dst", "se": "IN2P3-DST"},
        ],
        "size_bytes": 1048576,
        "checksum": {"adler32": "788c5caa"},
    }
})

read-the-docs-community · 2026-01-21T15:32:27Z

Documentation build overview

📚 diracx | 🛠️ Build #31176956 | 📁 Comparing 39bb470 against latest (a0b3053)

🔍 Preview build

Show files changed (1 files in total): 📝 1 modified | ➕ 0 added | ➖ 0 deleted

File	Status
admin/explanations/configuration/index.html	📝 modified

chrisburr · 2026-01-21T19:24:19Z

I think we need to decide on the layout of core as replica_catalogue feels to specific for the top level

aldbr · 2026-01-22T07:24:46Z

I think we need to decide on the layout of core as replica_catalogue feels to specific for the top level

I guess that, at some point (may be now?), it would make sense to have a core/models directory that would contain a module per type of models (instead of having all our models in the same models.py module). Example:

core/models/:
- auth.py
- jobs.py
- sandbox.py
- metadata.py
- search.py
- and replica_catalog.py

Any opinion?

fstagni · 2026-01-22T10:26:27Z

I think we need to decide on the layout of core as replica_catalogue feels to specific for the top level

I guess that, at some point (may be now?), it would make sense to have a core/models directory that would contain a module per type of models (instead of having all our models in the same models.py module). Example:

core/models/:

auth.py

jobs.py

sandbox.py

metadata.py

search.py

and replica_catalog.py

Any opinion?

It makes sense to me

diracx-core/src/diracx/core/models/replica_catalog.py

diracx-core/src/diracx/core/models/replica_map.py

diracx-core/src/diracx/core/models/replica_catalog.py

fstagni · 2026-01-22T13:15:56Z

I would split the refactoring in a separate PR

ryuwd · 2026-01-22T13:26:47Z

Will rebase after #746

chaen · 2026-01-23T08:54:37Z

Historically, the DFC (before LFC) has always been referred to as the Replica Catalog. Do you think you could come up with an alternative name (reflecting for example that it's a local file) ? Otherwise it's not dramatic :-)

chrisburr · 2026-01-23T15:27:49Z

My first thought is replica mapping?

ryuwd · 2026-01-23T15:29:12Z

I'm happy with ReplicaMapping or ReplicaMap (map sounds nice)

test: add unit tests for ReplicaMap feat: disallow relative paths with any depth other than zero

chaen · 2026-01-27T13:39:57Z

diracx-core/src/diracx/core/models/replica_map.py

+        if "/" in value:
+            raise ValueError(
+                "LFN must be an absolute path starting with '/' "
+                "or have no slashes at all (e.g. refers to a file in the current working directory)."


What's the use case for no slash at all ?
Even if we were to support working directory (which soulds like a bad idea at first sight), that would be more of a URL thing than an LFN

It deals with a Gaudi behaviour: #741 (comment)

Sorry I had missed it ! It makes sense

My guess multiple steps in a job e.g. the HLT1 needs to be given a mapping that includes the output of Boole earlier in the job.

chaen · 2026-01-27T13:46:41Z

diracx-core/src/diracx/core/models/replica_map.py

+        root={
+            "/lhcb/MC/2024/file.dst": {
+                "replicas": [
+                    {"url": "https://storage1.cern.ch/file.dst", "se": "CERN-DST"},


for local file, I wonder if we should enforce the file:// URI schema.

I think that's a good idea. Plus it's easier to add than remove if we change our mind.

chaen · 2026-01-27T13:49:58Z

diracx-core/src/diracx/core/models/replica_map.py

+        if "/" in value:
+            raise ValueError(
+                "LFN must be an absolute path starting with '/' "
+                "or have no slashes at all (e.g. refers to a file in the current working directory)."


Sorry I had missed it ! It makes sense

ryuwd changed the title ~~feat (core): added replica catalog to core~~ feat (core): added replica catalog Jan 21, 2026

ryuwd force-pushed the roneil-replica-catalog-json branch from fb62e9c to 6693b87 Compare January 21, 2026 15:38

ryuwd mentioned this pull request Jan 21, 2026

feat: Implement a "JSON catalog" input data resolution format DIRACGrid/dirac-cwl#62

Closed

1 task

fstagni reviewed Jan 22, 2026

View reviewed changes

diracx-core/src/diracx/core/models/replica_catalog.py Outdated Show resolved Hide resolved

diracx-core/src/diracx/core/models/replica_map.py Show resolved Hide resolved

diracx-core/src/diracx/core/models/replica_catalog.py Outdated Show resolved Hide resolved

ryuwd changed the title ~~feat (core): added replica catalog~~ feat (core): added replica catalog / refactor: diracx.core.models into a package Jan 22, 2026

ryuwd force-pushed the roneil-replica-catalog-json branch 3 times, most recently from 6a595bb to 4af54f4 Compare January 22, 2026 13:07

ryuwd mentioned this pull request Jan 22, 2026

refactor: import TokenResponse from diracx.core.models.auth DIRACGrid/DIRAC#8433

Open

ryuwd force-pushed the roneil-replica-catalog-json branch from 4af54f4 to fd28f0f Compare January 22, 2026 13:21

ryuwd changed the title ~~feat (core): added replica catalog / refactor: diracx.core.models into a package~~ feat (core): added replica catalog Jan 22, 2026

ryuwd force-pushed the roneil-replica-catalog-json branch from 04dd4d1 to 1ea1fb2 Compare January 22, 2026 13:26

aldbr linked an issue Jan 22, 2026 that may be closed by this pull request

feat: Implement a "JSON catalog" input data resolution format DIRACGrid/dirac-cwl#62

Closed

1 task

chrisburr force-pushed the roneil-replica-catalog-json branch from 1ea1fb2 to 9506a40 Compare January 23, 2026 07:59

feat: add pydantic model for ReplicaMap

2b54d75

test: add unit tests for ReplicaMap feat: disallow relative paths with any depth other than zero

ryuwd force-pushed the roneil-replica-catalog-json branch from 9506a40 to 2b54d75 Compare January 24, 2026 15:43

ryuwd changed the title ~~feat (core): added replica catalog~~ feat (core): added ReplicaMap Jan 24, 2026

chaen reviewed Jan 27, 2026

View reviewed changes

don't use FilePath

39bb470

chrisburr approved these changes Jan 28, 2026

View reviewed changes

aldbr merged commit 82e7d15 into DIRACGrid:main Jan 29, 2026
27 checks passed

feat (core): added ReplicaMap #741

feat (core): added ReplicaMap #741

Uh oh!

Conversation

ryuwd commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Description

Uh oh!

read-the-docs-community bot commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

chrisburr commented Jan 21, 2026

Uh oh!

aldbr commented Jan 22, 2026

Uh oh!

fstagni commented Jan 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fstagni commented Jan 22, 2026

Uh oh!

ryuwd commented Jan 22, 2026

Uh oh!

chaen commented Jan 23, 2026

Uh oh!

chrisburr commented Jan 23, 2026

Uh oh!

ryuwd commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ryuwd commented Jan 21, 2026 •

edited

Loading

read-the-docs-community bot commented Jan 21, 2026 •

edited

Loading

ryuwd commented Jan 23, 2026 •

edited

Loading