feat(networking): native mTLS with subject-name authorization for fabric inter-node communication

[rushabhvaria]

Closes #3306
Related: #3583

Summary

• Add optional TLS/mTLS configuration for the fabric port (5122), securing inter-node communication at the application layer without requiring Kubernetes NetworkPolicy or external service meshes
• Support strict mode (TLS only) and optional mode (accepts both plaintext and TLS) for zero-downtime rolling upgrades
• Periodic certificate hot-reload from disk (configurable interval, default 1h)
• Subject-name authorization (allowed-subject-names): after mTLS authentication, verify the peer's Subject CN and SANs match allowed patterns — prevents unauthorized services from connecting when using a shared CA
• Fail-safe config validation: allowed-subject-names is required when require-client-auth is true — prevents accidental fail-open. Use ["*"] to explicitly opt into CA-only trust

Motivation

Restate's security docs state: "You are expected to secure access to [the fabric port] using the network and proxy layers available in your deployment environment." The recommended approach is Kubernetes NetworkPolicy — but many production environments don't support it (shared clusters, certain CNI plugins, platform constraints). Most distributed systems (etcd, CockroachDB, Consul) offer built-in inter-node TLS — this brings Restate to parity, especially for enterprise environments.

The authorization layer addresses feedback that mTLS alone is insufficient when using a shared CA (e.g., SPIFFE). Without identity checking, any service holding a cert from the same CA could connect to the fabric port.

Configuration

[networking.tls]
mode = "strict"                          # or "optional" for rolling upgrades
cert-file = "/certs/node.crt"
key-file = "/certs/node.key"
ca-files = ["/certs/ca.crt"]
require-client-auth = true               # default: mTLS enabled
refresh-interval = "1h"                  # hot-reload certs from disk

# Authorization: required when require-client-auth is true
# Use ["*"] for CA-only trust, or specify identity patterns
allowed-subject-names = [
    "spiffe://svc.example.com/restate/*",
]

# Optional: separate client certs for outbound (inherits from above if omitted)
[networking.tls.client]
cert-file = "/certs/client.crt"
key-file = "/certs/client.key"
root-ca-files = ["/certs/client-ca.crt"]

Without [networking.tls], behavior is identical to today (plaintext).

Authorization behavior

require-client-auth	allowed-subject-names	Result
true	omitted/empty	Startup error — must specify patterns
true	["*"]	CA-only trust (explicit opt-in)
true	["spiffe://domain/*"]	Identity-based authorization
false	omitted/empty	OK — no client auth, no subject check

Design

Encryption and Authentication (mTLS):

• TLS termination at the tonic/hyper layer using rustls (already a workspace dep)
• Inbound strict: tokio-rustls::TlsAcceptor wraps TcpStream before hyper
• Inbound optional: peek first byte — 0x16 (TLS ClientHello) routes to TLS, else plaintext
• Outbound: custom tower::service_fn connector using tokio_rustls::TlsConnector, reads latest certs from ArcSwap per-connection
• Cert rotation: background tokio task reloads PEM files on interval, swaps via ArcSwap (lock-free)
• Scheme signaling: TLS-enabled nodes advertise https:// — peers use the scheme to decide connection type

Authorization (subject-name verification):

• SubjectNameVerifier wraps WebPkiClientVerifier — delegates chain validation, then checks identity
• Checks Subject CN first, then SANs (DNS names and URIs) against glob patterns
• Uses x509-parser for DER certificate parsing
• ["*"] explicitly skips identity checking (CA-only trust, no SubjectNameVerifier overhead)
• Config validation at startup prevents empty allowed-subject-names when client auth is enabled

Rolling upgrade path:

1. Deploy all nodes with mode = "optional" and TLS certs — nodes advertise https://, accept both
2. Verify all nodes communicate via TLS
3. Switch to mode = "strict" — plaintext rejected

Note on restatectl compatibility (related to #3583):
Port 5122 currently serves both internal (CoreNodeSvc) and external (ClusterCtrlSvc, NodeCtlSvc) gRPC services. In optional mode, restatectl connects via plaintext while inter-node traffic uses TLS. Once #3583 splits these into separate ports, strict mode can be applied to the internal port without affecting restatectl.

Changes

File	Change
crates/types/src/config/networking.rs	FabricTlsOptions, TlsMode, allowed-subject-names, validate() + 11 unit tests
crates/types/src/net/address.rs	PeerNetAddress::is_tls(), derive_from_bind_address_with_tls() + 2 tests
crates/core/src/network/tls.rs	NEW — TlsCertResolver, SubjectNameVerifier, cert loading, hot-reload, glob_match + 19 unit tests
crates/core/src/network/net_util.rs	TLS accept (strict) + protocol sniff (optional)
crates/core/src/network/grpc/connector.rs	Custom TLS connector for outbound https:// peers
crates/core/src/network/server_builder.rs	Thread TlsCertResolver to listener
crates/core/src/network/networking.rs	Accept TLS resolver in constructor
crates/node/src/lib.rs	TLS init + config validation at startup, spawn reloader, wire to server + connector
crates/admin/src/service.rs	Pass None for admin port (no TLS on admin)
server/tests/fabric_tls.rs	NEW — 2 integration tests (strict cluster, optional mode)

Verification

• cargo check — all modified crates compile
• cargo clippy -D warnings — zero warnings
• cargo fmt --check — clean
• 32 unit tests pass (11 config + 2 address + 19 TLS)
• 2 integration tests compile (strict cluster, optional mode)
• No regressions in existing tests

Test plan

Config and validation (11 tests):

• TOML parsing: defaults, modes, client inheritance, allowed-subject-names
• Validation: empty + client auth = error, ["*"] = OK, no client auth = skip, specific patterns = OK

TLS core (19 tests):

• PEM cert/key loading: valid, missing, empty, invalid
• is_tls() detection: https, http, bare host, UDS
• derive_from_bind_address_with_tls(): http:// vs https:// scheme
• Glob matching: exact, trailing *, middle *, prefix, multiple wildcards
• Subject-name verifier with real X.509 certs (via rcgen):
  • Accept matching SAN URI / SAN DNS / CN
  • Reject non-matching, reject no match anywhere
  • Multi-pattern authorization, CN fallback without SANs

Integration (2 tests):

• fabric_tls_strict_cluster: 3-node cluster with strict mTLS
• fabric_tls_optional_mode: 3-node cluster with optional TLS

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[rushabhvaria]

I have read the CLA Document and I hereby sign the CLA

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[nickpan47]

Overall lgtm. Minor comment on duplicated code section.

[tillrohrmann]

Thanks a lot for adding mTLS support to Restate @rushabhvaria. It looks like a great contribution.

Right now the team is a little bit busy with finalizing the 1.7 release and that's why we probably need a bit of time to give your PR the deserved attention. So please bear with us.

[rushabhvaria]

@AhmedSoliman can you approve the workflow to have claude code review the pull request? also could you help provide an ETA for potentially merging this? Thank you

[AhmedSoliman]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0b449679a4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[tillrohrmann]

@rushabhvaria sorry for the slow response time. We are currently in the process of creating the v1.7 release which requires most of our attention. At the latest, we are going to review this PR once the release is created.