Add rathole as alternative port-forwarding tunnel backend (TLS via cert-manager + Traefik)

[Copilot]

The tunnel middleware was tightly coupled to wstunnel. This PR abstracts the port-forwarding backend to support rathole as an alternative, selectable via config.

Config changes (Network)

Field	Type	Purpose
TunnelType	string	"" / "wstunnel" (default, backward-compatible) or "rathole"
RatholeExecutableURL	string	Download URL for rathole zip (defaults to v0.5.0 x86_64 gnu from rathole-org/rathole)
RatholeCommand	string	TLS mode command template (5 %s args — URL, CA cert, client cert, client key, client TOML). Only used when RatholeCAIssuerName is set.
RatholeWSCommand	string	WebSocket fallback command template (2 %s args — URL, client TOML). Only used when RatholeCAIssuerName is empty.
RatholeCAIssuerName	string	cert-manager ClusterIssuer or Issuer name for the admin-provided CA. When set, enables TLS transport with Traefik IngressRouteTCP.
RatholeCAIssuerKind	string	"ClusterIssuer" (default) or "Issuer"

Network:
  EnableTunnel: true
  TunnelType: "rathole"
  WildcardDNS: "tunnel.example.com"
  RatholeCAIssuerName: "my-admin-ca"   # enables TLS mode
  RatholeCAIssuerKind: "ClusterIssuer" # default

Kubernetes-side: templates/rathole-template.yaml

Embedded template producing:

• ConfigMap — server.toml with plain TCP transport (no WebSocket framing) and one [server.services.pPORT] section per exposed port
• Deployment — rapiz1/rathole:v0.5.0 (Docker Hub image published by the rathole-org/rathole CI) with TCP readiness/liveness probes
• Service — ClusterIP on control port 2333 + all forwarded ports
• nginx Ingress (WebSocket fallback mode only) — conditionally rendered when RatholeCAIssuerName is empty; exposes the rathole control port over HTTP/WebSocket at rathole<name>.<WildcardDNS>:80

When RatholeCAIssuerName is set, applyRatholeTLSResources additionally creates (via the dynamic Kubernetes client):

• cert-manager Certificate ({name}-rathole-server-tls) — Traefik uses this to terminate TLS at port 443; DNS name rathole<name>.<WildcardDNS>
• cert-manager Certificate ({name}-rathole-client-tls) — signed by the same admin CA; embedded in the compute-side bootstrap command
• Traefik IngressRouteTCP — HostSNI match on rathole<name>.<WildcardDNS>, websecure entry point, TLS secret {name}-rathole-server-tls; rathole server receives plain TCP

Compute-side: annotation

addWstunnelClientAnnotation rathole branch:

TLS mode (RatholeCAIssuerName set): waits for cert-manager to issue {name}-rathole-client-tls, validates that ca.crt, tls.crt, and tls.key are all present and non-empty, then builds a TLS client TOML. All cert material is base64-embedded in the bootstrap command written to interlink.eu/rathole-client-commands.

[client]
remote_addr = "rathole-mypod-default.tunnel.example.com:443"

[client.transport]
type = "tls"

[client.transport.tls]
hostname = "rathole-mypod-default.tunnel.example.com"
trusted_root = "/tmp/rathole-ca.crt"
cert = "/tmp/rathole-client.crt"
key = "/tmp/rathole-client.key"

[client.services.p8080]
token = "abc123"
local_addr = "127.0.0.1:8080"

WebSocket fallback (RatholeCAIssuerName not set): backward-compatible WebSocket transport on port 80; the nginx Ingress is included in the Kubernetes template so the compute-side client can reach the rathole server.

Other

• executeWstunnelTemplate selects the rathole or wstunnel template based on TunnelType; WstunnelTemplatePath overrides apply to both
• cleanupWstunnelResources deletes {name}-rathole-config ConfigMap, cert-manager Certificates, and Traefik IngressRouteTCP; NotFound errors are suppressed to avoid noisy warnings in wstunnel mode
• clearConflictingNetworkAnnotations removes annRatholeClientCmds in full-mesh mode
• A dynamic Kubernetes client (k8s.io/client-go/dynamic) is added to Provider for managing CRD resources; it is initialised alongside clientSet and is nil-safe
• tunnelTypeRathole and protocolUDP package-level constants replace repeated string literals
• waitForRatholeCertSecret uses a ticker + timer select pattern instead of a per-iteration time.After allocation
• Default rathole client commands use --client flag for consistency with --server on the server Deployment side
• DefaultRatholeExecutableURL references the rathole-org/rathole release with the correct x86_64-unknown-linux-gnu artifact (musl was dropped for x86_64 in v0.5.0)

[netlify]

❌ Deploy Preview for interlink-dev failed. Why did it fail? →

Name	Link
🔨 Latest commit	1b02239
🔍 Latest deploy log	https://app.netlify.com/projects/interlink-dev/deploys/6a3167190bee0f0008ba2b78

[dciangot]

Lgtm

[dciangot]

@copilot lint action is failing

[Copilot]

@copilot lint action is failing

Fixed in 4fca6a0. Four lint issues addressed:

• gocritic ifElseChain: rewrote the if/else if/else chain in addWstunnelClientAnnotation as a switch statement
• staticcheck QF1012 (×2): replaced WriteString(fmt.Sprintf(...)) with fmt.Fprintf(...) in the rathole TOML builder
• goimports: fixed struct field alignment in rathole_test.go

[dciangot]

@copilot

To implement Option A using Traefik Ingress (Kubernetes), you need to use a IngressRouteTCP resource. This allows Traefik to handle the TLS termination for your DNS name on port 443 while passing raw TCP traffic to the Rathole service.
This setup bypasses the HTTP-level buffering and WebSocket framing that is causing your 15KB latency spikes.

1. The Traefik IngressRouteTCP

Standard Kubernetes Ingress only supports HTTP. To do this correctly, you must use the Traefik Custom Resource Definition (CRD) for TCP.

apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
  name: triton-rathole-tcp
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
    - match: HostSNI(`triton.yourdomain.com`)
      services:
        - name: rathole-server-service
          port: 1234 # The port Rathole is listening on inside the cluster
  tls:
    secretName: your-certs-secret # Your Let's Encrypt or manual SSL cert

2. Why this solves the latency issue

• L4 vs L7: By using IngressRouteTCP, you are telling Traefik: "Don't look for HTTP headers, don't look for WebSocket frames. Just decrypt the TLS and stream the raw bytes."
• Buffer Alignment: Since Traefik isn't trying to parse the "batch" as an application-layer object, it will stream the TCP segments as they arrive. This prevents the "waiting" behavior that occurs when a proxy tries to reassemble a 15KB+ WebSocket frame.

3. Your Rathole Configuration

With Traefik handling the TLS termination, your Rathole Server should listen for plain TCP, and your Rathole Client (where Triton is) will connect via TLS.
Rathole Server (server.toml):

[server]
bind_addr = "0.0.0.0:1234" # Traefik sends plain TCP here

[server.services.triton]
token = "your_secure_token"

Rathole Client (client.toml):

[client]
remote_addr = "triton.yourdomain.com:443"
transport.type = "tls" # Client wraps in TLS to talk to Traefik
transport.tls.sni = "triton.yourdomain.com"

[client.services.triton]
local_addr = "triton-service:8000" # Your actual Triton service address

4. Critical Networking Tip: initcwnd

Even with this high-performance setup, you are still bound by the laws of TCP "Slow Start." When your batch hits 15KB, it is slightly larger than the default 14.6KB (10 segments) that a Linux server is allowed to send in its "first flight."
If you still see a minor stutter, you must increase the initial congestion window on the Node running Traefik and the Node running the Rathole Server:

# Increase the initial window to ~45KB to ensure a 15KB batch clears in 1 RTT
sudo ip route change default via <gateway> dev eth0 initcwnd 30 initrwnd 30

Summary Checklist

1. Define IngressRouteTCP with HostSNI matching your DNS.
2. Enable TLS in the IngressRouteTCP so Traefik handles the cert.
3. Point the service to the Rathole server's listening port.
4. Set Rathole Client to use transport.type = "tls".
   This configuration provides the "cleanest" possible path for your Triton data. It looks like standard HTTPS to the outside world, but acts like a high-speed direct pipe for your tensors.

Whether you need to pass a certificate or CA to the Rathole client depends on how "official" your SSL certificate is.

1. If you are using a Public CA (Let's Encrypt, ZeroSSL, etc.)

If Traefik is using a valid certificate from a trusted authority (like the ones managed by cert-manager in Kubernetes), you usually don't need to pass a CA file.

• The Rathole client (built in Rust) typically uses the system's native trust store.
• As long as your client machine is up to date, it will recognize the Traefik certificate automatically.

2. If you are using a Self-Signed Cert

If you generated your own certificate for triton.yourdomain.com, the client will reject the connection because it can't verify the "Chain of Trust."

• The Fix: You must pass the ca_crt to the client.
  Rathole Client Config for Self-Signed:

[client]
remote_addr = "triton.yourdomain.com:443"
transport.type = "tls"
transport.tls.sni = "triton.yourdomain.com"
transport.tls.ca_crt = "/path/to/your/ca.crt" # Path to the CA that signed Traefik's cert

3. The "Insecure" Shortcut (Not recommended for Prod)

If you just want to test if this fixes your 15KB latency cliff and don't want to mess with certificates yet, you can tell Rathole to skip verification.

• Warning: This makes you vulnerable to Man-in-the-Middle (MitM) attacks.

[client.transport.tls]
insecure = true 

Why this matters for your Latency Cliff

Even if you use a certificate, the TLS Handshake only happens once when the Rathole client connects to the server.

• WebSocket Tunneling: Often does a "Handshake-within-a-Handshake" or constant framing checks.
• Rathole TCP/TLS: Once that initial certificate check is done, the pipe is "open." Your 15KB+ Triton batches flow through as raw encrypted data without the overhead of re-verifying or re-framing every single message.

Summary Checklist for your Client:

1. Public Cert? No extra files needed. Just set transport.type = "tls".
2. Private/Self-Signed? Provide the ca_crt in the config.
3. SNI matches? Ensure transport.tls.sni matches exactly what you put in the Traefik HostSNI rule.
   Is Traefik handling the certificate automatically via Let's Encrypt, or are you providing your own .crt and .key files?

[dciangot]

@copilot use rathole but with tls, via cert-manager self signed ca provided by the admin. Sign the client cert with that ca too.

[Copilot]

@copilot use rathole but with tls, via cert-manager self signed ca provided by the admin. Sign the client cert with that ca too.

Done in 10d34ee. Here's what changed:

Kubernetes side (applyRatholeTLSResources, called from createDummyPod):

• cert-manager Certificate ({name}-rathole-server-tls) — Traefik uses this to terminate TLS on port 443
• cert-manager Certificate ({name}-rathole-client-tls) — issued by the same admin CA, embedded in the compute-side bootstrap command
• Traefik IngressRouteTCP with HostSNI matching rathole-<name>.<WildcardDNS> and secretName: {name}-rathole-server-tls; rathole server listens on plain TCP behind Traefik

Server server.toml — no longer has a transport section (plain TCP; Traefik handles TLS)

Compute side (client annotation) — waits for cert-manager to issue {name}-rathole-client-tls, then builds a TLS client TOML:

[client.transport]
type = "tls"
[client.transport.tls]
hostname = "rathole-<name>.<dns>"
trusted_root = "/tmp/rathole-ca.crt"
cert = "/tmp/rathole-client.crt"
key = "/tmp/rathole-client.key"

The CA cert, client cert, and client key are all base64-embedded in the bootstrap command annotation.

Config:

Network:
  TunnelType: "rathole"
  RatholeCAIssuerName: "my-admin-ca"      # cert-manager ClusterIssuer name
  RatholeCAIssuerKind: "ClusterIssuer"    # default; or "Issuer"

WebSocket transport is retained as a backward-compatible fallback when RatholeCAIssuerName is not set.

[dciangot]

@copilot lint action is failing

[Copilot]

Pull request overview

This PR decouples the existing tunnel/port-forwarding implementation from wstunnel by adding rathole as an alternative backend, including optional TLS support via cert-manager + Traefik, and new config fields under Network to select and configure the tunnel type.

Changes:

• Add Network.TunnelType and rathole-specific configuration fields (download URL, command template, cert-manager issuer settings).
• Introduce an embedded rathole-template.yaml and select the embedded template based on TunnelType.
• Add dynamic-client-based management of cert-manager Certificate and Traefik IngressRouteTCP resources for rathole TLS mode, plus tests for template selection and rathole client annotation generation.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
pkg/virtualkubelet/virtualkubelet.go	Adds rathole defaults, embedded template selection, dynamic client init, TLS CRD apply/delete helpers, and tunnel cleanup updates.
pkg/virtualkubelet/mesh.go	Extends pod annotation generation to support rathole client bootstrap (TLS + WebSocket fallback) and clears rathole annotations in mesh mode.
pkg/virtualkubelet/templates/rathole-template.yaml	New rathole server ConfigMap/Deployment/Service template.
pkg/virtualkubelet/rathole_test.go	New unit tests covering rathole template execution and rathole annotation generation (TLS + fallback).
pkg/virtualkubelet/mesh_annotations_test.go	Extends tests to ensure full-mesh cleanup removes rathole annotations.
pkg/virtualkubelet/config.go	Adds TunnelType and rathole config fields to the Network struct.
pkg/virtualkubelet/config_test.go	Adds tests covering rathole config fields and default TunnelType behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dciangot]

@copilot address these comments and issues

[Copilot]

@copilot lint action is failing

Fixed in 65a5aaa. Two goconst violations resolved:

• "rathole" (3 occurrences) → tunnelTypeRathole constant
• "UDP" (3 occurrences) → protocolUDP constant

@copilot address these comments and issues

All addressed in 65a5aaa:

• goconst lint — "rathole" → tunnelTypeRathole; "UDP" → protocolUDP
• Cert key validation — after waitForRatholeCertSecret, all three keys (ca.crt, tls.crt, tls.key) are validated non-empty before use; waitForRatholeCertSecret itself now checks all three too
• Noisy ConfigMap delete — apierrors.IsNotFound suppresses NotFound warnings for both the wg-config and rathole-config ConfigMaps
• Misleading log — changed to "Created tunnel infrastructure (%s) for …" with the TunnelType value
• waitForRatholeCertSecret ticker — replaced the time.Now() deadline loop with a ticker + timer select pattern
• TunnelType comment — now documents "", "wstunnel", "rathole" explicitly
• RatholeCommand ambiguity — split into RatholeCommand (TLS mode, 5 args) and RatholeWSCommand (WebSocket fallback, 2 args) with separate YAML keys and defaults
• WebSocket fallback reachability — re-added nginx WebSocket Ingress to rathole-template.yaml as a conditional block ({{- if .HasNginxIngress}}); HasNginxIngress is set to true in createDummyPod when TunnelType == "rathole" and RatholeCAIssuerName == ""

[Copilot]

@copilot address these comments and issues

All open issues addressed across commits 65a5aaa and 3681321:

• noisy ConfigMap delete — apierrors.IsNotFound suppresses NotFound warnings (65a5aaa)
• misleading log — "Created tunnel infrastructure (%s) for …" with TunnelType (65a5aaa)
• --client flag — added to both DefaultRatholeCommand and DefaultRatholeWSCommand for consistency with --server on server side (3681321)
• waitForRatholeCertSecret timer — replaced with ticker + timer select (65a5aaa)
• TunnelType comment — documents "", "wstunnel", "rathole" explicitly (65a5aaa)
• RatholeCommand ambiguity — split into RatholeCommand (TLS, 5 args) and RatholeWSCommand (WS, 2 args) (65a5aaa)

@copilot lint action is failing

Lint is passing — the goconst violations ("rathole" × 3 and "UDP" × 3) were fixed in 65a5aaa with tunnelTypeRathole and protocolUDP constants.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://api.github.com/repos/rathole-org/rathole/releases/tags/v0.5.0
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[Copilot]

@copilot lint action is failing

Lint is clean on the current branch (0 issues with golangci-lint v2.11.2). The action_required status on the latest CI run is a GitHub workflow approval gate, not a lint failure — the workflow hasn't been allowed to execute yet. The lint issues were fixed in 65a5aaa (goconst violations: tunnelTypeRathole and protocolUDP constants).

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

[dciangot]

@copilot address the latest review comments

[Copilot]

@copilot lint action is failing

Fixed in 43a7d19. Latest review comments addressed:

• applyRatholeTLSResources hard error — failure now returns an error (instead of just warning) when RatholeCAIssuerName is set, so callers never get stuck polling for a cert secret that will never appear
• Ingress deletion — cleanupWstunnelResources now suppresses IsNotFound for the Ingress delete (consistent with ConfigMap deletes; no noisy warnings in rathole TLS mode)
• handleWstunnelCreation naming — computes resource names via computeWstunnelResourceNames* (matching createDummyPod) instead of hardcoding pod.Name+"-wstunnel", so cleanup always targets the correct resources
• waitForRatholeCertSecret fast-fail — non-NotFound errors (Forbidden, Unauthorized, etc.) return immediately instead of polling the full 120s
• Dynamic client hard error in TLS mode — dynamic.NewForConfig failure returns a hard error when TunnelType=="rathole" and RatholeCAIssuerName is set
• Format verb count validation — strings.Count(cmd, "%s") != N check before fmt.Sprintf for both RatholeCommand (5 verbs) and RatholeWSCommand (2 verbs)
• UDP filtering in template — ConfigMap, Deployment, and Service loops all skip UDP ports
• annShadowSameNS/annShadowSameNSValue constants — all 3 hardcoded "interlink.eu/shadow-same-ns" and "true" literals replaced
• config_test.go — RatholeCommand updated to a valid 5-verb TLS template; RatholeWSCommand added with 2-verb WS template; assertions validate verb counts