Reimplementation of Orchestration, Scheduling, and Credential Management in Golang

[yasithdev]

🎯 Conceptual Model

This PR is more of a RFC to think through what the fundamental problem that airavata should tackle. This POC is built around 6 core domain interfaces that represent the fundamental operations of distributed task execution. The POC uses a gRPC-based worker system for task execution:

┌─────────────────────────────────────────────────────────────────┐
│                    Core Domain Interfaces                       │
├─────────────────────────────────────────────────────────────────┤
│  ResourceRegistry    │  CredentialVault     │  ExperimentOrch   │
│  • Register compute  │  • Secure storage    │  • Create exper   │
│  • Register storage  │  • Unix permissions  │  • Generate tasks │
│  • Validate access   │  • Encrypt/decrypt   │  • Submit for exec│
├─────────────────────────────────────────────────────────────────┤
│  TaskScheduler       │  DataMover          │  WorkerLifecycle  │
│  • Cost optimization │  • 3-hop staging    │  • Spawn workers  │
│  • Worker distrib    │  • Persistent cache │  • gRPC workers   │
│  • Atomic assignment │  • Lineage tracking │  • Task execution │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                    gRPC Worker System                          │
├─────────────────────────────────────────────────────────────────┤
│  Worker Binary      │  Script Generation   │  Task Execution   │
│  • Standalone exec  │  • SLURM scripts     │  • Poll for tasks │
│  • gRPC client      │  • K8s manifests     │  • Execute tasks  │
│  • Auto-deployment  │  • Bare metal scripts│  • Report results │
└─────────────────────────────────────────────────────────────────┘

🔐 Credential Management Architecture

The Airavata Scheduler implements a three-layer credential architecture that separates authorization logic from storage for maximum security and scalability:

┌─────────────────────────────────────────────────────────────┐
│                    Application Layer                        │
│   (Experiments, Resources, Users, Groups)                   │
└────────────┬────────────────────────────────────────────────┘
             │
             ├──────────────────┬──────────────────┐
             │                  │                  │
┌────────────▼─────┐  ┌────────▼────────┐  ┌─────▼──────────┐
│   PostgreSQL     │  │    SpiceDB      │  │   OpenBao      │
│                  │  │                 │  │                │
│  Domain Data     │  │  Authorization  │  │  Secrets       │
│  - Users         │  │  - Permissions  │  │  - SSH Keys    │
│  - Groups        │  │  - Ownership    │  │  - Passwords   │
│  - Experiments   │  │  - Sharing      │  │  - Tokens      │
│  - Resources     │  │  - Hierarchies  │  │  (Encrypted)   │
└──────────────────┘  └─────────────────┘  └────────────────┘

Key Benefits

• 🔒 Separation of Concerns: Authorization (SpiceDB) separate from secret storage (OpenBao)
• 🛡️ Fine-grained Permissions: Read/write/delete permissions with hierarchical group inheritance
• 📋 Complete Audit Trail: All operations logged across all three systems
• 🔄 Credential Rotation: Support for automatic key rotation with zero downtime
• 👥 Group Management: Groups can contain groups with transitive permission inheritance
• 🔗 Resource Binding: Credentials bound to specific compute/storage resources

To address this issue, the code must validate the host key provided by the server against a trusted list. The single best way in this scenario is to switch from ssh.InsecureIgnoreHostKey() to ssh.FixedHostKey(pubKey) where pubKey is the trusted public key of the server. This requires loading the public key from a file (e.g., a user-specific or resource-specific location such as "~/.ssh/known_hosts" or from a configuration variable), parsing it, and setting it in the ssh.ClientConfig. All changes are localized to the block around the SSH client config creation, lines 747–751 and nearby as needed. You need to:

• Load the trusted public key (from a path—this should be obtained from known resource metadata, or you may use a new field in resource/metadata if available),
• Parse it with ssh.ParsePublicKey,
• Replace ssh.InsecureIgnoreHostKey() with ssh.FixedHostKey(pubKey),
• Import "io/ioutil" if not already present.

If there’s no pre-existing way to get the trusted public key path, a fallback could be a configurable constant or extracting a metadata field.

---

To fix this problem, we need to ensure that the WorkerID component (and, by extension, those values from which it is constructed: computeResourceID and experimentID) is sanitized/validated before being used as part of a file path. Since the path is supposed to be a filename in /tmp, we should ensure the WorkerID portion contains only safe characters, such as alphanumerics, underscores, and dashes. Any other input should result in an error. This prevents directory traversal, absolute paths, or any injection.

We should add a helper function, for example, isSafeFileName, to check if the Worker ID is safe before using it when constructing the filename. If not, the function should return an error and not proceed.

Where to change:

• In scheduler/adapters/compute_baremetal.go, in SpawnWorker, before constructing scriptPath, validate req.WorkerID with the helper.
• The helper function can be added to this file.
• If invalid, return an error instead of writing/executing the file.

---

The best way to fix this vulnerability is to validate the user input before using it to construct a file path. Since WorkerID is meant to be a single file name component, we need to ensure it does not contain any path separators (/, \) or parent directory references (..). The recommended fix is to add a check right before constructing or using the path, and reject or sanitize the input if it is invalid.

Changes to make:

• In scheduler/adapters/compute_baremetal.go, inside the SpawnWorker function, add validation right before scriptPath is constructed and used.
• If the input is invalid (contains /, \, or ..), return an error and do not proceed with file operations.
• This fix requires importing the strings package if not already imported (it is already imported).
• No changes beyond the affected code block are needed.

In general, to fix uncontrolled data used in path expression vulnerabilities, path components constructed from user input should be validated or sanitized before use. In this case, the best approach is to sanitize or validate req.WorkerID before using it in the script filename so that it cannot introduce path traversal, path separators, or otherwise produce file names outside of the intended /tmp directory.

Specifically, in scheduler/adapters/compute_baremetal.go, before using req.WorkerID, restrict it to a safe pattern (e.g., only alphanumeric, dash, and underscore), or, if that's too restrictive, at minimum reject WorkerIDs that contain /, \, or ... If invalid, return an error. This can be done at the start of the SpawnWorker method. No external dependencies are needed: standard library functions like strings.Contains and/or regular expressions are sufficient.

Add the validation at the top of the SpawnWorker function or immediately before constructing the script path. If using regex, add "regexp" to imports, but for this case, a basic check is sufficient.

---

To fix this, we must validate req.WorkerID in the SlurmAdapter's SpawnWorker method before using it to construct a filesystem path. The best practice is to ensure WorkerID is a safe filename or single path component: it should not contain /, \, or .., and should match a safe pattern (e.g., only alphanumerics, underscore, hyphen). If validation fails, return an error and do not proceed with file operations.
Edit only the scheduler/adapters/compute_slurm.go file, in the SpawnWorker() method. Specifically, right before constructing scriptPath, validate req.WorkerID using a regular expression or by checking for forbidden characters.
No new methods required; a local check using regex or strings.Contains is sufficient.

To address the problem, we need to ensure that the WorkerID used in the path is safe and cannot contain user-controlled path traversal or absolute path characters. The best way is to restrict WorkerID to a known-safe pattern (e.g., only alphanumeric, hyphen, and underscore), or at minimum reject any containing /, \, or ... Since the value is constructed from user input and an untrusted source, it's prudent to validate or sanitize it on use before constructing the file path.

Specifically, within scheduler/adapters/compute_slurm.go, in the SpawnWorker method, we should add a validation step before using req.WorkerID for the file name. If WorkerID does not match the safe pattern (e.g. using regexp), we should return a validation error. For defense in depth, it's usually best to validate as close as possible to where the variable is used in the sensitive context.

No new imports are strictly necessary, but since the snippet already imports regexp, we can use a regular expression for the safe pattern check.

---

To fix the problem, we should ensure that req.WorkerID, as it is incorporated in the temporary file path, cannot be manipulated to affect directory traversal or file naming in an unsafe way. The best way is to strictly validate req.WorkerID before using it in the filename. Since WorkerID is likely intended to be a single identifier without path components, the safest fix is to check if it contains any "/", "\\", or ".." sequences and reject such values, returning an error if found.

Implement this check near the point of file creation in SpawnWorker of scheduler/adapters/compute_slurm.go, immediately before constructing/writing to scriptPath. If the value is unsafe, return an error promptly. This avoids any risk, keeps the logic simple, and preserves existing behavior for proper, expected IDs.

No new external libraries are needed—this can be done using standard library functions like strings.Contains.

---

The best way to fix this problem is to replace the use of ssh.InsecureIgnoreHostKey() with a proper host key verification mechanism. We should obtain the allowed host’s public key—this can be read from a file or from resource metadata (the latter being convenient if the resource definition includes it). For simplicity and safety, we should use ssh.FixedHostKey, which will reject the connection unless the server presents exactly the trusted key. The fix can be implemented in the connect method by adding code to fetch the allowed host public key (e.g., from resource.Metadata["host_public_key"] if available, or alternatively from a file path), parse it, and assign ssh.FixedHostKey(parsedKey) to the HostKeyCallback field.

Necessary imports: The code already has golang.org/x/crypto/ssh imported, so no new import is needed.

Required changes:

• In the connect method of SFTPAdapter (lines 42–162), specifically:
  • Insert code to read the allowed public key (from resource metadata).
  • Parse the public key.
  • Set HostKeyCallback: ssh.FixedHostKey(pubKey) in ssh.ClientConfig.

---

The best way to fix this issue is to restrict tar archive entry paths so that they cannot escape the specified destination directory and do not contain any path traversal (..), absolute paths, or other dangerous constructs. To do this, we should:

• Use filepath.Join(destDir, header.Name) to construct the intended path.
• Clean the path using filepath.Clean.
• Convert both the target path and the destination directory to their absolute forms with filepath.Abs.
• Check that the target path is a strict subpath of the destination directory: specifically, that the absolute target path starts with the absolute destination directory path, followed by a path separator—this ensures containment.
• Skip extraction for entries which fail these checks, optionally logging a warning.

These changes should be implemented inside the extractTarGz function (scheduler/cmd/cli/data.go, lines 644–694). No new packages are needed because filepath and strings are already imported.

---