progress.txt

# Jarvy Implementation Progress

> **Note**: Jarvy is a **non-profit, open-source project**. It exists to help teams provision and standardize developer workstations without relying on VMs, virtualization, containers, or any of the commercial/proprietary solutions listed in `docs/competitors/`. The goal is simple: give developers a fast, native way to get their machines set up consistently across an organization.

> **Scope Note**: Jarvy currently supports **online installations only**. Offline/air-gapped features (bundle mode, cache warming, portable mode) are not covered at this time. PRD-025 is scoped to local mirror support for organizations with internal package repositories—users still need network connectivity to those mirrors.
>
> **Not Implemented**: The following features are explicitly out of scope:
> - **Programmatic/API Access** - No REST/GraphQL API, no embeddable library crate, no webhook integrations
> - **Cost & Resource Visibility** - No bandwidth tracking, disk space reporting, or download analytics
> - **Plugin/Extension Ecosystem** - No third-party tool definitions, plugin registry, or custom package manager extensions
> - **Parallel/Group Downloads** - No parallel binary downloads or download batching. Package managers (brew, apt, etc.) handle their own download optimization, and custom installers (rustup, nvm) already use efficient strategies. Complexity not justified for marginal benefit.
> - **Tool Rollback** - No rollback functionality for individual tool installations (Jarvy self-update has rollback via `jarvy update --rollback`)
> - **Cross-Project Consistency/Management** - No centralized management or synchronization of jarvy.toml across multiple projects

---

## Completed PRDs Summary

| PRD | Title | Completed | Key Outcome |
|-----|-------|-----------|-------------|
| PRD-004 | Semantic Version Checking | 2025-01-13 | Semver operators (>=, <, ^, ~, =), smart version extraction |
| PRD-002 | Tool Specification Abstraction | 2025-01-14 | `define_tool!` macro, ~2000 lines reduced, inventory auto-registration |
| PRD-003 | Post-Install Hooks | 2026-01-14 | User-defined hooks via `[hooks]` config, timeout support |
| PRD-003.1 | Tool-Specific Default Hooks | 2026-01-15 | 29 tools with built-in hooks (shell init, completions, config) |
| PRD-008 | Environment Variables | 2026-01-15 | `[env.vars]`, `[env.secrets]`, .env generation |
| PRD-010 | CI/CD Detection | 2026-01-15 | 11 CI providers, workflow templates, `jarvy ci-info` |
| PRD-009 | Service Management | 2026-01-15 | Docker Compose & Tilt backends, `jarvy services` commands |
| PRD-001 | Optimized Tool Installation | 2026-01-17 | Batch install, parallel version checking, --jobs flag, dependency ordering |
| PRD-012 | Release Distribution | 2026-01-15 | Homebrew, crates.io, APT, RPM, AUR, winget, Chocolatey, install scripts |
| PRD-020 | Security Scanning | 2026-01-16 | SAST, SBOM, Sigstore signing, OpenSSF Scorecard, pre-commit hooks |
| PRD-016 | Developer Experience Commands | 2026-01-16 | doctor, diff, export, upgrade, search, validate, completions |
| PRD-022 | Remote Telemetry | 2026-01-18 | Unified OTEL module, PostHog removed, hooks/services/DX commands/config/CI/env instrumented |
| PRD-021 | MCP Server | 2026-01-16 | JSON-RPC 2.0, 5 tools, 3 resources, 2 prompts, rate limiting, audit |
| PRD-027 | Observability & Debugging | 2026-01-16 | Structured logging, profiler, sanitizer, `jarvy diagnose`, network tracing |
| PRD-015 | Config Validation & Remote Loading | 2026-01-16 | `jarvy validate`, `--from URL`, `--header`, 1MB size limit |
| PRD-013 | Expand Tool Coverage | 2026-01-18 | 174 tools total, Phase 1-4 complete |
| PRD-033 | Role-Based Configurations | 2026-01-17 | Role inheritance, `jarvy roles list/show/diff`, `--role` flag |
| PRD-034 | Enhanced Dependency System | 2026-01-18 | `depends_on` (strict) + `depends_on_one_of` (flexible), doctor/diff/validate integration, --ignore-missing-deps flag |
| PRD-032 | Support & Community Infrastructure | 2026-01-18 | FAQ system, contributors CLI, discussion templates, 22 integration tests |
| PRD-018 | Quality and Testing Infrastructure | 2026-01-18 | Coverage (cargo-llvm-cov + Codecov), proptest, cargo-fuzz, criterion benchmarks, mutation testing |
| PRD-019 | Corporate Proxy Support | 2026-01-18 | `[network]` config section, HTTP/HTTPS/SOCKS proxies, CA bundles, per-tool overrides |
| PRD-023 | User Onboarding & Education | 2026-01-18 | `jarvy init`, `jarvy quickstart`, `jarvy templates`, first-run detection, 20 built-in templates |
| PRD-035 | Self-Updating Functionality | 2026-01-19 | `jarvy update`, multi-method detection (Homebrew/Cargo/apt/etc), rollback support, CI auto-disable |
| PRD-037 | Main.rs Code Maintainability Refactor | 2026-01-19 | main.rs reduced 83% (3175→542 lines), CLI extracted to `src/cli/`, commands to `src/commands/` |
| PRD-038 | Hybrid Cross-Platform E2E Testing Harness | 2026-01-19 | GitHub-hosted runners (free) + AWS EC2 Spot (~$5-15/mo), 10 platforms, tiered tool testing |
| PRD-039 | Language Package Dependencies | 2026-01-23 | `[npm]`, `[pip]`, `[cargo]` sections, package manager detection, virtual env support |
| PRD-041 | Git Configuration Automation | 2026-01-24 | `[git]` section, identity, signing (SSH/GPG), aliases, defaults, credential helpers |
| PRD-043 | Configuration Drift Detection | 2026-01-24 | `[drift]` section, state capture, version comparison, file hashing, remediation |
| PRD-050 | Debug Ticket & Enhanced Logging | 2026-01-26 | File logging, rotation, sanitization, `jarvy logs`, `jarvy ticket`, ZIP bundles |

---

## Tool Statistics

| Metric | Count |
|--------|-------|
| Total tools | 174 |
| Tools with default hooks | 32 |
| Tools with dependencies | 24 |
| macOS support | 174 |
| Linux support | ~163 |
| Windows support | ~95 |

---

## Test Summary (420+ tests)

| Module | Tests |
|--------|-------|
| Version checking | 24 |
| ToolSpec/Registry | 20+ |
| Hooks | 27 |
| Environment | 9 |
| CI/CD | 19 |
| Services | 33 |
| DX Commands | 37 |
| MCP Server | 61 |
| Roles | 37 |
| Community | 22 |
| Observability | 185+ |

---

## Key Technical Learnings

### Rust Patterns
- **Macros**: Helper patterns (`@macos`, `@linux`) make complex `macro_rules!` manageable
- **Inventory Crate**: Compile-time static collection via linker - zero runtime cost
- **Serde Flatten**: `#[serde(flatten)]` for capturing dynamic TOML keys
- **Serde Untagged Enums**: Ideal for flexible TOML parsing (single value vs array)
- **LazyLock**: Use `std::sync::LazyLock` for compiled regex patterns
- **Rust 2024**: `set_var`/`remove_var` now unsafe due to race condition potential

### Version & Dependency Handling
- **Semver Gotcha**: `VersionReq::parse("3.10.5")` returns `^3.10.5`, not exact - use `=3.10.5`
- **Soft Dependencies**: Only affect installation order when both tools are in config
- **First Match Strategy**: For flexible deps in topological sort, use first matching option

### TOML & Configuration
- **Serde Conflicts**: `alias` and `rename` can compete for same TOML key
- **TOML Exclusivity**: A key can only be ONE type (array OR table, not both)

### CLI & API
- **Clap Parent Flags**: Flags on parent commands must appear BEFORE subcommand
- **ureq 3.x**: `Agent::new_with_defaults()` replaced `Agent::new()`
- **ureq Body Reading**: Use `response.into_body().as_reader()` not `.into_reader()`
- **ureq Headers**: Use `.header()` not `.set()` for request headers
- **ureq JSON**: Use `body.read_json()` for parsing JSON response bodies
- **tracing-subscriber**: Needs `env-filter` and `json` features explicitly enabled

---

## Tool Dependency Graph

```
Version Managers: node→nvm, python→pyenv, ruby→rbenv
Docker Tools: lazydocker→docker, kind→docker, minikube→docker|podman, k3d→docker, dive→docker|podman
K8s Tools: kubectl→minikube|kind|k3d|docker|podman, krew→kubectl, k9s→kubectl, helm→kubectl, stern→kubectl, kubescape→kubectl
JVM Languages: kotlin→java, scala→java
BEAM Languages: elixir→erlang
Git Security: gitleaks→git, trufflehog→git
Lua Ecosystem: luarocks→lua
Ansible Ecosystem: molecule→ansible
CI/CD Engines: dagger→docker|podman, earthly→docker|podman, localstack→docker|podman
```

**Dependency Types**:
- **Strict** (`depends_on`): ALL must be available
- **Flexible** (`depends_on_one_of`): AT LEAST ONE must be available

---

## Role-Based Configurations (PRD-033)

```toml
# Assignment
role = "frontend"              # Single role
role = ["frontend", "devops"]  # Multiple roles

# Definition
[roles.base]
tools = ["git", "docker"]

[roles.frontend]
extends = "base"               # Inheritance (max 5 levels)
description = "Frontend stack"
tools = ["node", "bun"]

[roles.frontend.tools]         # Version overrides
node = "20"
```

**CLI Commands**: `jarvy roles list|show|diff`, `jarvy setup --role <name>`

---

## Community Features (PRD-032)

### FAQ System
- **Database**: `data/faq.json` with 15 FAQs across 5 categories
- **Documentation**: `docs/faq/` with getting-started, troubleshooting, advanced guides
- **Search**: Full-text search across questions, answers, and tags

```bash
jarvy faq categories       # List all FAQ categories
jarvy faq list             # List all FAQs
jarvy faq list -c getting-started  # Filter by category
jarvy faq search "install" # Search FAQs
jarvy faq show what-is-jarvy       # Show specific FAQ
```

### Contributors System
- **Database**: `data/contributors.json` with role tracking
- **Roles**: Maintainer, Contributor, Tool Author, Documentation, Tester, Community
- **Workflow**: `.github/workflows/contributors.yml` auto-updates weekly

```bash
jarvy contributors list           # List all contributors
jarvy contributors list --role maintainer  # Filter by role
jarvy contributors list --active  # Active only
jarvy contributors show <username>  # Show details
jarvy contributors tool docker    # Show tool authors
jarvy contributors stats          # Show statistics
```

### GitHub Templates
- Discussion template for Q&A with pre-submission checklist
- Auto-update workflow for CONTRIBUTORS.md

---

## Outstanding Work

### PRD-022 Telemetry Completion (2026-01-19)
**Completed Tasks:**
- T5: Instrument tool operations - `tool_requested()`, `tool_installed()`, `tool_failed()` calls in main.rs
- T7: Remove PostHog - Deleted src/posthog.rs, replaced all posthog:: calls with telemetry::
- T9: Unit tests - 6 tests in src/telemetry.rs covering config, source, redaction
- T10: Integration tests - tests/telemetry_smoke.rs with mock OTLP collector
- T11: Documentation - CLAUDE.md already has comprehensive Telemetry section

**Remaining (Low Priority):**
- T8: Optional traces (nice-to-have)
- T12-T18: Module instrumentation (hooks, services, commands, config, CI, env, package managers) - event functions exist in telemetry.rs, modules not yet calling them

### PRD-034 Deferred Tasks (Completed 2026-01-18)
- T10: `jarvy doctor` shows dependency satisfaction (DependencyInfo struct)
- T11: `jarvy diff` shows flexible dependency resolution (DependencyResolution struct)
- T12: `jarvy validate` checks dependency configuration
- T13: `--ignore-missing-deps` flag for advanced users
- T15: Integration tests for dependency resolution (tests/dependency_resolution.rs)

### PRD-013 Phase-4 Tools (Completed 2026-01-18)
Added 18 tools: choose, grex, broot, lsd, dog, gping, julia, nim, crystal, ocaml, haskell, luarocks, sdkman, molecule, infracost, localstack, dagger, earthly

**New features**:
- broot: Default hook for shell integration (br function)
- sdkman: Default hook for shell initialization
- luarocks: Depends on lua
- molecule: Depends on ansible
- dagger, earthly, localstack: Flexible dependency on docker|podman

### PRD-018 Quality & Testing Infrastructure (Completed 2026-01-18)
**Coverage Infrastructure:**
- `cargo-llvm-cov` for code coverage measurement
- Codecov integration with PR comments
- Coverage badge in README
- Target: 80% line coverage

**Property-Based Testing:**
- `proptest` crate for fuzzing valid inputs
- `tests/property/config_properties.rs` - Config parsing properties
- `tests/property/version_properties.rs` - Version matching properties

**Fuzz Testing:**
- `cargo-fuzz` with libfuzzer (nightly required)
- 4 fuzz targets: config_parser, version_parser, toml_input, tool_spec
- CI runs short fuzz sessions on PRs
- Weekly scheduled extended fuzz runs

**Benchmarks:**
- `criterion` crate for benchmarking
- 3 benchmark suites: config_parsing, registry_lookup, version_comparison
- PR comparison using critcmp
- Regression detection with alerting

**Mutation Testing:**
- `cargo-mutants` for test quality verification
- Weekly scheduled runs
- Critical modules: config.rs, tools/common.rs
- Target: 60% mutation score

**Test Containers:**
- Dockerfiles for Ubuntu, Fedora, Alpine
- Isolated integration testing
- CI matrix across platforms

### PRD-019 Corporate Proxy Support (Completed 2026-01-18)
**Network Module** (`src/network/`):
- `config.rs` - NetworkConfig, ProxyAuth, TlsConfig, NetworkOverride types
- `resolve.rs` - ProxyResolver with priority handling (env > tool > global)
- `auth.rs` - Credential injection and secure password handling
- `propagate.rs` - Environment variable generation for child processes
- `package_managers.rs` - Git, npm, apt, dnf proxy configuration helpers
- `testing.rs` - Proxy connectivity validation

**Configuration Example:**
```toml
[network]
https_proxy = "http://proxy.corp.com:8080"
no_proxy = ["localhost", "127.0.0.1", ".corp.com"]

[network.auth]
username = "jdoe"
password = { env = "PROXY_PASSWORD" }

[network.tls]
ca_bundle = "/etc/ssl/certs/corporate-ca.crt"

[network.overrides.git]
https_proxy = "http://git-proxy.corp.com:8888"
```

**Key Features:**
- HTTP/HTTPS/SOCKS5 proxy support
- Custom CA certificate propagation
- Per-tool proxy overrides
- Secure credential handling (env, file, prompt)
- Environment variables propagated to child processes

**Tool Integration** (`src/tools/common.rs`):
- `run_with_network()` - Execute commands with proxy env vars applied
- `run_maybe_sudo_with_network()` - Sudo-aware execution with proxy support
- Proxy vars are applied via `apply_network_config()` from `src/network/propagate.rs`
- sudo runs use `-E` flag to preserve environment (proxy vars)

**Automatic System Proxy Detection:**
The `ProxyResolver` automatically checks environment variables first:
- If `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY` are already set in the environment, they are used
- Config file `[network]` section is only used if env vars are not set
- Tool-specific overrides can further customize per-tool behavior

### PRD-023 User Onboarding & Education (Completed 2026-01-18)
**New Commands:**

| Command | Purpose |
|---------|---------|
| `jarvy init` | Interactive config wizard, template support |
| `jarvy quickstart` | Guided 3-step first-run experience |
| `jarvy templates` | Browse and use 20 built-in templates |

**Modules Added:**
- `src/onboarding/` - First-run detection, project type detection, welcome banner
- `src/templates/` - Template schema, 20 built-in templates
- `src/commands/init.rs` - Interactive wizard with stack categories
- `src/commands/quickstart.rs` - Guided setup flow
- `src/commands/templates.rs` - Template list/show/use commands

**Built-in Templates:**
- **Frontend**: react, vue, nextjs, angular, svelte
- **Backend**: node-api, go-api, rust-cli, python-api, java-spring
- **Data**: python-ml, jupyter
- **DevOps**: k8s-admin, terraform, docker-dev, cicd
- **Mobile**: flutter, react-native
- **General**: essential, shell-power

**First-Run Features:**
- Automatic detection via marker file (`~/.jarvy/.jarvy_initialized`)
- Welcome banner with quick action suggestions
- CI environment detection (skips first-run prompts)
- Project type detection (Node.js, Rust, Go, Python, Java, etc.)

**CLI Examples:**
```bash
jarvy init                           # Interactive wizard
jarvy init --template react          # Use template
jarvy init --template essential --stdout  # Output to stdout
jarvy templates list                 # List all templates
jarvy templates show react           # Show template details
jarvy templates use react            # Create jarvy.toml from template
jarvy quickstart                     # Guided first-run flow
```

**Integration Tests:**
- 18 tests in `tests/onboarding_integration.rs`
- Template content validation
- File collision handling
- CLI argument parsing

### PRD-024 Scope Update (2026-01-18)
**Changed**: Replaced "Version Pinning & Lock Files" with "Automatic Tool Updates"

**Rationale**: Version pinning via lock files adds complexity without significant benefit for Jarvy's use case. Automatic tool updates better serve the goal of keeping developer environments current.

### PRD-035 Self-Updating Functionality (Completed 2026-01-19)
**Module**: `src/update/` - Self-updating functionality for Jarvy CLI

**Files Created:**
- `config.rs` - UpdateConfig, Channel (Stable/Beta/Nightly), AutoInstallPolicy, CI detection
- `method.rs` - InstallMethod detection (Homebrew, Cargo, apt, dnf, pacman, winget, Chocolatey, Scoop, Binary)
- `release.rs` - GitHub Releases API client (GitHubRelease, ReleaseAsset, ReleaseClient)
- `checker.rs` - UpdateChecker with throttling, UpdateState, version comparison
- `installer.rs` - BinaryInstaller for direct downloads, SHA256 checksum verification
- `rollback.rs` - RollbackInfo, RollbackManager for backup/restore
- `commands.rs` - UpdateAction enum, CLI command handlers
- `signature.rs` - Checksum verification utilities

**CLI Commands:**
```bash
jarvy update                      # Check and install latest update
jarvy update check                # Check for updates without installing
jarvy update --version 1.2.3      # Install specific version
jarvy update --channel beta       # Use beta channel
jarvy update --rollback           # Rollback to previous version
jarvy update history              # Show update history
jarvy update config               # Show update configuration
jarvy update enable               # Enable auto-updates
jarvy update disable              # Disable auto-updates
```

**Configuration** (`~/.jarvy/config.toml`):
```toml
[update]
enabled = true
channel = "stable"          # stable, beta, nightly
check_interval = "24h"
auto_install = "never"      # never, patch-only, patch-minor, all
show_notifications = true
```

**Environment Variables:**
- `JARVY_UPDATE=0` - Disable update checks
- `JARVY_UPDATE_CHANNEL=beta` - Override release channel
- `JARVY_PINNED_VERSION=1.2.3` - Pin to specific version

**Key Technical Details:**
- Installation method detection via path analysis and package manager queries
- Method caching in `~/.jarvy/install-method.json` for performance
- GitHub Releases API for version checking and binary downloads
- SHA256 checksum verification for binary downloads
- Backup current binary before update for rollback support
- Auto-disable in CI environments (`CI=true`)
- Version throttling to avoid excessive API calls

**Installation Methods Supported:**
1. Homebrew (macOS) - `brew upgrade jarvy`
2. Cargo (Rust) - `cargo install jarvy --force`
3. apt (Debian/Ubuntu) - `apt-get update && apt-get install jarvy`
4. dnf (Fedora/RHEL) - `dnf upgrade jarvy`
5. pacman (Arch) - `pacman -Syu jarvy`
6. winget (Windows) - `winget upgrade jarvy`
7. Chocolatey (Windows) - `choco upgrade jarvy`
8. Scoop (Windows) - `scoop update jarvy`
9. Binary (fallback) - Direct download from GitHub Releases

**Technical Learnings:**
- ureq 3.x API: Use `Agent::new_with_defaults().get().header()` not `.set()`
- Body reading: Use `response.into_body().as_reader()` not `.into_reader()`
- JSON parsing: Use `body.read_json()` for parsing response bodies
- Binary replacement: `self_update` crate for atomic binary updates

### PRD-037 Main.rs Code Maintainability Refactor (Completed 2026-01-19)
**Goal**: Reduce main.rs from 3175 lines to maintainable modules

**Files Created:**
- `src/cli/mod.rs` - CLI module structure
- `src/cli/args.rs` - Cli struct, Commands enum, OutputFormat, parse functions
- `src/cli/subcommands.rs` - TemplatesSubcommand, TelemetryAction, ServicesAction, TeamAction, LockAction, ConfigAction, UpdateSubcommand
- `src/remote.rs` - Remote config fetching with caching (fetch_remote_config, transform_github_url)
- `src/interactive.rs` - Interactive menu (user_select, print_logo)
- `src/commands/get.rs` - Get command handler
- `src/commands/tools_cmd.rs` - Tools command handler
- `src/commands/env_cmd.rs` - Env command handler
- `src/commands/ci_cmd.rs` - CI commands (run_ci_config, run_ci_info)
- `src/commands/services_cmd.rs` - Services command
- `src/commands/mcp_cmd.rs` - MCP server command
- `src/commands/telemetry_cmd.rs` - Telemetry commands
- `src/commands/team_cmd.rs` - Team commands
- `src/commands/lock_cmd.rs` - Lock commands
- `src/commands/config_cmd.rs` - Config commands
- `src/commands/roles_cmd.rs` - Roles commands
- `src/commands/bootstrap_cmd.rs` - Bootstrap command
- `src/commands/configure_cmd.rs` - Configure command
- `src/commands/setup_cmd.rs` - Setup command (~500 lines, largest handler)

**Results:**
- main.rs reduced from 3175 to 542 lines (83% reduction)
- CLI definitions cleanly separated in `src/cli/`
- All command handlers extracted to `src/commands/`
- Utility modules (remote.rs, interactive.rs) standalone
- Code compiles and passes cargo check

**Module Structure:**
```
src/
├── main.rs (542 lines - entry point, dispatch only)
├── cli/
│   ├── mod.rs (re-exports)
│   ├── args.rs (Cli struct, Commands enum)
│   └── subcommands.rs (nested enums)
├── commands/
│   ├── mod.rs (re-exports all handlers)
│   ├── setup_cmd.rs (largest - ~500 lines)
│   ├── get.rs, tools_cmd.rs, env_cmd.rs
│   ├── ci_cmd.rs, services_cmd.rs, mcp_cmd.rs
│   ├── telemetry_cmd.rs, team_cmd.rs
│   ├── lock_cmd.rs, config_cmd.rs
│   ├── roles_cmd.rs, bootstrap_cmd.rs
│   └── configure_cmd.rs
├── remote.rs (remote config fetching)
└── interactive.rs (interactive menu)
```

### PRD-038 Hybrid Cross-Platform E2E Testing Harness (Completed 2026-01-19)
**Goal**: Validate base tool installations across all supported platforms with cost-efficient infrastructure.

**Architecture Decision - Hybrid Approach:**
- **GitHub-hosted runners** (FREE): macOS Intel, macOS ARM, Ubuntu 22.04/24.04, Windows
- **AWS EC2 Spot** (~$0.01/run): Fedora 40, Arch Linux, Alpine, FreeBSD 14
- **Cost reduction**: From ~$925/month (AWS dedicated hosts) to ~$5-15/month

**Files Created:**

| File | Purpose |
|------|---------|
| `tests/e2e_base_tools.rs` | Integration test for base tool validation |
| `tests/fixtures/e2e-base-tools.toml` | Test fixture with minimal tool set |
| `.github/workflows/e2e-cross-platform.yml` | GitHub Actions workflow with matrix strategy |
| `infra/README.md` | Infrastructure documentation |
| `infra/modules/ec2-runner/main.tf` | EC2 Spot runner Terraform module |
| `infra/modules/ec2-runner/variables.tf` | Module variables |
| `infra/modules/ec2-runner/outputs.tf` | Module outputs |
| `infra/modules/ec2-runner/user-data.sh` | Bootstrap script for ephemeral runners |
| `infra/packer/fedora-40.pkr.hcl` | Packer template for Fedora AMI |
| `infra/packer/arch-linux.pkr.hcl` | Packer template for Arch Linux AMI |
| `infra/packer/alpine.pkr.hcl` | Packer template for Alpine AMI |
| `infra/packer/freebsd-14.pkr.hcl` | Packer template for FreeBSD AMI |
| `infra/environments/prod/main.tf` | Production environment Terraform |
| `infra/environments/prod/terraform.tfvars` | Environment variables template |

**Tool Test Tiers:**
```rust
tier1_core: ["git", "jq", "ripgrep", "curl", "wget"]      // Always required
tier2_runtimes: ["node", "python", "go"]                   // Language runtimes
tier3_devops: ["kubectl", "terraform"]                     // DevOps tools
tier4_dependencies: ["lazygit"]                            // Tools with deps
```

**Key Technical Details:**
- Matrix strategy with `fail-fast: false` for complete test coverage
- Ephemeral self-hosted runners that auto-terminate after job completion
- IAM roles with SSM access for GitHub runner registration
- Packer AMIs pre-install Rust toolchain and GitHub Actions runner
- Test results aggregated and posted as PR comments
- Nightly schedule (2 AM UTC) + manual trigger support

**Platform Coverage (10 total):**
| Platform | Runner Type | Status |
|----------|-------------|--------|
| macOS 13 Intel | `macos-13` | GitHub-hosted |
| macOS 14 ARM | `macos-14` | GitHub-hosted |
| Ubuntu 22.04 | `ubuntu-22.04` | GitHub-hosted |
| Ubuntu 24.04 | `ubuntu-24.04` | GitHub-hosted |
| Windows | `windows-latest` | GitHub-hosted |
| Fedora 40 | `self-hosted-fedora` | AWS EC2 Spot |
| Arch Linux | `self-hosted-arch` | AWS EC2 Spot |
| Alpine | `self-hosted-alpine` | AWS EC2 Spot |
| FreeBSD 14 | `self-hosted-freebsd` | AWS EC2 Spot |

**AWS Infrastructure Notes:**
- EC2 Spot instances (t3.medium/t3.small) for 90% cost reduction
- Security groups allow SSH (22), HTTPS (443) egress only
- User data script fetches GitHub runner token from SSM Parameter Store
- Runner auto-deregisters and instance self-terminates on job completion

**Alpine/FreeBSD Considerations:**
- Alpine uses musl libc; gcompat package required for GitHub runner
- FreeBSD has no official GitHub Actions runner support; custom solution needed

### PRD-039 Language Package Dependencies (Completed 2026-01-23)
**Goal**: Enable language-specific package management (npm, pip, cargo) alongside CLI tools.

**Module**: `src/packages/` - Language package dependency management

**Files Created:**
- `mod.rs` - Main module with `install_packages()` orchestration function
- `config.rs` - PackagesConfig, NpmConfig, PipConfig, CargoConfig, PackageSpec types
- `common.rs` - PackageError enum, run_package_command(), command_exists() utilities
- `npm.rs` - NpmHandler with package manager detection (npm/yarn/pnpm)
- `pip.rs` - PipHandler with virtual environment support
- `cargo_pkg.rs` - CargoHandler for cargo install

**Files Modified:**
- `src/config.rs` - Added npm, pip, cargo fields to Config struct
- `src/lib.rs` - Exported packages module and PackagesConfig
- `src/main.rs` - Added `mod packages;`
- `src/commands/setup_cmd.rs` - Integrated package installation after tool hooks

**Configuration Example:**
```toml
[npm]
typescript = "^5.0"
eslint = "latest"
package_manager = "pnpm"    # Auto-detected from lock file if not set

[pip]
pytest = ">=7.0"
black = "latest"
venv = ".venv"              # Virtual environment path
create_venv = true          # Auto-create venv if missing

[cargo]
cargo-watch = "latest"
cargo-nextest = "0.9"
locked = true               # Use --locked flag
```

**Key Technical Details:**
- `#[serde(flatten)]` for package maps: Allows `typescript = "^5.0"` syntax
- `#[serde(untagged)]` enum PackageSpec: Supports simple strings OR detailed specs
- Package manager detection from lock files (pnpm-lock.yaml, yarn.lock)
- Virtual environment auto-creation with activation hints
- Package installation runs after tool hooks, before environment setup
- Dry-run support shows what would be installed

**Package Manager Features:**

| Manager | Lock File | Version Operators | Extra Features |
|---------|-----------|-------------------|----------------|
| npm/yarn/pnpm | package-lock.json, yarn.lock, pnpm-lock.yaml | ^, ~, >=, latest | Auto-detect from lock file |
| pip | requirements.txt | ==, >=, ~=, !=, latest | Virtual env creation |
| cargo | N/A | exact versions, latest | --locked flag, features |

**Tests Added:** 17 unit tests covering:
- Config parsing (npm, pip, cargo)
- Package spec variants (simple vs detailed)
- Package manager detection
- Version specifier formatting
- Command existence checks

**Technical Learnings:**
- Rust 2024: `if let Some(ref x) = ...` triggers "cannot explicitly borrow" error; use `if let Some(x) = ...` instead
- `#[serde(flatten)]` with HashMap captures dynamic TOML keys while allowing fixed fields like `package_manager`
- PackageSpec with `#[serde(untagged)]` allows flexible parsing of version strings vs detailed objects

### PRD-041 Git Configuration Automation (Completed 2026-01-24)
**Goal**: Configure Git settings from `jarvy.toml` to ensure consistent identity, signing, and preferences.

**Module**: `src/git/` - Git configuration automation

**Files Created:**
- `mod.rs` - Public API exports
- `config.rs` - GitConfig, ConfigValue, ConfigScope, SigningFormat, AutoCrlf types
- `setup.rs` - GitSetup for applying configuration
- `signing.rs` - SSH/GPG signing detection and validation
- `aliases.rs` - Recommended aliases and formatting
- `identity.rs` - Interactive prompting and key detection

**Configuration Example:**
```toml
[git]
user_name = "John Doe"
user_email = { env = "GIT_EMAIL", default = "john@example.com" }
signing = true
signing_key = "~/.ssh/id_ed25519.pub"
default_branch = "main"
pull_rebase = true
auto_stash = true
push_autosetup = true
editor = "vim"
autocrlf = "input"
scope = "global"

[git.aliases]
co = "checkout"
lg = "log --oneline --graph --decorate"
```

**Key Features:**
- `ConfigValue` with environment variable support and defaults
- SSH vs GPG signing auto-detection from key path
- OS-specific credential helper defaults
- Alias configuration
- Dry-run mode support in setup

**Tests Added:** 16 unit tests covering:
- Config parsing and ConfigValue resolution
- AutoCrlf string conversion
- Signing format detection
- Key path detection
- Alias formatting

**Technical Learnings:**
- `shellexpand::tilde()` for path expansion in signing keys
- `#[serde(untagged)]` allows ConfigValue to be either plain string or `{ env, default }` object
- Credential helper auto-detection by target_os cfg flags

### PRD-043 Configuration Drift Detection (Completed 2026-01-24)
**Goal**: Detect when a developer's environment has drifted from the expected configuration.

**Module**: `src/drift/` - Configuration drift detection

**Files Created:**
- `mod.rs` - Module structure and DriftError enum
- `config.rs` - DriftConfig, VersionPolicy (Major/Minor/Patch/Exact)
- `state.rs` - EnvironmentState, ToolState, SHA-256 file hashing
- `detector.rs` - DriftDetector, DriftReport, version comparison
- `reporter.rs` - Human-readable and JSON report formatting
- `fixer.rs` - DriftFixer for remediation

**Files Modified:**
- `src/cli/args.rs` - Added Drift command
- `src/cli/subcommands.rs` - Added DriftAction enum
- `src/commands/mod.rs` - Added drift_cmd module
- `src/commands/setup_cmd.rs` - Integrated state capture after setup
- `src/config.rs` - Added drift field to Config struct
- `src/lib.rs` - Exported drift module
- `src/main.rs` - Added drift module and command routing
- `Cargo.toml` - Added `which` crate for path detection

**Configuration Example:**
```toml
[drift]
enabled = true
check_on_run = false
track_files = [".vscode/settings.json", "package.json"]
version_policy = "minor"    # major, minor, patch, exact
ignore_tools = ["vim", "neovim"]
allow_upgrades = true       # Only flag downgrades as drift
```

**CLI Commands:**
```bash
jarvy drift check              # Detect drift, exit 0 if clean
jarvy drift check --format json  # JSON output for CI
jarvy drift status             # Show baseline state
jarvy drift status -v          # Verbose with paths/methods
jarvy drift accept             # Accept current state as baseline
jarvy drift accept --tools node,docker  # Accept specific tools
jarvy drift fix                # Remediate auto-fixable issues
jarvy drift fix --dry-run      # Preview what would be fixed
```

**Key Features:**
- State capture after `jarvy setup` in `.jarvy/state.json`
- Version comparison with semver-aware policies
- SHA-256 file hashing for tracked config files
- Version direction detection (upgrade vs downgrade)
- Auto-fixable flag for reinstallable tools
- Color-coded terminal output with issue counts
- JSON output for CI/CD integration
- Exit codes: 0=clean, 1=drift detected, 2=no baseline

**Tests Added:** 27 unit tests covering:
- Version policy matching (Major, Minor, Patch, Exact)
- Config parsing and defaults
- State save/load
- Version extraction from tool output
- Upgrade/downgrade detection
- Auto-fixability determination
- JSON serialization
- Timestamp formatting

**Technical Learnings:**
- `#[serde(default = "fn")]` only works during deserialization, not `Default::default()` - implement Default manually
- `which` crate for cross-platform binary path detection
- Install method detection by analyzing binary paths (homebrew, cargo, nvm, etc.)

### PRD-050 Debug Ticket & Enhanced Logging System (Completed 2026-01-26, Unified 2026-01-27)
**Goal**: Add persistent file-based logging with rotation and debug ticket generation.

**Module**: `src/logging/` - Thin wrapper re-exporting from observability
**Module**: `src/observability/` - Unified logging through tracing-subscriber
**Module**: `src/ticket/` - Debug ticket generation

**Logging Unification (2026-01-27):**
- Merged logging into OTEL/tracing pipeline per user request
- Removed redundant files: `src/logging/sanitizer.rs`, `src/logging/writer.rs`, `src/logging/rotator.rs`, `src/logging/formatter.rs`, `src/logging/config.rs`
- `src/logging/mod.rs` now re-exports from `src/observability/logging` and provides helper functions
- Uses `tracing-appender` for daily file rotation instead of custom rotation
- Removed `flate2` dependency (no longer needed for custom gzip compression)
- File layer always writes JSON format for machine-readability
- Console layer respects user's format preference (text/JSON)
- Added `tracing-appender = "0.2"` to Cargo.toml

**Files in Logging Module (after unification):**
- `mod.rs` - Re-exports LogConfig, LogFormat, LogLevel, Sanitizer from observability; provides LogError, LogStats, default_log_directory(), current_log_file(), read_recent_logs(), get_log_stats(), clean_logs(), format_size()

**Files Created (Ticket):**
- `mod.rs` - TicketData, TicketScope, TicketError, list_tickets(), clean_tickets()
- `collector.rs` - SystemInfo, ToolInfo, TicketCollector with sanitization
- `bundler.rs` - TicketBundler for ZIP archive creation, preview_ticket()

**Files Created (Commands):**
- `src/commands/logs_cmd.rs` - Handler for logs command (view, stats, clean, config)
- `src/commands/ticket_cmd.rs` - Handler for ticket command (create, show, list, clean)

**Files Modified:**
- `src/cli/args.rs` - Added Logs and Ticket commands to Commands enum
- `src/cli/subcommands.rs` - Added LogsAction and TicketAction enums
- `src/commands/mod.rs` - Added logs_cmd and ticket_cmd modules
- `src/main.rs` - Added logging and ticket module declarations and dispatch
- `src/lib.rs` - Exported logging and ticket modules
- `Cargo.toml` - Added flate2, sys-info, num_cpus, chrono (serde feature)

**Sensitive Data Redaction Patterns:**
- API keys and tokens (generic patterns)
- Bearer tokens and Authorization headers
- Passwords (password, passwd, pwd)
- AWS credentials (access key ID, secret access key)
- GitHub tokens (ghp_, gho_, ghs_)
- npm tokens
- SSH private keys
- Email addresses (partial masking)
- Database connection strings (postgres, mysql, mongodb, redis)
- JSON secrets (secret, password, token, key, credential fields)
- Home directory path replacement with ~

**Configuration:**
```toml
[logging]
enabled = true
level = "info"         # error, warn, info, debug, trace
format = "json"        # text, json
max_file_size = "10MB"
max_files = 5
max_age_days = 30
```

**CLI Commands:**
```bash
jarvy logs view [--lines N] [--level LEVEL] [--grep PATTERN]
jarvy logs stats              # Show log statistics
jarvy logs clean [--all]      # Remove old/all logs
jarvy logs config             # Show logging configuration

jarvy ticket create [--tool NAME] [--logs N] [--output PATH] [--dry-run]
jarvy ticket show <id>        # View ticket contents
jarvy ticket list             # List existing tickets
jarvy ticket clean [--older-than DAYS]
```

**Ticket Contents:**
- `ticket.json` - Main data file with all collected information
- System info: OS, version, arch, CPU cores, memory, shell, locale, hostname
- Tool status: installed tools with versions and paths
- Configuration: Sanitized jarvy.toml content
- Environment: Filtered allowlist of safe env vars
- Logs: Recent log entries (configurable count)

**Technical Details:**
- Thread-safe writer using `Arc<Mutex<BufWriter<File>>>`
- Rotation triggered when file exceeds max_file_size
- Rotated logs named `jarvy.log.1.gz`, `jarvy.log.2.gz`, etc.
- Enforce max_files limit by deleting oldest rotated logs
- ZIP archives created in `~/.jarvy/tickets/` with unique ticket IDs
- Ticket IDs format: `JARVY-YYYYMMDD-xxxxxxxx` (date + UUID prefix)

**Tests Added:** 15+ unit tests covering:
- Log directory/file paths
- Sanitizer patterns (API keys, passwords, tokens, emails, SSH keys, DB URLs)
- Rotating file writer (create, write, size tracking, clone state sharing)
- Log rotation and compression
- Ticket scope configurations
- TOML to JSON conversion
- System info collection

**Technical Learnings:**
- Raw string regex: Use `r#""#` syntax to avoid escaping issues with `\-` inside character classes
- chrono serialization: Must enable `serde` feature for DateTime serialization
- Cow<str> returns: Call `.to_string()` when function expects String but sanitizer returns Cow
- Module visibility: Use `crate::cli::LogsAction` not `crate::cli::subcommands::LogsAction` when re-exported
- Clap subcommands: Need `#[derive(Clone, Subcommand)]` to use `.clone()` on action enum
- Loop bounds for max_files: Use `(max_files + 1)..100` not `max_files..100` (1-indexed files)
- Rust 2024 unsafe: `std::env::set_var` requires `unsafe { }` block in tests
- tracing-subscriber layers: Use `tracing_subscriber::registry().with(layer1).with(layer2).init()` for multiple outputs
- tracing-appender: Use `RollingFileAppender::new(Rotation::DAILY, dir, prefix)` for daily log rotation
- Layer composition: Each layer needs its own filter via `.with_filter(env_filter.clone())`

**New Dependencies:**
- `tracing-appender = "0.2"` - Rolling file logging with rotation
- `sys-info = "0.9"` - System information collection
- `num_cpus = "1.16"` - CPU core count detection
- `chrono` with `serde` feature - Timestamp serialization

---

### Future Enhancements
- `jarvy search --category` (category filtering)
- `jarvy validate --from URL` (remote validation)
- Package manager expansion: gem, go modules, snap, flatpak
- MacinCloud integration if GitHub macOS runners prove insufficient

---

## Documentation References

- **Architecture**: `CLAUDE.md` - module structure, tool patterns, build commands
- **MCP Server**: `docs/mcp-server.md` - Claude Desktop/Cursor/VS Code integration
- **Security**: `SECURITY.md`, `docs/MAINTAINER_SECURITY_SETUP.md`
- **Releases**: `docs/MAINTAINER_RELEASE_GUIDE.md`

---