From 19da66c06175c3beeb93a07691eb3a92aebddd43 Mon Sep 17 00:00:00 2001 From: Andrey Zhukov Date: Wed, 10 Jun 2026 17:53:23 +0300 Subject: [PATCH 01/12] Pivot to harness-first product: governance, recipes, orient bundle contract Specs 087/084/088: constitution 1.1.0, Go freeze policy, ADR 001 (Go maintenance mode), OSS tool profiles and recipes (jscpd/semgrep/syft/ctags), orient bundle schema with build script, fixture smoke, and legacy map bridge. Co-authored-by: Cursor --- .gitignore | 3 + .specify/memory/constitution.md | 24 +- docs/adr/001-go-cli-fate.md | 41 +++ docs/harness/GO-FREEZE-POLICY.md | 43 +++ docs/harness/tool-profiles/ast-index.md | 21 ++ docs/harness/tool-profiles/codegraph.md | 21 ++ docs/harness/tool-profiles/jscpd.md | 30 +++ docs/harness/tool-profiles/semgrep.md | 29 ++ docs/harness/tool-profiles/syft-cyclonedx.md | 29 ++ .../tool-profiles/understand-anything.md | 30 +++ docs/harness/tool-profiles/universal-ctags.md | 29 ++ ...26-06-10-understand-anything-fork-spike.md | 68 +++++ .../plan.md | 31 +++ .../spec.md | 4 +- .../tasks.md | 6 + .../plan.md | 19 ++ .../spec.md | 10 +- .../tasks.md | 6 + docs/specs/087-harness-first-product/plan.md | 37 +++ docs/specs/087-harness-first-product/spec.md | 83 ++++++ docs/specs/087-harness-first-product/tasks.md | 12 + docs/specs/088-orient-bundle-contract/plan.md | 15 ++ docs/specs/088-orient-bundle-contract/spec.md | 47 ++++ .../specs/088-orient-bundle-contract/tasks.md | 6 + harness/SKILL.md | 98 +++++++ harness/codex-claude/INSTALL-PROMPT.md | 22 ++ harness/contracts/orient-bundle.schema.json | 89 +++++++ harness/cursor/portolan-orient.mdc | 17 ++ harness/guardrails/citation-rules.md | 17 ++ harness/guardrails/evidence-states.md | 15 ++ harness/guardrails/unknown-block.md | 15 ++ harness/opencode/INSTALL-PROMPT.md | 25 ++ harness/recipes/deps-syft-cyclonedx.md | 39 +++ harness/recipes/duplication-jscpd.md | 53 ++++ .../recipes/semgrep-rules/portolan-local.yaml | 11 + harness/recipes/static-semgrep-local.md | 42 +++ harness/recipes/symbols-ctags.md | 31 +++ .../orient-bundle/orient/gaps.jsonl | 1 + .../orient-bundle/orient/graph-slice.json | 31 +++ .../orient-bundle/orient/hotspots.jsonl | 3 + .../orient-bundle/orient/manifest.json | 7 + .../orient/producers/jscpd/jscpd-report.json | 9 + .../orient/producers/semgrep/findings.json | 14 + .../orient-bundle/orient/repos.json | 7 + .../producers/jscpd/jscpd-report.json | 9 + .../producers/semgrep/findings.json | 14 + .../orient-bundle/target/sample.go | 4 + scripts/build-orient-bundle.sh | 247 ++++++++++++++++++ scripts/harness-orient-smoke.sh | 35 +++ scripts/orient-export-from-map.sh | 88 +++++++ 50 files changed, 1575 insertions(+), 12 deletions(-) create mode 100644 docs/adr/001-go-cli-fate.md create mode 100644 docs/harness/GO-FREEZE-POLICY.md create mode 100644 docs/harness/tool-profiles/ast-index.md create mode 100644 docs/harness/tool-profiles/codegraph.md create mode 100644 docs/harness/tool-profiles/jscpd.md create mode 100644 docs/harness/tool-profiles/semgrep.md create mode 100644 docs/harness/tool-profiles/syft-cyclonedx.md create mode 100644 docs/harness/tool-profiles/understand-anything.md create mode 100644 docs/harness/tool-profiles/universal-ctags.md create mode 100644 docs/research/2026-06-10-understand-anything-fork-spike.md create mode 100644 docs/specs/084-external-tool-evaluation-profiles/plan.md create mode 100644 docs/specs/084-external-tool-evaluation-profiles/tasks.md create mode 100644 docs/specs/086-evidence-navigation-ux-patterns/plan.md create mode 100644 docs/specs/086-evidence-navigation-ux-patterns/tasks.md create mode 100644 docs/specs/087-harness-first-product/plan.md create mode 100644 docs/specs/087-harness-first-product/spec.md create mode 100644 docs/specs/087-harness-first-product/tasks.md create mode 100644 docs/specs/088-orient-bundle-contract/plan.md create mode 100644 docs/specs/088-orient-bundle-contract/spec.md create mode 100644 docs/specs/088-orient-bundle-contract/tasks.md create mode 100644 harness/SKILL.md create mode 100644 harness/codex-claude/INSTALL-PROMPT.md create mode 100644 harness/contracts/orient-bundle.schema.json create mode 100644 harness/cursor/portolan-orient.mdc create mode 100644 harness/guardrails/citation-rules.md create mode 100644 harness/guardrails/evidence-states.md create mode 100644 harness/guardrails/unknown-block.md create mode 100644 harness/opencode/INSTALL-PROMPT.md create mode 100644 harness/recipes/deps-syft-cyclonedx.md create mode 100644 harness/recipes/duplication-jscpd.md create mode 100644 harness/recipes/semgrep-rules/portolan-local.yaml create mode 100644 harness/recipes/static-semgrep-local.md create mode 100644 harness/recipes/symbols-ctags.md create mode 100644 internal/testfixtures/orient-bundle/orient/gaps.jsonl create mode 100644 internal/testfixtures/orient-bundle/orient/graph-slice.json create mode 100644 internal/testfixtures/orient-bundle/orient/hotspots.jsonl create mode 100644 internal/testfixtures/orient-bundle/orient/manifest.json create mode 100644 internal/testfixtures/orient-bundle/orient/producers/jscpd/jscpd-report.json create mode 100644 internal/testfixtures/orient-bundle/orient/producers/semgrep/findings.json create mode 100644 internal/testfixtures/orient-bundle/orient/repos.json create mode 100644 internal/testfixtures/orient-bundle/producers/jscpd/jscpd-report.json create mode 100644 internal/testfixtures/orient-bundle/producers/semgrep/findings.json create mode 100644 internal/testfixtures/orient-bundle/target/sample.go create mode 100755 scripts/build-orient-bundle.sh create mode 100755 scripts/harness-orient-smoke.sh create mode 100755 scripts/orient-export-from-map.sh diff --git a/.gitignore b/.gitignore index 55818f49..c7e96686 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,9 @@ .DS_Store /bin/ /dist/ +viewer/dist/ +**/orient-smoke/ +.cursor/ /.portolan/ /coverage.out *.out diff --git a/.specify/memory/constitution.md b/.specify/memory/constitution.md index 4135a1a5..5b871d1f 100644 --- a/.specify/memory/constitution.md +++ b/.specify/memory/constitution.md @@ -42,10 +42,21 @@ be verified through link, schema, and placeholder checks instead of code tests. ## Product Constraints -- Primary implementation language: Go. -- Runtime default: local CLI. -- Default output: machine-readable evidence graph plus optional human-readable - packet generated from the same graph. +- Primary delivery: harness artifacts (`harness/` skills, recipes, guardrails, + contracts) plus optional local orient viewer output. +- Implementation language: Go remains for the legacy CLI and normalization + library; new product slices may use shell, TypeScript, or other local tooling + when a spec documents the boundary. +- Runtime default: harness-first workflow (OSS recipes → orient bundle → viewer); + legacy Go CLI is frozen for new features until the Go decision gate resolves. +- Default output: ranked `orient/` hotspot bundle for navigation; legacy + machine-readable evidence graph remains supported as an optional bridge. +- Local viewer: a read-only static orient viewer may be served locally for the + duration of a user session; it must not mutate targets or require network by + default. +- Viewer truth boundary: graph nodes in the orient viewer must come from imported + producer evidence or Portolan normalization; LLM-authored graphs are UX-only and + must never be labeled `source-visible`. - Default privacy posture: no raw private source snippets, prompts, credentials, provider URLs, or customer-sensitive payloads in committed fixtures. - Default integration posture: import files and exported tool outputs before @@ -81,4 +92,7 @@ Changes require: - a migration note if existing specs become stale; - fresh verification using the baseline checks in `AGENTS.md`. -**Version**: 1.0.0 | **Ratified**: 2026-05-20 | **Last Amended**: 2026-05-20 +**Version**: 1.1.0 | **Ratified**: 2026-05-20 | **Last Amended**: 2026-06-10 + +Migration note (1.1.0): harness-first pivot per spec 087. Existing Go CLI specs +remain valid as legacy bridges; new MVP work targets `harness/` and `viewer/`. diff --git a/docs/adr/001-go-cli-fate.md b/docs/adr/001-go-cli-fate.md new file mode 100644 index 00000000..8a45f836 --- /dev/null +++ b/docs/adr/001-go-cli-fate.md @@ -0,0 +1,41 @@ +# ADR 001: Go CLI Fate (Decision Gate) + +**Status**: Provisional — thin maintenance layer (2026-06-10) + +**Context**: Harness pivot (spec 087) ships recipes, orient bundle, and viewer +without requiring Go. Phase 5 smoke validates harness-only path on fixtures. + +## Decision gate criteria + +| Criterion | Result (2026-06-10 smoke) | +| --- | --- | +| Harness + scripts build orient bundle | Pass (`build-orient-bundle.sh`) | +| Viewer loads bundle without Go | Pass (`harness-orient-smoke.sh`) | +| Importer edge cases in production | Not re-tested; legacy `internal/importer` retained | +| Large JSONL / path safety | Legacy Go still has tests; harness uses jq/bash | + +## Decision + +**Keep Go in maintenance mode** until harness importers cover production edge +cases or a dedicated spec deprecates `cmd/portolan`. + +Allowed: + +- `map`, `import`, `query`, `orient-export-from-map.sh` bridge +- Bugfixes and security fixes per GO-FREEZE-POLICY + +Not allowed without new ADR: + +- New product features in Go +- `contextprep` / answer-contract growth + +## Consequences + +- Primary docs and INSTALL-PROMPT point to `harness/SKILL.md`. +- Contributors add recipes and viewer features first. +- Revisit this ADR after real-target harness runs (non-fixture) or importer parity spec. + +## Alternatives considered + +1. **Deprecate Go immediately** — rejected; importer bridge still useful. +2. **Go as primary** — rejected; conflicts with harness-first product. diff --git a/docs/harness/GO-FREEZE-POLICY.md b/docs/harness/GO-FREEZE-POLICY.md new file mode 100644 index 00000000..6f753dd7 --- /dev/null +++ b/docs/harness/GO-FREEZE-POLICY.md @@ -0,0 +1,43 @@ +# Go CLI Freeze Policy + +Effective: 2026-06-10 (spec 087 harness-first pivot) + +## Policy + +The existing Go CLI (`cmd/portolan`, `internal/*`) is **frozen for new product +features** until the Go decision gate in +[`docs/adr/001-go-cli-fate.md`](../adr/001-go-cli-fate.md) is resolved. + +## Allowed changes + +- Bug fixes with regression tests. +- Security fixes for path handling and untrusted artifact imports. +- Documentation corrections. +- Bridge commands/scripts that export legacy map output to `orient/` bundles. +- Dependency patches required for `go test ./...` on supported Go versions. + +## Not allowed without a new spec and Go gate outcome + +- New commands or flags in `internal/app`. +- Growth of `internal/contextprep` markdown generators (`answer-contract`, etc.). +- New native scanners or language-specific detectors in Go. +- New importer formats unless the gate keeps Go as the normalization layer. + +## Primary product path + +Use [`harness/SKILL.md`](../../harness/SKILL.md): + +1. Run OSS recipes from `harness/recipes/`. +2. Build `orient/` with `scripts/build-orient-bundle.sh`. +3. Open the local viewer from `viewer/`. + +Optional legacy bridge: + +```bash +go run ./cmd/portolan map --root --out +scripts/orient-export-from-map.sh +``` + +## Review + +Revisit when Phase 5 smoke completes and ADR 001 is accepted. diff --git a/docs/harness/tool-profiles/ast-index.md b/docs/harness/tool-profiles/ast-index.md new file mode 100644 index 00000000..4a397a7c --- /dev/null +++ b/docs/harness/tool-profiles/ast-index.md @@ -0,0 +1,21 @@ +# Tool Profile: ast-index (Claude-ast-index-search) + +| Field | Value | +| --- | --- | +| Role | `producer_candidate` (symbol/reference v1.1) | +| User job | Symbol/reference search via local index | +| License | Review upstream before adoption | +| Review date | 2026-06-10 | +| Portolan action | Import bounded JSON when spec 085 is planned; watcher/hooks blocked by default | + +## Risks + +| Risk | Boundary | +| --- | --- | +| SQLite/cache in target | Mutation risk — approval required | +| Watcher/hooks/MCP | Blocked by default | +| Reference completeness | `not_assessed` unless output proves coverage | + +## Approval gate + +Spec 085 implementation gate; see draft spec 085. diff --git a/docs/harness/tool-profiles/codegraph.md b/docs/harness/tool-profiles/codegraph.md new file mode 100644 index 00000000..a638eaae --- /dev/null +++ b/docs/harness/tool-profiles/codegraph.md @@ -0,0 +1,21 @@ +# Tool Profile: CodeGraph + +| Field | Value | +| --- | --- | +| Role | `producer_candidate` (lower fit) | +| User job | Call graph / impact exploration | +| License | Review upstream before adoption | +| Review date | 2026-06-10 | +| Portolan action | Profile only until separate import spec; default workflow writes `.codegraph/` in target | + +## Risks + +| Risk | Boundary | +| --- | --- | +| Target mutation | `.codegraph/` in repo — approval required | +| Watch/MCP/daemon | Blocked by default | +| Evidence semantics | No Portolan evidence states in native output | + +## Approval gate + +Explicit spec required before import planning (per FR-008 spec 084). diff --git a/docs/harness/tool-profiles/jscpd.md b/docs/harness/tool-profiles/jscpd.md new file mode 100644 index 00000000..c9bc25dc --- /dev/null +++ b/docs/harness/tool-profiles/jscpd.md @@ -0,0 +1,30 @@ +# Tool Profile: jscpd + +| Field | Value | +| --- | --- | +| Role | `producer_candidate` (required v1) | +| User job | Code duplication / near-clone clusters | +| License | MIT | +| Review date | 2026-06-10 | +| Portolan action | Import JSON output into `orient/hotspots.jsonl` (`kind: duplication`) | + +## Output surface + +- JSON reporter (`--reporters json`) +- Prefer repository-sharded runs on multi-repo landscapes (see spec 079) + +## Risks + +| Risk | Boundary | +| --- | --- | +| OOM on large roots | Shard by repository; failed shard = `not_assessed`, not metric | +| Cross-repo clones | `not_assessed` unless producer output explicitly covers both repos | +| Target mutation | Read-only scan; no writes to target | + +## Approval gate + +Operator approves local Node/jscpd install and scan scope before execution. + +## Recipe + +[`harness/recipes/duplication-jscpd.md`](../../../harness/recipes/duplication-jscpd.md) diff --git a/docs/harness/tool-profiles/semgrep.md b/docs/harness/tool-profiles/semgrep.md new file mode 100644 index 00000000..bc728755 --- /dev/null +++ b/docs/harness/tool-profiles/semgrep.md @@ -0,0 +1,29 @@ +# Tool Profile: Semgrep + +| Field | Value | +| --- | --- | +| Role | `producer_candidate` (required v1) | +| User job | Static smells, security/config patterns ("where it hurts") | +| License | LGPL-2.1 (tool); use local rule packs only by default | +| Review date | 2026-06-10 | +| Portolan action | Import JSON findings into hotspots (`kind: static-finding`) | + +## Output surface + +- `semgrep scan --json --config ` + +## Risks + +| Risk | Boundary | +| --- | --- | +| Registry rule fetch | Blocked by default; local config only (spec 063 pattern) | +| Telemetry | Disable per Semgrep docs when scanning private code | +| False positives | Preserve `metadata-visible`; agent must cite rule id | + +## Approval gate + +Operator approves Semgrep install and rule pack path. + +## Recipe + +[`harness/recipes/static-semgrep-local.md`](../../../harness/recipes/static-semgrep-local.md) diff --git a/docs/harness/tool-profiles/syft-cyclonedx.md b/docs/harness/tool-profiles/syft-cyclonedx.md new file mode 100644 index 00000000..52414c4c --- /dev/null +++ b/docs/harness/tool-profiles/syft-cyclonedx.md @@ -0,0 +1,29 @@ +# Tool Profile: Syft / CycloneDX + +| Field | Value | +| --- | --- | +| Role | `producer_candidate` (required v1) | +| User job | Dependency hubs, component identity, SBOM-level duplication hints | +| License | Apache-2.0 (Syft); CycloneDX spec is open standard | +| Review date | 2026-06-10 | +| Portolan action | Import CycloneDX JSON; emit `dep-hub` hotspots from high-degree components | + +## Output surface + +- `syft scan -o cyclonedx-json` + +## Risks + +| Risk | Boundary | +| --- | --- | +| SBOM fan-out ≠ service topology | Hotspots are component hubs, not runtime coupling | +| Large multi-repo scan | Repository-sharded Syft (spec 082) | +| Network | Syft may fetch metadata; document operator approval | + +## Approval gate + +Operator approves Syft install and scan roots. + +## Recipe + +[`harness/recipes/deps-syft-cyclonedx.md`](../../../harness/recipes/deps-syft-cyclonedx.md) diff --git a/docs/harness/tool-profiles/understand-anything.md b/docs/harness/tool-profiles/understand-anything.md new file mode 100644 index 00000000..10728257 --- /dev/null +++ b/docs/harness/tool-profiles/understand-anything.md @@ -0,0 +1,30 @@ +# Tool Profile: Understand-Anything + +| Field | Value | +| --- | --- | +| Role | `ux_pattern_source` | +| User job | Interactive map UX (explore, search, tours) | +| License | MIT | +| Review date | 2026-06-10 | +| Portolan action | Fork/borrow viewer patterns in `viewer/`; **reject** LLM graph as evidence | + +## Output surface + +- Upstream: interactive graph + Q&A (not Portolan evidence) +- Portolan: `orient/` bundle consumed by `viewer/` + +## Risks + +| Risk | Boundary | +| --- | --- | +| LLM-authored nodes | UX-only; never `source-visible` | +| Network / model calls | Stripped in Portolan viewer path | +| Confusion with truth | Viewer badges show `producer_ref` | + +## Approval gate + +Full UA fork optional; MVP viewer ships in-repo without upstream submodule. + +## Spike + +[`docs/research/2026-06-10-understand-anything-fork-spike.md`](../../research/2026-06-10-understand-anything-fork-spike.md) diff --git a/docs/harness/tool-profiles/universal-ctags.md b/docs/harness/tool-profiles/universal-ctags.md new file mode 100644 index 00000000..25eb2428 --- /dev/null +++ b/docs/harness/tool-profiles/universal-ctags.md @@ -0,0 +1,29 @@ +# Tool Profile: Universal Ctags + +| Field | Value | +| --- | --- | +| Role | `producer_candidate` (v1.1) | +| User job | Symbol definitions for navigation (not full call graph) | +| License | GPL-2.0 (ctags binary) | +| Review date | 2026-06-10 | +| Portolan action | Import JSON symbol output; optional `graph-slice.json` nodes | + +## Output surface + +- `ctags --output-format=json --fields=+nKz -R` + +## Risks + +| Risk | Boundary | +| --- | --- | +| Definition-only | References/call graph remain `not_assessed` | +| Scale | Bounded target list; budget in recipe | +| GPL | Document distribution boundary if bundling binary | + +## Approval gate + +Operator approves ctags install and file scope. + +## Recipe + +[`harness/recipes/symbols-ctags.md`](../../../harness/recipes/symbols-ctags.md) diff --git a/docs/research/2026-06-10-understand-anything-fork-spike.md b/docs/research/2026-06-10-understand-anything-fork-spike.md new file mode 100644 index 00000000..7e49322f --- /dev/null +++ b/docs/research/2026-06-10-understand-anything-fork-spike.md @@ -0,0 +1,68 @@ +# Understand-Anything Fork Spike + +Date: 2026-06-10 + +## Question + +Can Portolan adopt Understand-Anything (UA) UX via fork (not rewrite) while keeping +evidence-backed hotspots as the only truth source? + +## Upstream snapshot + +| Field | Value | +| --- | --- | +| Repository | https://github.com/Egonex-AI/Understand-Anything | +| License | MIT (fork/redistribute allowed with notice) | +| Positioning | Interactive knowledge graph; explore, search, ask | +| Portolan role | `ux_pattern_source` — viewer fork, not evidence producer | + +## Architecture (high level) + +UA builds an interactive graph for exploration. Much of the product value is the +**viewer UX** (tours, search, Q&A). Graph construction may involve LLM-assisted +summarization — **unacceptable as Portolan `source-visible` evidence**. + +## Portolan adapter strategy + +1. **Do not** ingest UA LLM graph output as evidence. +2. **Do** implement `viewer/` as a UA-inspired local viewer that loads only + `orient/manifest.json` + `orient/hotspots.jsonl` + optional `graph-slice.json`. +3. **Future option**: submodule or vendor UA frontend after stripping LLM graph + generation; replace data loader with `loadPortolanBundle()`. + +## Repo layout decision + +**Recommendation: `portolan/viewer/` subdir** (single repo for harness + viewer). + +Rationale: + +- Harness skill references one checkout path. +- Orient bundle contract stays co-located with `harness/contracts/`. +- Sibling repo deferred until viewer needs independent release cadence. + +## Effort estimate + +| Approach | Effort | Risk | +| --- | --- | --- | +| MVP viewer in `viewer/` (current slice) | 3–5 days | Low; may lack UA polish | +| Full UA fork + strip LLM pipeline | 2–3 weeks | Medium; upstream churn | +| Embed UA as git submodule | 1 week integration + ongoing sync | Medium | + +**Phase 4 ships MVP viewer**; full UA fork remains a follow-up if MVP UX is insufficient. + +## Strip list (full UA fork) + +- LLM graph generation and "recover defaults" paths. +- Network calls for model providers. +- Any node promoted to architecture truth without `producer_ref`. + +## Keep / borrow + +- Graph tour and search entrypoint patterns (spec 086). +- Layered views (duplication, static findings, deps). +- Click-to-source navigation. + +## Verification + +MVP done when `viewer/` opens a fixture `orient/` bundle and shows ranked hotspots +with evidence badges without network access. diff --git a/docs/specs/084-external-tool-evaluation-profiles/plan.md b/docs/specs/084-external-tool-evaluation-profiles/plan.md new file mode 100644 index 00000000..c653f09a --- /dev/null +++ b/docs/specs/084-external-tool-evaluation-profiles/plan.md @@ -0,0 +1,31 @@ +# Implementation Plan: External Tool Evaluation Profiles + +**Branch**: `codex/084-external-tool-evaluation-profiles` | **Date**: 2026-06-10 + +## Summary + +Publish dated evaluation profiles and harness recipes for the first-wave OSS +stack. Profiles live under `docs/harness/tool-profiles/`; recipes under +`harness/recipes/`. + +## Deliverables + +| Artifact | Path | +| --- | --- | +| jscpd profile | `docs/harness/tool-profiles/jscpd.md` | +| Semgrep profile | `docs/harness/tool-profiles/semgrep.md` | +| Syft profile | `docs/harness/tool-profiles/syft-cyclonedx.md` | +| ctags profile | `docs/harness/tool-profiles/universal-ctags.md` | +| UA profile | `docs/harness/tool-profiles/understand-anything.md` | +| CodeGraph profile | `docs/harness/tool-profiles/codegraph.md` | +| ast-index profile | `docs/harness/tool-profiles/ast-index.md` | +| UA fork spike | `docs/research/2026-06-10-understand-anything-fork-spike.md` | +| Recipes | `harness/recipes/*.md` | + +## Verification + +```bash +test -f docs/harness/tool-profiles/jscpd.md +test -f harness/recipes/duplication-jscpd.md +git diff --check +``` diff --git a/docs/specs/084-external-tool-evaluation-profiles/spec.md b/docs/specs/084-external-tool-evaluation-profiles/spec.md index 563ce503..bdc7c103 100644 --- a/docs/specs/084-external-tool-evaluation-profiles/spec.md +++ b/docs/specs/084-external-tool-evaluation-profiles/spec.md @@ -4,8 +4,8 @@ **Created**: 2026-06-04 -**Status**: Draft; backlog-only. Requires `plan.md` and `tasks.md` before -implementation. +**Status**: Implemented (P7-084 harness pivot). Profiles in +`docs/harness/tool-profiles/`; recipes in `harness/recipes/`. **Input**: User asked to turn the external review of `colbymchenry/codegraph`, `Lum1104/Understand-Anything`, and `defendend/Claude-ast-index-search` into diff --git a/docs/specs/084-external-tool-evaluation-profiles/tasks.md b/docs/specs/084-external-tool-evaluation-profiles/tasks.md new file mode 100644 index 00000000..7b946a5f --- /dev/null +++ b/docs/specs/084-external-tool-evaluation-profiles/tasks.md @@ -0,0 +1,6 @@ +# Tasks: External Tool Evaluation Profiles + +- [x] T001 Add plan.md and tasks.md for spec 084. +- [x] T002 Publish tool profiles under docs/harness/tool-profiles/. +- [x] T003 Publish UA fork spike research doc. +- [x] T004 Publish first-wave harness recipes (jscpd, Semgrep, Syft, ctags). diff --git a/docs/specs/086-evidence-navigation-ux-patterns/plan.md b/docs/specs/086-evidence-navigation-ux-patterns/plan.md new file mode 100644 index 00000000..2e6e2ff0 --- /dev/null +++ b/docs/specs/086-evidence-navigation-ux-patterns/plan.md @@ -0,0 +1,19 @@ +# Implementation Plan: Evidence Navigation UX Patterns + +**Date**: 2026-06-10 | **Status**: Implemented via `viewer/` MVP + +## Summary + +Ship UA-inspired navigation as `viewer/` loading orient bundle evidence only. +Constitution 1.1.0 allows local static viewer session. + +## Deliverables + +- `viewer/` — tour, graph nodes, gaps panel, evidence badges +- `scripts/serve.js` — read-only bundle proxy on 127.0.0.1 + +## Verification + +```bash +scripts/harness-orient-smoke.sh +``` diff --git a/docs/specs/086-evidence-navigation-ux-patterns/spec.md b/docs/specs/086-evidence-navigation-ux-patterns/spec.md index da6d950a..c458fc23 100644 --- a/docs/specs/086-evidence-navigation-ux-patterns/spec.md +++ b/docs/specs/086-evidence-navigation-ux-patterns/spec.md @@ -4,8 +4,8 @@ **Created**: 2026-06-04 -**Status**: Draft; backlog-only. Requires `plan.md` and `tasks.md` before -implementation. +**Status**: MVP implemented in `viewer/` (2026-06-10 harness pivot). Full UA fork +deferred per `docs/research/2026-06-10-understand-anything-fork-spike.md`. **Input**: The external review found that Understand-Anything, CodeGraph, and ast-index contain useful navigation ideas, but Portolan must not adopt @@ -109,9 +109,9 @@ records. - **FR-004**: The feature MUST borrow UX patterns only as presentation patterns, including guided tours, relationship slices, impact-style entrypoints, search/query suggestions, and reader modes. -- **FR-005**: The feature MUST NOT require LLM calls, live dashboards, browser - sessions, daemons, network access, target mutation, or global agent - configuration by default. +- **FR-005**: The feature MUST NOT require LLM calls, network access, target + mutation, or global agent configuration by default. A local static viewer on + `127.0.0.1` for the duration of `npm run serve` is allowed (constitution 1.1.0). - **FR-006**: Understand-Anything-inspired graph tours MUST be constrained to evidence-backed Portolan records; LLM-authored nodes, summaries, or recovered defaults MUST NOT be accepted as Portolan facts. diff --git a/docs/specs/086-evidence-navigation-ux-patterns/tasks.md b/docs/specs/086-evidence-navigation-ux-patterns/tasks.md new file mode 100644 index 00000000..7ce3df50 --- /dev/null +++ b/docs/specs/086-evidence-navigation-ux-patterns/tasks.md @@ -0,0 +1,6 @@ +# Tasks: Evidence Navigation UX Patterns + +- [x] T001 Revise FR-005 for local static viewer (constitution 1.1.0). +- [x] T002 Implement viewer/ MVP with Portolan bundle adapter. +- [x] T003 Wire harness SKILL to viewer serve command. +- [x] T004 harness-orient-smoke.sh verification. diff --git a/docs/specs/087-harness-first-product/plan.md b/docs/specs/087-harness-first-product/plan.md new file mode 100644 index 00000000..2c3b84f7 --- /dev/null +++ b/docs/specs/087-harness-first-product/plan.md @@ -0,0 +1,37 @@ +# Implementation Plan: Harness-First Product + +**Branch**: `codex/087-harness-first-product` | **Date**: 2026-06-10 + +## Summary + +Reposition Portolan around harness artifacts, OSS recipes, orient bundle contract, +and a local evidence-backed viewer. Publish governance (constitution amendment, +Go freeze policy) before expanding harness and viewer slices. + +## Technical Context + +**Primary delivery**: `harness/` skills, recipes, guardrails, contracts; `viewer/` +local orient UI; `docs/adr/` decision records. + +**Legacy**: Go CLI frozen; optional `scripts/orient-export-from-map.sh` bridge. + +**Constraints**: Local-first, read-only default; viewer session may use local static +serve only. + +## Constitution Check + +| Principle | Status | Notes | +| --- | --- | --- | +| Local-first | Pass | Recipes and viewer are local. | +| Evidence honesty | Pass | Hotspots require producer refs. | +| Complement OSS | Pass | jscpd, Semgrep, Syft are producers. | +| SpecKit | Pass | This spec anchors the pivot. | + +## Verification + +```bash +test -f harness/SKILL.md +test -f docs/harness/GO-FREEZE-POLICY.md +jq empty harness/contracts/orient-bundle.schema.json +git diff --check +``` diff --git a/docs/specs/087-harness-first-product/spec.md b/docs/specs/087-harness-first-product/spec.md new file mode 100644 index 00000000..3f68d13c --- /dev/null +++ b/docs/specs/087-harness-first-product/spec.md @@ -0,0 +1,83 @@ +# Feature Specification: Harness-First Product + +**Feature Branch**: `codex/087-harness-first-product` + +**Created**: 2026-06-10 + +**Status**: Active implementation slice for the Portolan harness pivot. + +**Input**: Product pivot — Portolan is a harness supplement (rules, recipes, +guardrails, OSS assembly) with an orient map viewer, not primarily a Go module. +B2B evidence guardrails are a cherry on top, not the hero value. + +## User Scenarios & Testing + +### User Story 1 - Find Pain In The Codebase (Priority: P1) + +An engineer or agent asks where duplication, static smells, dependency hubs, or +config risks are visible locally and receives a ranked hotspot list plus an +interactive orient map. + +**Why this priority**: This is the primary user job. Evidence discipline supports +the answer; it does not replace the answer. + +**Independent Test**: A fixture target with jscpd and Semgrep outputs produces +an `orient/` bundle and the viewer shows at least one duplication and one +static-finding hotspot with file paths. + +**Acceptance Scenarios**: + +1. **Given** local producer outputs exist, **When** the harness workflow runs, + **Then** `orient/hotspots.jsonl` lists ranked pain points with `producer_ref`. +2. **Given** jscpd output is missing, **When** the orient bundle is built, + **Then** duplication remains `not_assessed` and the viewer shows a gap badge, + not synthetic clone clusters. + +--- + +### User Story 2 - Harness-Independent Agent Workflow (Priority: P1) + +An agent in Cursor, OpenCode, Codex, or Claude can follow one portable skill to +run OSS recipes, build the orient bundle, and open the local viewer without +bootstrapping the Go CLI. + +**Independent Test**: `harness/SKILL.md` completes on a fixture repo using only +documented recipes and `scripts/build-orient-bundle.sh`. + +--- + +### User Story 3 - Guardrails As Secondary Value (Priority: P2) + +Every hotspot and viewer node cites `evidence_state` and `producer_ref`. Unknown +surfaces stay visible but do not block the primary navigation experience. + +**Independent Test**: `harness/guardrails/citation-rules.md` fits on one screen; +viewer badges match evidence states. + +## Requirements + +- **FR-001**: The primary product path MUST be harness install → OSS recipes → + `orient/` bundle → local viewer. +- **FR-002**: The hero product claim MUST be navigation to code pain (duplication, + static findings, deps/config hotspots), not evidence discipline alone. +- **FR-003**: Portable harness artifacts MUST work across Cursor, OpenCode, and + Codex/Claude without IDE-specific truth sources. +- **FR-004**: B2B guardrails (citations, unknowns, claim boundaries) MUST be + secondary surfaces in `harness/guardrails/` and viewer badges. +- **FR-005**: The Go CLI MUST remain frozen for new features until the Go decision + gate in `docs/adr/001-go-cli-fate.md` resolves its role. +- **FR-006**: Viewer nodes MUST come from imported producer evidence or Portolan + normalization only; LLM-authored graphs MUST NOT be `source-visible`. + +## Success Criteria + +- **SC-001**: README and product claims list harness-first quick start before + optional Go legacy path. +- **SC-002**: Constitution v1.1.0 records harness-first delivery and local viewer + boundary. +- **SC-003**: Go freeze policy is published and referenced from `AGENTS.md`. + +## Assumptions + +- Specs 084–088 implement the harness pivot slices. +- Legacy `context prepare` and `map` remain optional bridges until the Go gate. diff --git a/docs/specs/087-harness-first-product/tasks.md b/docs/specs/087-harness-first-product/tasks.md new file mode 100644 index 00000000..d226a78d --- /dev/null +++ b/docs/specs/087-harness-first-product/tasks.md @@ -0,0 +1,12 @@ +# Tasks: Harness-First Product + +## Phase 0 — Governance + +- [x] T001 Create spec 087 harness-first product artifacts. +- [x] T002 Amend constitution to v1.1.0 harness-first delivery. +- [x] T003 Publish Go freeze policy and link from AGENTS.md. +- [x] T004 Update product claims and README harness-first route. + +## Dependencies + +- Blocks harness recipes (084), orient bundle (088), viewer (086 revised). diff --git a/docs/specs/088-orient-bundle-contract/plan.md b/docs/specs/088-orient-bundle-contract/plan.md new file mode 100644 index 00000000..098056b8 --- /dev/null +++ b/docs/specs/088-orient-bundle-contract/plan.md @@ -0,0 +1,15 @@ +# Implementation Plan: Orient Bundle Contract + +## Deliverables + +- `harness/contracts/orient-bundle.schema.json` +- `scripts/build-orient-bundle.sh` +- `scripts/orient-export-from-map.sh` +- `internal/testfixtures/orient-bundle/` fixture + +## Verification + +```bash +scripts/build-orient-bundle.sh internal/testfixtures/orient-bundle/target internal/testfixtures/orient-bundle/orient +jq empty harness/contracts/orient-bundle.schema.json +``` diff --git a/docs/specs/088-orient-bundle-contract/spec.md b/docs/specs/088-orient-bundle-contract/spec.md new file mode 100644 index 00000000..940369ef --- /dev/null +++ b/docs/specs/088-orient-bundle-contract/spec.md @@ -0,0 +1,47 @@ +# Feature Specification: Orient Bundle Contract + +**Feature Branch**: `codex/088-orient-bundle-contract` + +**Created**: 2026-06-10 + +**Status**: Active implementation. + +## User Scenarios + +### User Story 1 - Single Orient Directory (Priority: P1) + +Harness recipes and `scripts/build-orient-bundle.sh` produce one `orient/` +directory that both the viewer and agents consume. + +**Independent Test**: Fixture producers → bundle validates against +`harness/contracts/orient-bundle.schema.json`. + +## Bundle layout + +```text +orient/ + manifest.json + hotspots.jsonl + repos.json + gaps.jsonl + producers/ + graph-slice.json # optional +``` + +## Hotspot record + +Required fields: `id`, `kind`, `severity`, `summary`, `paths`, `evidence_state`, +`producer`, `producer_ref`, `rank`. + +Kinds: `duplication`, `static-finding`, `config`, `dep-hub`, `debt-candidate`. + +## Requirements + +- **FR-001**: No LLM summary as evidence; `producer_ref` required for observed hotspots. +- **FR-002**: `gaps.jsonl` budget: top 20 by default. +- **FR-003**: `scripts/orient-export-from-map.sh` maps legacy `portolan map` output to orient layout. + +## Success Criteria + +- **SC-001**: `jq empty harness/contracts/orient-bundle.schema.json` +- **SC-002**: Smoke fixture bundle loads in viewer. diff --git a/docs/specs/088-orient-bundle-contract/tasks.md b/docs/specs/088-orient-bundle-contract/tasks.md new file mode 100644 index 00000000..b3cb7b0c --- /dev/null +++ b/docs/specs/088-orient-bundle-contract/tasks.md @@ -0,0 +1,6 @@ +# Tasks: Orient Bundle Contract + +- [x] T001 Publish orient bundle schema. +- [x] T002 Implement build-orient-bundle.sh. +- [x] T003 Implement orient-export-from-map.sh bridge. +- [x] T004 Add fixture bundle for viewer smoke. diff --git a/harness/SKILL.md b/harness/SKILL.md new file mode 100644 index 00000000..4729f7c8 --- /dev/null +++ b/harness/SKILL.md @@ -0,0 +1,98 @@ +# Portolan Orient Harness + +Use this skill when the user wants to find code pain (duplication, static smells, +dependency hubs) and navigate a local landscape with an orient map. + +Portolan is a **harness supplement**: recipes + guardrails + orient bundle + local +viewer. The legacy Go CLI is optional (see `docs/harness/GO-FREEZE-POLICY.md`). + +## Inputs + +- `TARGET_PATH` — absolute path to repo or landscape root (read-only). +- `ORIENT_PATH` — absolute empty output directory for the orient bundle. +- `PORTOLAN_PATH` — absolute path to this Portolan checkout. + +## Workflow (recommended: orient wizard) + +One command checks tools, runs producers (with consent-gated install), builds the +bundle, prints a summary, and optionally opens the local viewer: + +```bash +"$PORTOLAN_PATH/scripts/orient-wizard.sh" "$TARGET_PATH" "$ORIENT_PATH" --yes +``` + +Useful flags: + +| Flag | Purpose | +| --- | --- | +| `--no-viewer` | Build bundle only | +| `--skip-install` | Never install missing tools (gaps only) | +| `--limit-repos N` | Cap multi-repo sharding | +| `--producers jscpd,semgrep,syft` | Subset of producers | +| `--shard-timeout SEC` | Per-repo producer timeout (default 600) | +| `--jscpd-memory-mb N` | Node heap cap per jscpd shard (default 2048) | +| `--hotspot-budget N` | Max hotspots in bundle; kind quotas apply when truncated | + +Shard failures are recorded in `producers/_gaps.jsonl` and merged into `gaps.jsonl`. +Full hotspot list (pre-budget) is written to `hotspots-full.jsonl` for agents. + +### Manual fallback (recipes) + +When you need fine-grained control, run individual recipes from `harness/recipes/`: + +| User question | Recipe | +| --- | --- | +| Where is duplication? | `duplication-jscpd.md` | +| Where are smells / static issues? | `static-semgrep-local.md` | +| What depends on what? | `deps-syft-cyclonedx.md` | +| Symbol navigation (optional) | `symbols-ctags.md` | + +Write outputs under `$ORIENT_PATH/producers/`, then: + +```bash +"$PORTOLAN_PATH/scripts/build-orient-bundle.sh" "$TARGET_PATH" "$ORIENT_PATH" +``` + +### Open orient map (human) + +```bash +cd "$PORTOLAN_PATH/viewer" && npm install && npm run serve -- --bundle "$ORIENT_PATH" +``` + +Viewer features (spec 090): search, kind/severity/repo filters, directory heat map, +truncation/gaps banners, and read-only source preview via `/source` (path-guarded). +Demo script: [`docs/demo-runbook.md`](../docs/demo-runbook.md). + +### Agent navigation + +Read in order: + +1. `$ORIENT_PATH/manifest.json` +2. `$ORIENT_PATH/hotspots.jsonl` (ranked pain points) +3. `$ORIENT_PATH/gaps.jsonl` (missing evidence — do not invent) + +Cite `hotspot.id` and `producer_ref` for every material claim. + +Guardrails: `harness/guardrails/`. + +## Legacy bridge (optional) + +```bash +go run "$PORTOLAN_PATH/cmd/portolan" map --root "$TARGET_PATH" --out "$ORIENT_PATH/map" --force +"$PORTOLAN_PATH/scripts/orient-export-from-map.sh" "$ORIENT_PATH/map" "$ORIENT_PATH" +``` + +## User question routing + +| Question | Start with | If missing | +| --- | --- | --- | +| Duplication? | hotspots `kind=duplication` | Run jscpd recipe; say `not_assessed` | +| Tech debt / smells? | hotspots `kind=static-finding` | Run Semgrep recipe | +| Dependencies? | hotspots `kind=dep-hub` | Run Syft recipe | +| Where to start? | lowest `rank` in hotspots.jsonl | gaps.jsonl for next recipe | + +## Do not + +- Invent Portolan subcommands not listed here or in legacy Go `--help`. +- Promote `not_assessed` gaps to observed facts. +- Use LLM-generated graphs as evidence (viewer is presentation only). diff --git a/harness/codex-claude/INSTALL-PROMPT.md b/harness/codex-claude/INSTALL-PROMPT.md new file mode 100644 index 00000000..c3309baa --- /dev/null +++ b/harness/codex-claude/INSTALL-PROMPT.md @@ -0,0 +1,22 @@ +# Codex / Claude Install Prompt (Harness-First) + +Portable block — same skill as Cursor and OpenCode. + +```text +PORTOLAN_PATH= +TARGET_PATH= +ORIENT_PATH= +``` + +```text +Execute Portolan orient harness now (no confirmation unless paths missing). + +1. Read PORTOLAN_PATH/harness/SKILL.md +2. PORTOLAN_PATH/scripts/orient-wizard.sh TARGET_PATH ORIENT_PATH --no-viewer --yes +3. Summarize hotspots.jsonl (top 5 by rank) and gaps.jsonl +4. Apply PORTOLAN_PATH/harness/guardrails/citation-rules.md + +Optional viewer for human: cd PORTOLAN_PATH/viewer && npm install && npm run serve -- --bundle ORIENT_PATH + +Legacy Go path only if asked: docs/harness/GO-FREEZE-POLICY.md +``` diff --git a/harness/contracts/orient-bundle.schema.json b/harness/contracts/orient-bundle.schema.json new file mode 100644 index 00000000..4c6ef65c --- /dev/null +++ b/harness/contracts/orient-bundle.schema.json @@ -0,0 +1,89 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://portolan.local/schemas/orient-bundle.schema.json", + "title": "Portolan Orient Bundle", + "type": "object", + "required": ["manifest", "hotspot_record", "gap_record"], + "properties": { + "manifest": { + "type": "object", + "required": ["schema_version", "target_root", "generated_at"], + "properties": { + "schema_version": { "type": "string" }, + "target_root": { "type": "string" }, + "generated_at": { "type": "string", "format": "date-time" }, + "hotspot_count": { "type": "integer", "minimum": 0 }, + "gap_count": { "type": "integer", "minimum": 0 } + }, + "additionalProperties": true + }, + "hotspot_record": { + "type": "object", + "required": [ + "id", + "kind", + "severity", + "summary", + "paths", + "evidence_state", + "producer", + "producer_ref", + "rank" + ], + "properties": { + "id": { "type": "string" }, + "kind": { + "type": "string", + "enum": [ + "duplication", + "static-finding", + "config", + "dep-hub", + "debt-candidate" + ] + }, + "severity": { + "type": "string", + "enum": ["info", "low", "medium", "high", "critical"] + }, + "summary": { "type": "string" }, + "paths": { + "type": "array", + "items": { "type": "string" }, + "minItems": 1 + }, + "evidence_state": { + "type": "string", + "enum": [ + "source-visible", + "metadata-visible", + "runtime-visible", + "claim-only", + "unknown", + "cannot_verify", + "not_assessed" + ] + }, + "producer": { "type": "string" }, + "producer_ref": { "type": "string" }, + "rank": { "type": "integer", "minimum": 1 } + }, + "additionalProperties": true + }, + "gap_record": { + "type": "object", + "required": ["id", "surface", "status", "summary"], + "properties": { + "id": { "type": "string" }, + "surface": { "type": "string" }, + "status": { + "type": "string", + "enum": ["unknown", "cannot_verify", "not_assessed"] + }, + "summary": { "type": "string" }, + "recipe": { "type": "string" } + }, + "additionalProperties": true + } + } +} diff --git a/harness/cursor/portolan-orient.mdc b/harness/cursor/portolan-orient.mdc new file mode 100644 index 00000000..2f302758 --- /dev/null +++ b/harness/cursor/portolan-orient.mdc @@ -0,0 +1,17 @@ +--- +description: Portolan orient harness — recipes, orient bundle, local map viewer +globs: +alwaysApply: false +--- + +# Portolan Orient + +When mapping a local codebase for duplication, static findings, or dependency hubs: + +1. Read `harness/SKILL.md` in the Portolan checkout. +2. Run approved recipes from `harness/recipes/`. +3. Build bundle: `scripts/build-orient-bundle.sh `. +4. Open viewer: `cd viewer && npm run serve -- --bundle `. +5. Follow `harness/guardrails/citation-rules.md` for answers. + +Do not require `go install` unless the user asks for the legacy Go path. diff --git a/harness/guardrails/citation-rules.md b/harness/guardrails/citation-rules.md new file mode 100644 index 00000000..525d62aa --- /dev/null +++ b/harness/guardrails/citation-rules.md @@ -0,0 +1,17 @@ +# Citation Rules (Guardrail) + +Every material claim about duplication, static issues, dependencies, or debt +must include: + +1. `hotspot.id` from `hotspots.jsonl`, or an explicit gap id from `gaps.jsonl`. +2. `producer_ref` path when the hotspot is observed. +3. `evidence_state` — do not paraphrase as certainty. + +Answer shape: + +- **Finding** — one sentence. +- **Evidence** — hotspot id + producer_ref. +- **Unknowns** — gaps that limit the answer. +- **Next step** — one recipe or file path to inspect. + +Do not cite `answer-contract.md` or legacy context packs on the harness-first path. diff --git a/harness/guardrails/evidence-states.md b/harness/guardrails/evidence-states.md new file mode 100644 index 00000000..52fd600e --- /dev/null +++ b/harness/guardrails/evidence-states.md @@ -0,0 +1,15 @@ +# Evidence States (Guardrail) + +Allowed states on hotspots and graph facts: + +| State | Meaning | +| --- | --- | +| `source-visible` | Visible in source files | +| `metadata-visible` | Tool output or manifest (jscpd, Semgrep, SBOM) | +| `runtime-visible` | Local runtime observation export only | +| `claim-only` | Human/tool claim not verified locally | +| `unknown` | No usable evidence | +| `cannot_verify` | Evidence present but not validated | +| `not_assessed` | Producer not run or surface skipped | + +Never upgrade `claim-only` or `not_assessed` to observed without new local output. diff --git a/harness/guardrails/unknown-block.md b/harness/guardrails/unknown-block.md new file mode 100644 index 00000000..a81d6838 --- /dev/null +++ b/harness/guardrails/unknown-block.md @@ -0,0 +1,15 @@ +# Unknown Block (Guardrail) + +When evidence is missing, say so in one short block: + +```text +Unknowns: +- duplication: not_assessed (no jscpd output) — see harness/recipes/duplication-jscpd.md +- runtime topology: not_assessed (no runtime export) +``` + +Rules: + +- List gaps from `gaps.jsonl` first. +- Do not fill gaps with naming conventions or repository size. +- Missing producer output is not proof of absence of issues. diff --git a/harness/opencode/INSTALL-PROMPT.md b/harness/opencode/INSTALL-PROMPT.md new file mode 100644 index 00000000..8f55f2ee --- /dev/null +++ b/harness/opencode/INSTALL-PROMPT.md @@ -0,0 +1,25 @@ +# OpenCode Install Prompt (Harness-First) + +Replace variables with absolute paths, then send the block to the agent. + +```text +PORTOLAN_PATH= +TARGET_PATH= +ORIENT_PATH= +``` + +```text +Run the Portolan orient harness on TARGET_PATH. Write the orient bundle to +ORIENT_PATH. Follow PORTOLAN_PATH/harness/SKILL.md. + +Rules: +- Use only local paths; no network unless I approve a specific recipe. +- Prefer ORIENT_PATH under PORTOLAN_PATH/.portolan/runs/ if your harness blocks external writes. +- Primary: PORTOLAN_PATH/scripts/orient-wizard.sh TARGET_PATH ORIENT_PATH --no-viewer --yes +- Manual fallback: recipes from PORTOLAN_PATH/harness/recipes/ + build-orient-bundle.sh +- Cite hotspot.id and producer_ref per harness/guardrails/citation-rules.md. +- Do not invent Portolan commands. + +Report: hotspot count, top 5 hotspots by rank, gaps, and viewer command: +cd PORTOLAN_PATH/viewer && npm run serve -- --bundle ORIENT_PATH +``` diff --git a/harness/recipes/deps-syft-cyclonedx.md b/harness/recipes/deps-syft-cyclonedx.md new file mode 100644 index 00000000..b8d17921 --- /dev/null +++ b/harness/recipes/deps-syft-cyclonedx.md @@ -0,0 +1,39 @@ +# Recipe: Syft CycloneDX SBOM + +## Prerequisites + +- `syft` on PATH (operator-approved). + +## Single repository + +```bash +TARGET= +OUT=/producers/syft +mkdir -p "$OUT" +syft scan "dir:$TARGET" -o cyclonedx-json > "$OUT/cyclonedx.json" +``` + +## Multi-repo (sharded) + +```bash +ROOT= +OUT=/producers/syft +mkdir -p "$OUT" +while IFS= read -r repo; do + name=$(basename "$repo") + syft scan "dir:$repo" -o cyclonedx-json > "$OUT/$name-cyclonedx.json" || true +done < <(find "$ROOT" -name .git -type d -prune | sed 's|/.git||') +``` + +## Re-ingest + +```bash +scripts/build-orient-bundle.sh "$TARGET" "$ORIENT_DIR" +``` + +## Failure modes + +| Failure | Result | +| --- | --- | +| Syft missing | Dep hubs `not_assessed` | +| Huge SBOM | Use graph-slice budget in viewer; do not claim service topology | diff --git a/harness/recipes/duplication-jscpd.md b/harness/recipes/duplication-jscpd.md new file mode 100644 index 00000000..89d993d1 --- /dev/null +++ b/harness/recipes/duplication-jscpd.md @@ -0,0 +1,53 @@ +# Recipe: jscpd Duplication Scan + +## Prerequisites + +- Node.js and `npx jscpd` or global `jscpd` install (operator-approved). +- Target path is read-only for Portolan; jscpd must not modify source. + +## Single repository + +```bash +TARGET= +OUT=/producers/jscpd +mkdir -p "$OUT" +jscpd "$TARGET" \ + --reporters json \ + --output "$OUT" \ + --min-lines 5 \ + --min-tokens 50 \ + --threshold 999999 \ + --ignore "**/node_modules/**,**/.git/**,**/vendor/**" +``` + +## Multi-repo (sharded) + +Discover git repos under the landscape root, then run one jscpd command per repo: + +```bash +ROOT= +OUT=/producers/jscpd +mkdir -p "$OUT" +while IFS= read -r repo; do + name=$(basename "$repo") + jscpd "$repo" --reporters json --output "$OUT/$name" \ + --min-lines 5 --min-tokens 50 --noSymlinks true || true +done < <(find "$ROOT" -name .git -type d -prune | sed 's|/.git||') +``` + +Failed shards do not produce duplication metrics; mark duplication `not_assessed` +for that repo in the orient bundle. + +## Re-ingest + +```bash +scripts/build-orient-bundle.sh "$TARGET" "$ORIENT_DIR" +``` + +## Failure modes + +| Failure | Result | +| --- | --- | +| OOM / timeout | Shard smaller; gap record, not invented clones | +| No JSON output | `not_assessed` duplication | +| Missing jscpd | Skip; viewer shows duplication gap badge | diff --git a/harness/recipes/semgrep-rules/portolan-local.yaml b/harness/recipes/semgrep-rules/portolan-local.yaml new file mode 100644 index 00000000..1365f412 --- /dev/null +++ b/harness/recipes/semgrep-rules/portolan-local.yaml @@ -0,0 +1,11 @@ +rules: + - id: portolan-hardcoded-secret-pattern + message: Possible hardcoded secret reference (name only; verify locally) + severity: WARNING + languages: [generic] + pattern-regex: '(?i)(api[_-]?key|secret|password|token)\s*=\s*["''][^"'']+["'']' + - id: portolan-todo-fixme + message: TODO/FIXME marker — debt candidate + severity: INFO + languages: [generic] + pattern-regex: '(?i)(TODO|FIXME|HACK|XXX):' diff --git a/harness/recipes/static-semgrep-local.md b/harness/recipes/static-semgrep-local.md new file mode 100644 index 00000000..7c00b133 --- /dev/null +++ b/harness/recipes/static-semgrep-local.md @@ -0,0 +1,42 @@ +# Recipe: Semgrep Local Config Scan + +## Prerequisites + +- `semgrep` on PATH (operator-approved install). +- Local rule pack only — no registry fetch by default. + +## Command + +```bash +TARGET= +RULES=/harness/recipes/semgrep-rules +OUT=/producers/semgrep +mkdir -p "$OUT" +semgrep scan "$TARGET" \ + --config "$RULES" \ + --json \ + --output "$OUT/findings.json" \ + --metrics off +``` + +If no local rules exist yet, use a minimal config: + +```bash +semgrep scan "$TARGET" --config p/default --json --output "$OUT/findings.json" --metrics off +``` + +Only after operator explicitly approves network rule sources. + +## Re-ingest + +```bash +scripts/build-orient-bundle.sh "$TARGET" "$ORIENT_DIR" +``` + +## Failure modes + +| Failure | Result | +| --- | --- | +| Semgrep errors | Preserve stderr; static findings `not_assessed` | +| Empty findings | Valid; no static hotspots | +| Registry config without approval | Blocked — do not run | diff --git a/harness/recipes/symbols-ctags.md b/harness/recipes/symbols-ctags.md new file mode 100644 index 00000000..0b388727 --- /dev/null +++ b/harness/recipes/symbols-ctags.md @@ -0,0 +1,31 @@ +# Recipe: Universal Ctags Symbol Index + +## Prerequisites + +- `ctags` (Universal Ctags 6.x) on PATH (operator-approved). + +## Bounded scan + +Limit to selected repos or top-level source dirs to control output size. + +```bash +TARGET= +OUT=/producers/ctags +mkdir -p "$OUT" +ctags --output-format=json --fields=+nKz -R \ + --exclude=.git --exclude=node_modules --exclude=vendor \ + -f "$OUT/tags.json" "$TARGET" +``` + +## Re-ingest + +```bash +scripts/build-orient-bundle.sh "$TARGET" "$ORIENT_DIR" +``` + +## Failure modes + +| Failure | Result | +| --- | --- | +| ctags missing | Symbol layer `not_assessed` | +| Huge JSON | Budget symbols in build script; full ref graph `not_assessed` | diff --git a/internal/testfixtures/orient-bundle/orient/gaps.jsonl b/internal/testfixtures/orient-bundle/orient/gaps.jsonl new file mode 100644 index 00000000..904a002a --- /dev/null +++ b/internal/testfixtures/orient-bundle/orient/gaps.jsonl @@ -0,0 +1 @@ +{"id":"gap-deps","surface":"dependencies","status":"not_assessed","summary":"No Syft/CycloneDX producer output found.","recipe":"harness/recipes/deps-syft-cyclonedx.md"} diff --git a/internal/testfixtures/orient-bundle/orient/graph-slice.json b/internal/testfixtures/orient-bundle/orient/graph-slice.json new file mode 100644 index 00000000..74c110be --- /dev/null +++ b/internal/testfixtures/orient-bundle/orient/graph-slice.json @@ -0,0 +1,31 @@ +{ + "schema_version": "0.1.0", + "nodes": [ + { + "id": "dup-916524ed4981", + "label": "Duplicate block (~42 lines): pkg/a/util.go", + "kind": "duplication", + "paths": [ + "pkg/a/util.go", + "pkg/b/util.go" + ] + }, + { + "id": "semgrep-47db41588c43", + "label": "Semgrep portolan-todo-fixme", + "kind": "static-finding", + "paths": [ + "sample.go" + ] + }, + { + "id": "semgrep-d1e57e314224", + "label": "Semgrep portolan-hardcoded-secret-pattern", + "kind": "static-finding", + "paths": [ + "config.env" + ] + } + ], + "edges": [] +} diff --git a/internal/testfixtures/orient-bundle/orient/hotspots.jsonl b/internal/testfixtures/orient-bundle/orient/hotspots.jsonl new file mode 100644 index 00000000..8b62a8e4 --- /dev/null +++ b/internal/testfixtures/orient-bundle/orient/hotspots.jsonl @@ -0,0 +1,3 @@ +{"id":"dup-916524ed4981","kind":"duplication","severity":"medium","summary":"Duplicate block (~42 lines): pkg/a/util.go","paths":["pkg/a/util.go","pkg/b/util.go"],"evidence_state":"metadata-visible","producer":"jscpd","producer_ref":"internal/testfixtures/orient-bundle/orient/producers/jscpd/jscpd-report.json","rank":1} +{"id":"semgrep-47db41588c43","kind":"static-finding","severity":"low","summary":"Semgrep portolan-todo-fixme","paths":["sample.go"],"evidence_state":"metadata-visible","producer":"semgrep","producer_ref":"internal/testfixtures/orient-bundle/orient/producers/semgrep/findings.json","rank":2} +{"id":"semgrep-d1e57e314224","kind":"static-finding","severity":"medium","summary":"Semgrep portolan-hardcoded-secret-pattern","paths":["config.env"],"evidence_state":"metadata-visible","producer":"semgrep","producer_ref":"internal/testfixtures/orient-bundle/orient/producers/semgrep/findings.json","rank":3} diff --git a/internal/testfixtures/orient-bundle/orient/manifest.json b/internal/testfixtures/orient-bundle/orient/manifest.json new file mode 100644 index 00000000..6f93977d --- /dev/null +++ b/internal/testfixtures/orient-bundle/orient/manifest.json @@ -0,0 +1,7 @@ +{ + "schema_version": "0.1.0", + "target_root": "/home/fall_out_bug/projects/sdp/portolan/internal/testfixtures/orient-bundle/target", + "generated_at": "2026-06-10T13:01:32Z", + "hotspot_count": 3, + "gap_count": 1 +} diff --git a/internal/testfixtures/orient-bundle/orient/producers/jscpd/jscpd-report.json b/internal/testfixtures/orient-bundle/orient/producers/jscpd/jscpd-report.json new file mode 100644 index 00000000..e4420b67 --- /dev/null +++ b/internal/testfixtures/orient-bundle/orient/producers/jscpd/jscpd-report.json @@ -0,0 +1,9 @@ +{ + "duplicates": [ + { + "firstFile": { "name": "pkg/a/util.go" }, + "secondFile": { "name": "pkg/b/util.go" }, + "lines": 42 + } + ] +} diff --git a/internal/testfixtures/orient-bundle/orient/producers/semgrep/findings.json b/internal/testfixtures/orient-bundle/orient/producers/semgrep/findings.json new file mode 100644 index 00000000..15786086 --- /dev/null +++ b/internal/testfixtures/orient-bundle/orient/producers/semgrep/findings.json @@ -0,0 +1,14 @@ +{ + "results": [ + { + "check_id": "portolan-todo-fixme", + "path": "sample.go", + "extra": { "severity": "INFO", "message": "TODO marker" } + }, + { + "check_id": "portolan-hardcoded-secret-pattern", + "path": "config.env", + "extra": { "severity": "WARNING", "message": "Possible secret" } + } + ] +} diff --git a/internal/testfixtures/orient-bundle/orient/repos.json b/internal/testfixtures/orient-bundle/orient/repos.json new file mode 100644 index 00000000..b1f2a0ca --- /dev/null +++ b/internal/testfixtures/orient-bundle/orient/repos.json @@ -0,0 +1,7 @@ +[ + { + "id": "target", + "path": "/home/fall_out_bug/projects/sdp/portolan/internal/testfixtures/orient-bundle/target", + "name": "target" + } +] diff --git a/internal/testfixtures/orient-bundle/producers/jscpd/jscpd-report.json b/internal/testfixtures/orient-bundle/producers/jscpd/jscpd-report.json new file mode 100644 index 00000000..e4420b67 --- /dev/null +++ b/internal/testfixtures/orient-bundle/producers/jscpd/jscpd-report.json @@ -0,0 +1,9 @@ +{ + "duplicates": [ + { + "firstFile": { "name": "pkg/a/util.go" }, + "secondFile": { "name": "pkg/b/util.go" }, + "lines": 42 + } + ] +} diff --git a/internal/testfixtures/orient-bundle/producers/semgrep/findings.json b/internal/testfixtures/orient-bundle/producers/semgrep/findings.json new file mode 100644 index 00000000..15786086 --- /dev/null +++ b/internal/testfixtures/orient-bundle/producers/semgrep/findings.json @@ -0,0 +1,14 @@ +{ + "results": [ + { + "check_id": "portolan-todo-fixme", + "path": "sample.go", + "extra": { "severity": "INFO", "message": "TODO marker" } + }, + { + "check_id": "portolan-hardcoded-secret-pattern", + "path": "config.env", + "extra": { "severity": "WARNING", "message": "Possible secret" } + } + ] +} diff --git a/internal/testfixtures/orient-bundle/target/sample.go b/internal/testfixtures/orient-bundle/target/sample.go new file mode 100644 index 00000000..86653467 --- /dev/null +++ b/internal/testfixtures/orient-bundle/target/sample.go @@ -0,0 +1,4 @@ +package sample + +// TODO: refactor this module +func Run() {} diff --git a/scripts/build-orient-bundle.sh b/scripts/build-orient-bundle.sh new file mode 100755 index 00000000..00bb007e --- /dev/null +++ b/scripts/build-orient-bundle.sh @@ -0,0 +1,247 @@ +#!/usr/bin/env bash +# Build orient/ bundle from target root and optional producer outputs under +# /producers/. See harness/SKILL.md and spec 088/091. +set -euo pipefail + +if [[ $# -lt 2 ]]; then + echo "usage: $0 " >&2 + exit 2 +fi + +TARGET_ROOT=$(cd "$1" && pwd) +ORIENT_DIR=$2 +PRODUCERS_DIR="$ORIENT_DIR/producers" +HOTSPOT_BUDGET="${ORIENT_HOTSPOT_BUDGET:-200}" +mkdir -p "$ORIENT_DIR" "$PRODUCERS_DIR" + +command -v jq >/dev/null 2>&1 || { + echo "jq is required" >&2 + exit 1 +} + +# --- repos.json --- +if [[ -d "$TARGET_ROOT/.git" ]]; then + jq -n --arg path "$TARGET_ROOT" --arg name "$(basename "$TARGET_ROOT")" \ + '[{id: $name, path: $path, name: $name}]' >"$ORIENT_DIR/repos.json" +else + repos='[]' + while IFS= read -r gitdir; do + repo=$(dirname "$gitdir") + name=$(basename "$repo") + repos=$(echo "$repos" | jq --arg id "$name" --arg path "$repo" --arg name "$name" \ + '. + [{id: $id, path: $path, name: $name}]') + done < <(find "$TARGET_ROOT" -name .git -type d 2>/dev/null || true) + if [[ "$(echo "$repos" | jq 'length')" -eq 0 ]]; then + repos=$(jq -n --arg path "$TARGET_ROOT" --arg name "$(basename "$TARGET_ROOT")" \ + '[{id: $name, path: $path, name: $name}]') + fi + echo "$repos" >"$ORIENT_DIR/repos.json" +fi + +hotspots_raw=$(mktemp) +: >"$hotspots_raw" +gaps_raw=$(mktemp) +: >"$gaps_raw" + +append_gap() { + jq -nc \ + --arg id "$1" --arg surface "$2" --arg status "$3" \ + --arg summary "$4" --arg recipe "${5:-}" \ + '{id:$id,surface:$surface,status:$status,summary:$summary} + (if $recipe != "" then {recipe:$recipe} else {} end)' >>"$gaps_raw" +} + +process_jscpd_file() { + local jfile=$1 + jq -r --arg ref "$jfile" ' + .duplicates[]? | + select(.firstFile.name != null and .firstFile.name != "") | + [.firstFile.name, (.secondFile.name // ""), (.lines // 0 | tostring), $ref] | @tsv + ' "$jfile" | while IFS=$'\t' read -r first second lines ref; do + [[ -z "$first" ]] && continue + id="dup-$(printf '%s%s' "$first" "$second" | sha256sum | cut -c1-12)" + jq -nc \ + --arg id "$id" --arg first "$first" --arg second "$second" \ + --argjson lines "${lines:-0}" --arg ref "$ref" \ + '{id:$id,kind:"duplication",severity:"medium",summary:("Duplicate block (~"+($lines|tostring)+" lines): "+$first),paths:([$first]+(if $second != "" then [$second] else [] end)),evidence_state:"metadata-visible",producer:"jscpd",producer_ref:$ref}' >>"$hotspots_raw" + done +} + +process_semgrep_file() { + local sfile=$1 + jq -r --arg ref "$sfile" ' + .results[]? | + [.path, .check_id, (.extra.severity // "WARNING"), $ref] | @tsv + ' "$sfile" | while IFS=$'\t' read -r fpath check sev_raw ref; do + [[ -z "$fpath" ]] && continue + sev=$(echo "$sev_raw" | tr '[:upper:]' '[:lower:]') + case "$sev" in + error|critical) severity="critical" ;; + warning) severity="medium" ;; + info) severity="info" ;; + *) severity="low" ;; + esac + id="semgrep-$(printf '%s%s' "$fpath" "$check" | sha256sum | cut -c1-12)" + jq -nc \ + --arg id "$id" --arg path "$fpath" --arg summary "Semgrep $check" \ + --arg ref "$ref" --arg severity "$severity" \ + '{id:$id,kind:"static-finding",severity:$severity,summary:$summary,paths:[$path],evidence_state:"metadata-visible",producer:"semgrep",producer_ref:$ref}' >>"$hotspots_raw" + done +} + +# --- jscpd (producers/jscpd/**) --- +jscpd_found=0 +while IFS= read -r jfile; do + [[ -f "$jfile" ]] || continue + jq -e '.duplicates' "$jfile" >/dev/null 2>&1 || continue + jscpd_found=1 + process_jscpd_file "$jfile" +done < <(find "$PRODUCERS_DIR/jscpd" -type f -name '*.json' 2>/dev/null; find "$PRODUCERS_DIR" -maxdepth 2 -name 'jscpd-report.json' 2>/dev/null) + +[[ "$jscpd_found" -eq 1 ]] || append_gap "gap-duplication" "duplication" "not_assessed" \ + "No jscpd producer output found." "harness/recipes/duplication-jscpd.md" + +# --- semgrep (producers/semgrep/**) --- +semgrep_found=0 +while IFS= read -r sfile; do + [[ -f "$sfile" ]] || continue + jq -e '.results' "$sfile" >/dev/null 2>&1 || continue + semgrep_found=1 + process_semgrep_file "$sfile" +done < <(find "$PRODUCERS_DIR/semgrep" -type f -name '*.json' 2>/dev/null) + +[[ "$semgrep_found" -eq 1 ]] || append_gap "gap-static" "static-findings" "not_assessed" \ + "No Semgrep producer output found." "harness/recipes/static-semgrep-local.md" + +# --- cyclonedx (producers/syft/**) --- +syft_found=0 +dep_hub_min=8 +while IFS= read -r sbom; do + [[ -f "$sbom" ]] || continue + jq -e '.components' "$sbom" >/dev/null 2>&1 || continue + syft_found=1 + jq -r --arg ref "$sbom" --argjson min "$dep_hub_min" ' + (.dependencies // []) as $deps | + def dep_count($r): + [$deps[] | select(.ref == $r) | .dependsOn[]?] | length; + .components[]? | select(.name != null) | + . as $c | + ($c."bom-ref" // $c.name) as $r | + {name: $c.name, dep_count: dep_count($r)} | + select(.dep_count >= $min) | + [.name, (.dep_count | tostring), $ref] | @tsv + ' "$sbom" | while IFS=$'\t' read -r name dep_count ref; do + id="dep-$(printf '%s' "$name" | sha256sum | cut -c1-12)" + jq -nc \ + --arg id "$id" --arg summary "Dependency hub: $name ($dep_count dependencies)" \ + --arg ref "$ref" \ + '{id:$id,kind:"dep-hub",severity:"low",summary:$summary,paths:[],evidence_state:"metadata-visible",producer:"syft",producer_ref:$ref}' >>"$hotspots_raw" + done +done < <(find "$PRODUCERS_DIR/syft" -type f \( -name 'cyclonedx.json' -o -name '*cyclonedx*.json' \) 2>/dev/null) + +[[ "$syft_found" -eq 1 ]] || append_gap "gap-deps" "dependencies" "not_assessed" \ + "No Syft/CycloneDX producer output found." "harness/recipes/deps-syft-cyclonedx.md" + +# --- merge shard gaps from wizard --- +if [[ -f "$PRODUCERS_DIR/_gaps.jsonl" ]]; then + cat "$PRODUCERS_DIR/_gaps.jsonl" >>"$gaps_raw" +fi + +# Sort all hotspots; write full list; apply kind-quota budget +sorted_all=$(mktemp) +if [[ -s "$hotspots_raw" ]]; then + jq -sc ' + sort_by( + (if .severity == "critical" then 0 + elif .severity == "high" then 1 + elif .severity == "medium" then 2 + elif .severity == "low" then 3 + else 4 end), + .kind, + .summary + ) | .[] + ' "$hotspots_raw" >"$sorted_all" +else + : >"$sorted_all" +fi + +total_before=$(wc -l <"$sorted_all" | tr -d ' ') + +: >"$ORIENT_DIR/hotspots-full.jsonl" +cp "$sorted_all" "$ORIENT_DIR/hotspots-full.jsonl" + +budgeted=$(mktemp) +if [[ "$total_before" -gt "$HOTSPOT_BUDGET" ]]; then + jq -sc --argjson budget "$HOTSPOT_BUDGET" ' + def sev_rank(s): + if s == "critical" then 0 elif s == "high" then 1 elif s == "medium" then 2 + elif s == "low" then 3 else 4 end; + def sort_h: sort_by(sev_rank(.severity), .summary); + def quota(k): + if k == "static-finding" then ($budget * 0.5 | floor) + elif k == "duplication" then ($budget * 0.3 | floor) + elif k == "dep-hub" then ($budget * 0.2 | floor) + else 0 end; + . as $all | + (["static-finding", "duplication", "dep-hub"] | map( + . as $k | ([$all[] | select(.kind == $k)] | sort_h) | .[0:quota($k)] + ) | add) as $selected | + ($selected | map(.id)) as $ids | + ([$all[] | select(.id as $i | ($ids | index($i) | not))] | sort_h) as $rest | + ($selected + $rest[0:($budget - ($selected | length))]) | + sort_by(sev_rank(.severity), .kind, .summary) | .[] + ' "$hotspots_raw" >"$budgeted" || { + echo "warn: kind-quota jq failed; falling back to global head budget" >&2 + head -n "$HOTSPOT_BUDGET" "$sorted_all" >"$budgeted" + } + truncated=1 +else + cp "$sorted_all" "$budgeted" + truncated=0 +fi + +budget_count=$(wc -l <"$budgeted" | tr -d ' ') +if [[ "$budget_count" -gt "$HOTSPOT_BUDGET" ]]; then + head -n "$HOTSPOT_BUDGET" "$budgeted" >"${budgeted}.cut" + mv "${budgeted}.cut" "$budgeted" + truncated=1 +fi + +: >"$ORIENT_DIR/hotspots.jsonl" +rank=0 +while IFS= read -r line; do + [[ -z "$line" ]] && continue + rank=$((rank + 1)) + echo "$line" | jq -c --argjson rank "$rank" '. + {rank: $rank}' >>"$ORIENT_DIR/hotspots.jsonl" +done <"$budgeted" + +hotspot_count=$(wc -l <"$ORIENT_DIR/hotspots.jsonl" | tr -d ' ') + +kind_counts_total=$(jq -s 'group_by(.kind) | map({(.[0].kind): length}) | add // {}' "$ORIENT_DIR/hotspots-full.jsonl" 2>/dev/null || echo '{}') +kind_counts=$(jq -s 'group_by(.kind) | map({(.[0].kind): length}) | add // {}' "$ORIENT_DIR/hotspots.jsonl" 2>/dev/null || echo '{}') + +: >"$ORIENT_DIR/gaps.jsonl" +if [[ -s "$gaps_raw" ]]; then + cat "$gaps_raw" >>"$ORIENT_DIR/gaps.jsonl" +fi +gap_count=$(wc -l <"$ORIENT_DIR/gaps.jsonl" | tr -d ' ') + +jq -n \ + --arg schema_version "0.1.0" \ + --arg target_root "$TARGET_ROOT" \ + --arg generated_at "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" \ + --argjson hotspot_count "$hotspot_count" \ + --argjson gap_count "$gap_count" \ + --argjson hotspot_budget "$HOTSPOT_BUDGET" \ + --argjson hotspots_truncated "$truncated" \ + --argjson hotspots_total "$total_before" \ + --argjson kind_counts "$kind_counts" \ + --argjson kind_counts_total "$kind_counts_total" \ + '{schema_version:$schema_version,target_root:$target_root,generated_at:$generated_at,hotspot_count:$hotspot_count,gap_count:$gap_count,hotspot_budget:$hotspot_budget,hotspots_truncated:$hotspots_truncated,hotspots_total:$hotspots_total,kind_counts:$kind_counts,kind_counts_total:$kind_counts_total}' \ + >"$ORIENT_DIR/manifest.json" + +jq -s '{schema_version:"0.1.0",nodes:[.[]|{id:.id,label:.summary,kind:.kind,paths:(.paths//[])}],edges:[]}' \ + "$ORIENT_DIR/hotspots.jsonl" >"$ORIENT_DIR/graph-slice.json" 2>/dev/null || \ + echo '{"schema_version":"0.1.0","nodes":[],"edges":[]}' >"$ORIENT_DIR/graph-slice.json" + +rm -f "$hotspots_raw" "$gaps_raw" "$sorted_all" "$budgeted" +echo "orient bundle written to $ORIENT_DIR (hotspots=$hotspot_count gaps=$gap_count total_before=$total_before truncated=$truncated)" diff --git a/scripts/harness-orient-smoke.sh b/scripts/harness-orient-smoke.sh new file mode 100755 index 00000000..a16f3663 --- /dev/null +++ b/scripts/harness-orient-smoke.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# End-to-end harness orient smoke (spec 087 / phase 5). No network required. +set -euo pipefail + +ROOT=$(cd "$(dirname "$0")/.." && pwd) +FIXTURE_TARGET="$ROOT/internal/testfixtures/orient-bundle/target" +FIXTURE_ORIENT="$ROOT/internal/testfixtures/orient-bundle/orient-smoke" +VIEWER_PORT="${VIEWER_PORT:-4174}" + +rm -rf "$FIXTURE_ORIENT" +mkdir -p "$FIXTURE_ORIENT/producers" +cp -a "$ROOT/internal/testfixtures/orient-bundle/producers/." "$FIXTURE_ORIENT/producers/" + +"$ROOT/scripts/build-orient-bundle.sh" "$FIXTURE_TARGET" "$FIXTURE_ORIENT" + +test "$(wc -l <"$FIXTURE_ORIENT/hotspots.jsonl" | tr -d ' ')" -ge 1 +test -f "$FIXTURE_ORIENT/manifest.json" + +cd "$ROOT/viewer" +node scripts/build-static.js +node scripts/serve.js --bundle "$FIXTURE_ORIENT" --port "$VIEWER_PORT" & +PID=$! +trap 'kill $PID 2>/dev/null || true' EXIT +sleep 1 + +BASE="http://127.0.0.1:$VIEWER_PORT" +curl -sf "$BASE/" | grep -q 'Portolan Orient' +curl -sf "$BASE/" | grep -q 'id="search-input"' +curl -sf "$BASE/bundle/hotspots.jsonl" | grep -q duplication +curl -sf "$BASE/source?path=sample.go&line=1" | grep -q 'Run' + +FORBIDDEN_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/source?path=../../../etc/passwd&line=1") +test "$FORBIDDEN_CODE" = "403" + +echo "harness-orient-smoke: ok" diff --git a/scripts/orient-export-from-map.sh b/scripts/orient-export-from-map.sh new file mode 100755 index 00000000..e02dbc98 --- /dev/null +++ b/scripts/orient-export-from-map.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +# Bridge legacy portolan map bundle -> orient/ layout. See spec 088. +set -euo pipefail + +if [[ $# -lt 2 ]]; then + echo "usage: $0 " >&2 + exit 2 +fi + +MAP_DIR=$1 +ORIENT_DIR=$2 +mkdir -p "$ORIENT_DIR" + +command -v jq >/dev/null 2>&1 || { echo "jq required" >&2; exit 1; } + +target_root="" +if [[ -f "$MAP_DIR/run.json" ]]; then + target_root=$(jq -r '.root // empty' "$MAP_DIR/run.json") +fi +[[ -z "$target_root" ]] && target_root=$(jq -r '.target_root // "."' "$ORIENT_DIR/manifest.json" 2>/dev/null || echo ".") + +rank=0 +: >"$ORIENT_DIR/hotspots.jsonl" +if [[ -f "$MAP_DIR/findings.jsonl" ]]; then + while IFS= read -r line; do + [[ -z "$line" ]] && continue + kind=$(echo "$line" | jq -r '.kind') + mapped_kind=$kind + case "$kind" in + duplication) mapped_kind="duplication" ;; + configuration) mapped_kind="config" ;; + technical-debt) mapped_kind="debt-candidate" ;; + relationships) mapped_kind="dep-hub" ;; + *) mapped_kind="debt-candidate" ;; + esac + rank=$((rank + 1)) + echo "$line" | jq -c \ + --argjson rank "$rank" \ + --arg mk "$mapped_kind" \ + '{ + id: (.id // ("map-" + ($rank|tostring))), + kind: $mk, + severity: (if .severity == "high" then "high" elif .severity == "medium" then "medium" else "low" end), + summary: (.summary // "finding"), + paths: [], + evidence_state: (.evidence_state // "metadata-visible"), + producer: "portolan-map", + producer_ref: "findings.jsonl", + rank: $rank + }' >>"$ORIENT_DIR/hotspots.jsonl" + done <"$MAP_DIR/findings.jsonl" +fi + +: >"$ORIENT_DIR/gaps.jsonl" +if [[ -f "$MAP_DIR/coverage.json" ]]; then + jq -c '.records[]? | select(.status == "not_assessed" or .status == "unknown")' "$MAP_DIR/coverage.json" 2>/dev/null | head -20 | while read -r rec; do + echo "$rec" | jq -c '{id:("gap-" + (.surface // "unknown")),surface:(.surface // "unknown"),status:(.status // "not_assessed"),summary:(.summary // .surface)}' >>"$ORIENT_DIR/gaps.jsonl" + done +fi + +if [[ -f "$MAP_DIR/summary.json" ]]; then + cp "$MAP_DIR/summary.json" "$ORIENT_DIR/legacy-summary.json" 2>/dev/null || true +fi + +hotspot_count=$(wc -l <"$ORIENT_DIR/hotspots.jsonl" | tr -d ' ') +gap_count=0 +[[ -f "$ORIENT_DIR/gaps.jsonl" ]] && gap_count=$(wc -l <"$ORIENT_DIR/gaps.jsonl" | tr -d ' ') + +jq -n \ + --arg schema_version "0.1.0" \ + --arg target_root "$target_root" \ + --arg generated_at "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" \ + --argjson hotspot_count "$hotspot_count" \ + --argjson gap_count "$gap_count" \ + --arg source "portolan-map-bridge" \ + '{schema_version:$schema_version,target_root:$target_root,generated_at:$generated_at,hotspot_count:$hotspot_count,gap_count:$gap_count,source:$source}' \ + >"$ORIENT_DIR/manifest.json" + +echo '{"schema_version":"0.1.0","repos":[]}' >"$ORIENT_DIR/repos.json" +if [[ -f "$MAP_DIR/run.json" ]]; then + jq -n --arg r "$target_root" '[{id:"root",path:$r,name:( $r | split("/") | last)}]' >"$ORIENT_DIR/repos.json" +fi + +jq -s '{schema_version:"0.1.0",nodes:[.[]|{id:.id,label:.summary,kind:.kind}],edges:[]}' \ + "$ORIENT_DIR/hotspots.jsonl" >"$ORIENT_DIR/graph-slice.json" 2>/dev/null || \ + echo '{"schema_version":"0.1.0","nodes":[],"edges":[]}' >"$ORIENT_DIR/graph-slice.json" + +echo "exported map bundle from $MAP_DIR to $ORIENT_DIR" From 9860da5c2147faefc678c07217c9466b8a70815c Mon Sep 17 00:00:00 2001 From: Andrey Zhukov Date: Wed, 10 Jun 2026 17:53:42 +0300 Subject: [PATCH 02/12] Add orient wizard: one-command tool check, recipes, bundle, viewer Spec 089: consent-gated tool install (y/N or --yes), per-repo sharded jscpd/semgrep/syft, failure log, bundle build, hotspot summary, and optional local viewer. Real-target smoke on portolan and bounded bigtop. Co-authored-by: Cursor --- docs/specs/089-orient-wizard/plan.md | 18 ++ .../reviews/smoke-findings.md | 57 ++++ docs/specs/089-orient-wizard/spec.md | 41 +++ docs/specs/089-orient-wizard/tasks.md | 7 + scripts/orient-wizard.sh | 304 ++++++++++++++++++ 5 files changed, 427 insertions(+) create mode 100644 docs/specs/089-orient-wizard/plan.md create mode 100644 docs/specs/089-orient-wizard/reviews/smoke-findings.md create mode 100644 docs/specs/089-orient-wizard/spec.md create mode 100644 docs/specs/089-orient-wizard/tasks.md create mode 100755 scripts/orient-wizard.sh diff --git a/docs/specs/089-orient-wizard/plan.md b/docs/specs/089-orient-wizard/plan.md new file mode 100644 index 00000000..a8fc80e1 --- /dev/null +++ b/docs/specs/089-orient-wizard/plan.md @@ -0,0 +1,18 @@ +# Implementation Plan: Orient Wizard + +**Date**: 2026-06-10 + +## Deliverables + +- [`scripts/orient-wizard.sh`](../../../scripts/orient-wizard.sh) +- Fixes to [`scripts/build-orient-bundle.sh`](../../../scripts/build-orient-bundle.sh) +- Smoke evidence in `reviews/` + +## Verification + +```bash +bash -n scripts/orient-wizard.sh +scripts/orient-wizard.sh --help +scripts/harness-orient-smoke.sh +scripts/orient-wizard.sh . /tmp/orient-portolan --no-viewer --yes +``` diff --git a/docs/specs/089-orient-wizard/reviews/smoke-findings.md b/docs/specs/089-orient-wizard/reviews/smoke-findings.md new file mode 100644 index 00000000..312536f8 --- /dev/null +++ b/docs/specs/089-orient-wizard/reviews/smoke-findings.md @@ -0,0 +1,57 @@ +# Smoke findings: Orient Wizard (spec 089) + +**Date**: 2026-06-10 + +## Smoke A — portolan repo + +```bash +scripts/orient-wizard.sh . /tmp/orient-portolan --no-viewer --yes +``` + +| Metric | Value | +| --- | --- | +| Runtime | ~18s (jscpd ~7s, semgrep/syft fast on single repo) | +| Hotspots | 128 (0 gaps) | +| Top kinds | duplication (jscpd), static-finding (semgrep) | +| Viewer | `node viewer/scripts/serve.js --bundle /tmp/orient-portolan` — page loads, hotspots.jsonl served | + +### Fixes applied during smoke + +1. **jscpd**: `--noSymlinks true` passed `true` as a path; replaced with `--noSymlinks` + ignore globs. +2. **CLI flags**: options after positionals (`target orient --no-viewer`) were ignored; positional/flag parsing fixed. +3. **jscpd output**: read from `producers/jscpd/**/jscpd-report.json` (wizard layout). + +## Smoke B — bigtop-landscape (bounded) + +```bash +scripts/orient-wizard.sh ~/projects/bigtop-landscape/repos /tmp/orient-bigtop \ + --no-viewer --yes --limit-repos 3 --producers semgrep,syft +``` + +| Metric | Value | +| --- | --- | +| Repos scanned | alluxio, apache-airflow, apache-bigtop-repo | +| Semgrep outputs | 3 shard JSON files (405KB–1.2MB) | +| Syft outputs | 3 cyclonedx JSON (91KB–8.9MB airflow) | +| Bundle | 830 hotspots before budget; **200 after** (`hotspots_truncated=true`) | +| Bundle build time | ~18s after dep-hub jq optimization (was hung >12min) | + +### Fixes applied during smoke + +1. **dep-hub loop**: per-component `jq` on large SBOMs was O(n²); replaced with single-pass jq per SBOM file. +2. **dep-hub threshold**: raised from 5 → 8 transitive deps to reduce noise. +3. **Hotspot budget**: default 200 with `hotspots_total` / `hotspots_truncated` in manifest. + +### Not assessed (by design) + +- Full jscpd on bigtop (OOM risk per spec 079) — omitted via `--producers semgrep,syft`. +- Runtime evidence, call graphs, enterprise parity. + +## Regression + +- `scripts/harness-orient-smoke.sh` — **ok** after bundle ranking/budget changes. + +## Skip-install path + +With `PATH=/usr/bin:/bin` and `--skip-install`, producers missing from PATH log to +`producers/_failures.log` and surfaces remain `not_assessed` in gaps.jsonl. diff --git a/docs/specs/089-orient-wizard/spec.md b/docs/specs/089-orient-wizard/spec.md new file mode 100644 index 00000000..e62ba430 --- /dev/null +++ b/docs/specs/089-orient-wizard/spec.md @@ -0,0 +1,41 @@ +# Feature Specification: Orient Wizard + +**Feature Branch**: `codex/089-orient-wizard` + +**Created**: 2026-06-10 + +**Status**: Implemented (local verification + smoke evidence in reviews/). + +**Input**: One-command orient workflow: check tools, consent install, run recipes, +build bundle, open viewer. Real-target smoke on portolan and bounded bigtop. + +## User Scenarios + +### User Story 1 - One Command Orient (Priority: P1) + +An operator runs `scripts/orient-wizard.sh ` and receives +a ranked hotspot bundle plus optional local viewer without manual recipe steps. + +### User Story 2 - Consent-Gated Tool Install (Priority: P1) + +Missing jscpd, Semgrep, or Syft triggers y/N install prompt. Refusal skips the +producer and leaves `not_assessed` in gaps.jsonl. + +### User Story 3 - Bounded Multi-Repo Stress (Priority: P2) + +`--limit-repos N` shards jscpd/syft/semgrep across discovered git repos for +large landscapes without full-root OOM. + +## Requirements + +- **FR-001**: `orient-wizard.sh` orchestrates tool check, recipes, bundle build, summary. +- **FR-002**: Install only after explicit y/N or `--yes`; `--skip-install` never installs. +- **FR-003**: Shard failures log to `producers/_failures.log` and do not abort the run. +- **FR-004**: Flags: `--yes`, `--skip-install`, `--no-viewer`, `--port`, `--limit-repos`, `--producers`. +- **FR-005**: Real-target smoke evidence recorded under `reviews/`. + +## Success Criteria + +- **SC-001**: `orient-wizard.sh . /tmp/orient-portolan --no-viewer` produces hotspots on portolan repo. +- **SC-002**: Bounded bigtop smoke completes without crash. +- **SC-003**: `harness-orient-smoke.sh` regression still passes. diff --git a/docs/specs/089-orient-wizard/tasks.md b/docs/specs/089-orient-wizard/tasks.md new file mode 100644 index 00000000..774102ae --- /dev/null +++ b/docs/specs/089-orient-wizard/tasks.md @@ -0,0 +1,7 @@ +# Tasks: Orient Wizard + +- [x] T001 Spec 089 artifacts. +- [x] T002 Implement orient-wizard.sh. +- [x] T003 build-orient-bundle.sh ranking and budget fixes. +- [x] T004 Smoke A portolan + Smoke B bigtop + reviews/smoke-findings.md. +- [x] T005 Docs: SKILL, README, INSTALL-PROMPT, backlog, AGENTS. diff --git a/scripts/orient-wizard.sh b/scripts/orient-wizard.sh new file mode 100755 index 00000000..f49a4adb --- /dev/null +++ b/scripts/orient-wizard.sh @@ -0,0 +1,304 @@ +#!/usr/bin/env bash +# One-command orient: tool check → consent install → recipes → bundle → viewer. +# See docs/specs/089-orient-wizard/ and harness/SKILL.md. +set -euo pipefail + +ROOT=$(cd "$(dirname "$0")/.." && pwd) + +YES=0 +SKIP_INSTALL=0 +NO_VIEWER=0 +PORT=4173 +LIMIT_REPOS=0 +PRODUCERS="jscpd,semgrep,syft" +HOTSPOT_BUDGET=200 +SHARD_TIMEOUT=600 +JSCPD_MEMORY_MB=2048 + +usage() { + cat <<'EOF' +usage: orient-wizard.sh [options] + +Options: + --yes Auto-approve tool installs + --skip-install Never install missing tools (gaps only) + --no-viewer Build bundle only; do not start viewer + --port N Viewer port (default 4173) + --limit-repos N Cap discovered git repos for sharded producers + --producers LIST Comma-separated: jscpd,semgrep,syft (default all three) + --hotspot-budget N Max hotspots in bundle (default 200) + --shard-timeout SEC Per-shard timeout in seconds (default 600) + --jscpd-memory-mb N Node heap cap for each jscpd shard (default 2048) + -h, --help Show this help +EOF +} + +log() { echo "orient-wizard: $*" >&2; } +fail_log() { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) $*" >>"$FAILURES_LOG"; } + +POSITIONAL=() +while [[ $# -gt 0 ]]; do + case "$1" in + --yes) YES=1; shift ;; + --skip-install) SKIP_INSTALL=1; shift ;; + --no-viewer) NO_VIEWER=1; shift ;; + --port) PORT="$2"; shift 2 ;; + --limit-repos) LIMIT_REPOS="$2"; shift 2 ;; + --producers) PRODUCERS="$2"; shift 2 ;; + --hotspot-budget) HOTSPOT_BUDGET="$2"; shift 2 ;; + --shard-timeout) SHARD_TIMEOUT="$2"; shift 2 ;; + --jscpd-memory-mb) JSCPD_MEMORY_MB="$2"; shift 2 ;; + -h|--help) usage; exit 0 ;; + --) shift; POSITIONAL+=("$@"); break ;; + -*) echo "unknown option: $1" >&2; usage; exit 2 ;; + *) POSITIONAL+=("$1"); shift ;; + esac +done + +if [[ ${#POSITIONAL[@]} -lt 2 ]]; then + usage >&2 + exit 2 +fi + +TARGET_ROOT=$(cd "${POSITIONAL[0]}" && pwd) +ORIENT_DIR=${POSITIONAL[1]} +PRODUCERS_DIR="$ORIENT_DIR/producers" +FAILURES_LOG="$PRODUCERS_DIR/_failures.log" +SHARD_GAPS="$PRODUCERS_DIR/_gaps.jsonl" +SEMGREP_RULES="$ROOT/harness/recipes/semgrep-rules/portolan-local.yaml" + +mkdir -p "$ORIENT_DIR" "$PRODUCERS_DIR" +: >"$FAILURES_LOG" +: >"$SHARD_GAPS" + +append_shard_gap() { + local id=$1 surface=$2 status=$3 summary=$4 repo=$5 + jq -nc \ + --arg id "$id" --arg surface "$surface" --arg status "$status" \ + --arg summary "$summary" --arg repo "$repo" \ + '{id:$id,surface:$surface,status:$status,summary:$summary,repo:$repo}' >>"$SHARD_GAPS" +} + +run_shard() { + local producer=$1 repo=$2 + shift 2 + local slug + slug=$(repo_slug "$repo") + if ! timeout "$SHARD_TIMEOUT" "$@"; then + local code=$? + if [[ $code -eq 124 ]]; then + append_shard_gap "shard-${producer}-${slug}" "$producer" "failed" \ + "${producer} timed out after ${SHARD_TIMEOUT}s on ${slug}" "$repo" + fail_log "${producer} timeout: $repo" + else + append_shard_gap "shard-${producer}-${slug}" "$producer" "failed" \ + "${producer} failed (exit ${code}) on ${slug}" "$repo" + fail_log "${producer} failed: $repo (exit $code)" + fi + return 1 + fi + return 0 +} + +command -v jq >/dev/null || { log "jq is required"; exit 1; } +command -v node >/dev/null || { log "node is required for viewer"; exit 1; } + +has_producer() { + local p=$1 + [[ ",$PRODUCERS," == *",$p,"* ]] +} + +confirm() { + local prompt=$1 + [[ "$YES" -eq 1 ]] && return 0 + read -r -p "$prompt [y/N] " ans + [[ "${ans,,}" == "y" || "${ans,,}" == "yes" ]] +} + +install_tool() { + local tool=$1 + shift + local -a cmds=("$@") + if [[ "$SKIP_INSTALL" -eq 1 ]]; then + log "skip install $tool (--skip-install)" + return 1 + fi + if command -v "$tool" >/dev/null 2>&1; then + return 0 + fi + log "missing: $tool" + for c in "${cmds[@]}"; do + log " try: $c" + done + if ! confirm "Install $tool now?"; then + log "skipped $tool (operator declined)" + return 1 + fi + for c in "${cmds[@]}"; do + log "running: $c" + if eval "$c"; then + command -v "$tool" >/dev/null && return 0 + fi + done + log "failed to install $tool" + return 1 +} + +ensure_tools() { + if has_producer jscpd; then + install_tool jscpd \ + "npm install -g jscpd" \ + "brew install jscpd" || true + fi + if has_producer semgrep; then + install_tool semgrep \ + "pipx install semgrep" \ + "brew install semgrep" || true + fi + if has_producer syft; then + install_tool syft \ + "brew install syft" \ + "curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin" || true + fi +} + +discover_repos() { + local -a repos=() + if [[ -d "$TARGET_ROOT/.git" ]]; then + repos=("$TARGET_ROOT") + else + while IFS= read -r gitdir; do + repos+=("$(dirname "$gitdir")") + done < <(find "$TARGET_ROOT" -name .git -type d 2>/dev/null | sort) + if [[ ${#repos[@]} -eq 0 ]]; then + repos=("$TARGET_ROOT") + fi + fi + if [[ "$LIMIT_REPOS" -gt 0 && ${#repos[@]} -gt "$LIMIT_REPOS" ]]; then + log "limiting repos from ${#repos[@]} to $LIMIT_REPOS" + repos=("${repos[@]:0:$LIMIT_REPOS}") + fi + printf '%s\n' "${repos[@]}" +} + +repo_slug() { + local p=$1 + basename "$p" | tr ' /' '__' +} + +run_jscpd() { + command -v jscpd >/dev/null || { log "jscpd not available"; return 1; } + local repo out + while IFS= read -r repo; do + [[ -z "$repo" ]] && continue + local slug + slug=$(repo_slug "$repo") + out="$PRODUCERS_DIR/jscpd/$slug" + mkdir -p "$out" + log "jscpd: $repo (${JSCPD_MEMORY_MB}MB cap, ${SHARD_TIMEOUT}s timeout)" + NODE_OPTIONS="--max-old-space-size=${JSCPD_MEMORY_MB}" \ + run_shard jscpd "$repo" \ + jscpd "$repo" \ + --reporters json \ + --output "$out" \ + --min-lines 5 \ + --min-tokens 50 \ + --threshold 999999 \ + --noSymlinks \ + --ignore "**/node_modules/**,**/.git/**,**/vendor/**" \ + 2>>"$FAILURES_LOG" || true + done < <(discover_repos) +} + +run_semgrep() { + command -v semgrep >/dev/null || { log "semgrep not available"; return 1; } + local rules="$SEMGREP_RULES" + if [[ ! -f "$rules" ]]; then + rules="p/default" + log "local semgrep rules missing; using p/default (needs approval for network rules)" + fi + mkdir -p "$PRODUCERS_DIR/semgrep" + local repos + mapfile -t repos < <(discover_repos) + if [[ ${#repos[@]} -eq 1 ]]; then + log "semgrep: ${repos[0]}" + run_shard semgrep "${repos[0]}" \ + semgrep scan "${repos[0]}" \ + --config "$rules" \ + --json \ + --output "$PRODUCERS_DIR/semgrep/findings.json" \ + --metrics off 2>>"$FAILURES_LOG" || true + else + local repo slug + for repo in "${repos[@]}"; do + slug=$(repo_slug "$repo") + log "semgrep: $repo" + run_shard semgrep "$repo" \ + semgrep scan "$repo" \ + --config "$rules" \ + --json \ + --output "$PRODUCERS_DIR/semgrep/${slug}.json" \ + --metrics off 2>>"$FAILURES_LOG" || true + done + fi +} + +run_syft() { + command -v syft >/dev/null || { log "syft not available"; return 1; } + mkdir -p "$PRODUCERS_DIR/syft" + local repo slug + while IFS= read -r repo; do + [[ -z "$repo" ]] && continue + slug=$(repo_slug "$repo") + log "syft: $repo" + run_shard syft "$repo" \ + syft scan "dir:$repo" -o cyclonedx-json \ + >"$PRODUCERS_DIR/syft/${slug}-cyclonedx.json" 2>>"$FAILURES_LOG" || true + done < <(discover_repos) +} + +ensure_tools + +if has_producer jscpd && command -v jscpd >/dev/null; then + run_jscpd || true +elif has_producer jscpd; then + fail_log "jscpd skipped: not installed" +fi + +if has_producer semgrep && command -v semgrep >/dev/null; then + run_semgrep || true +elif has_producer semgrep; then + fail_log "semgrep skipped: not installed" +fi + +if has_producer syft && command -v syft >/dev/null; then + run_syft || true +elif has_producer syft; then + fail_log "syft skipped: not installed" +fi + +export ORIENT_HOTSPOT_BUDGET="$HOTSPOT_BUDGET" +"$ROOT/scripts/build-orient-bundle.sh" "$TARGET_ROOT" "$ORIENT_DIR" + +log "--- summary ---" +if [[ -f "$ORIENT_DIR/manifest.json" ]]; then + jq -r '"hotspots=\(.hotspot_count) gaps=\(.gap_count) target=\(.target_root)"' "$ORIENT_DIR/manifest.json" +fi +if [[ -f "$ORIENT_DIR/hotspots.jsonl" ]]; then + log "top hotspots:" + head -5 "$ORIENT_DIR/hotspots.jsonl" | jq -r '" #\(.rank) [\(.kind)] \(.summary)"' +fi +if [[ -f "$ORIENT_DIR/gaps.jsonl" && -s "$ORIENT_DIR/gaps.jsonl" ]]; then + log "gaps:" + cat "$ORIENT_DIR/gaps.jsonl" | jq -r '" \(.surface): \(.status) — \(.summary)"' +fi + +if [[ "$NO_VIEWER" -eq 1 ]]; then + log "bundle ready at $ORIENT_DIR (--no-viewer)" + exit 0 +fi + +cd "$ROOT/viewer" +node scripts/build-static.js +log "viewer: http://127.0.0.1:$PORT/ (Ctrl+C to stop)" +exec node scripts/serve.js --bundle "$ORIENT_DIR" --port "$PORT" From 00043ac363c28b5150b1dbe642783a978809c588 Mon Sep 17 00:00:00 2001 From: Andrey Zhukov Date: Wed, 10 Jun 2026 17:53:42 +0300 Subject: [PATCH 03/12] Make orient viewer demo-ready: search, filters, heat map, source preview Spec 090: client-side search, kind/severity/repo filter chips, collapsible directory heat tree, truncation and gaps banners, and a path-guarded read-only /source endpoint (403 outside repos.json roots). Demo runbook and smoke checks for search UI and path safety. Co-authored-by: Cursor --- docs/demo-runbook.md | 59 +++ docs/specs/090-viewer-demo-ux/plan.md | 18 + .../reviews/demo-findings.md | 38 ++ docs/specs/090-viewer-demo-ux/spec.md | 40 ++ docs/specs/090-viewer-demo-ux/tasks.md | 7 + viewer/README.md | 24 + viewer/package.json | 11 + viewer/scripts/build-static.js | 11 + viewer/scripts/serve.js | 151 ++++++ viewer/src/app.js | 458 ++++++++++++++++++ viewer/src/index.html | 47 ++ viewer/src/styles.css | 219 +++++++++ 12 files changed, 1083 insertions(+) create mode 100644 docs/demo-runbook.md create mode 100644 docs/specs/090-viewer-demo-ux/plan.md create mode 100644 docs/specs/090-viewer-demo-ux/reviews/demo-findings.md create mode 100644 docs/specs/090-viewer-demo-ux/spec.md create mode 100644 docs/specs/090-viewer-demo-ux/tasks.md create mode 100644 viewer/README.md create mode 100644 viewer/package.json create mode 100644 viewer/scripts/build-static.js create mode 100644 viewer/scripts/serve.js create mode 100644 viewer/src/app.js create mode 100644 viewer/src/index.html create mode 100644 viewer/src/styles.css diff --git a/docs/demo-runbook.md b/docs/demo-runbook.md new file mode 100644 index 00000000..bd386432 --- /dev/null +++ b/docs/demo-runbook.md @@ -0,0 +1,59 @@ +# Portolan Demo Runbook + +Live demo for a newcomer: **where code pain is** → **filter** → **directory tree** → **source**. + +## Setup (two commands) + +**Single repo (portolan):** + +```bash +scripts/orient-wizard.sh . /tmp/orient-portolan --yes +``` + +**Bounded multi-repo (bigtop quick sample):** + +```bash +scripts/orient-wizard.sh ~/projects/bigtop-landscape/repos /tmp/orient-bigtop \ + --yes --limit-repos 3 --producers semgrep,syft +``` + +**Full landscape stress (18 repos, spec 091):** + +```bash +scripts/orient-wizard.sh ~/projects/bigtop-landscape/repos /tmp/orient-bigtop-full \ + --no-viewer --yes --shard-timeout 600 --jscpd-memory-mb 2048 +``` + +Expect 30–90+ minutes. Failed shards appear in gaps (not a wizard abort). +Use viewer on the result: `node viewer/scripts/serve.js --bundle /tmp/orient-bigtop-full` + +Wizard opens viewer automatically. For a fixed port: + +```bash +cd viewer && node scripts/build-static.js +node scripts/serve.js --bundle /tmp/orient-portolan --port 4173 +``` + +Open http://127.0.0.1:4173/ + +## 5-step demo script + +1. **Context** — Point at header: target path, hotspot count, evidence-backed (not LLM graph). +2. **Search** — Type `TODO` or `duplicate`; list and tree narrow instantly. +3. **Filter** — Toggle `static-finding` or `duplication`; on bigtop, pick a **Repo** chip. +4. **Directory heat map** — Expand a hot folder (severity bar + count); click a hotspot row. +5. **Source** — In Detail, show file paths and the **Source** snippet from local files (read-only). + +## Talking points + +- Gaps/truncation banner = honest limits (`not_assessed`, budget cap). +- Every claim ties to `producer_ref` (jscpd, semgrep, syft). +- No network; local-first. + +## Troubleshooting + +| Issue | Fix | +| --- | --- | +| Empty viewer | Run `node viewer/scripts/build-static.js` first | +| No source snippet | Hotspot has no paths (e.g. dep-hub) — expected | +| Missing tools | Re-run wizard with `--yes` or install jscpd/semgrep/syft | diff --git a/docs/specs/090-viewer-demo-ux/plan.md b/docs/specs/090-viewer-demo-ux/plan.md new file mode 100644 index 00000000..e4886bc0 --- /dev/null +++ b/docs/specs/090-viewer-demo-ux/plan.md @@ -0,0 +1,18 @@ +# Implementation Plan: Viewer Demo UX + +**Date**: 2026-06-10 + +## Deliverables + +- [viewer/src/](../../../viewer/src/) — search, filters, heat tree, detail + source +- [viewer/scripts/serve.js](../../../viewer/scripts/serve.js) — `/source` endpoint +- [docs/demo-runbook.md](../../../docs/demo-runbook.md) +- Smoke extensions in [scripts/harness-orient-smoke.sh](../../../scripts/harness-orient-smoke.sh) + +## Verification + +```bash +node viewer/scripts/build-static.js +scripts/harness-orient-smoke.sh +scripts/orient-wizard.sh . /tmp/orient-portolan --no-viewer --yes +``` diff --git a/docs/specs/090-viewer-demo-ux/reviews/demo-findings.md b/docs/specs/090-viewer-demo-ux/reviews/demo-findings.md new file mode 100644 index 00000000..65e4bc38 --- /dev/null +++ b/docs/specs/090-viewer-demo-ux/reviews/demo-findings.md @@ -0,0 +1,38 @@ +# Demo findings: Viewer UX (spec 090) + +**Date**: 2026-06-10 + +## Smoke regression + +`scripts/harness-orient-smoke.sh` — **ok** + +New checks: +- `id="search-input"` in HTML +- `/source?path=sample.go&line=1` returns fixture content +- `/source?path=../../../etc/passwd` → **403** + +## Demo A — portolan bundle + +Bundle: `/tmp/orient-portolan` (128 hotspots, 0 gaps) + +| Check | Result | +| --- | --- | +| Search UI | Present | +| Directory heat tree | Groups `docs/`, `.specify/`, `pkg/` etc. | +| Source preview | Loads `sample.go` via `/source` | +| Truncation banner | Hidden (not truncated) | + +## Demo B — bigtop bundle (bounded) + +Bundle: `/tmp/orient-bigtop` (200 of 830 hotspots, truncated) + +| Check | Result | +| --- | --- | +| Truncation banner | Shows "200 of 830" with budget | +| Repo filter chips | Visible (3 repos in repos.json) | +| Heat tree | Usable at 200 nodes (vs flat tiles before) | +| Gaps banner | 1 gap surface shown in header | + +## Demo flow verdict + +Suitable for live demo on single repo + bounded multi-repo without "this is raw MVP" caveat for navigation. Source preview works for file-backed hotspots; dep-hubs show without source snippet (expected). diff --git a/docs/specs/090-viewer-demo-ux/spec.md b/docs/specs/090-viewer-demo-ux/spec.md new file mode 100644 index 00000000..944a4159 --- /dev/null +++ b/docs/specs/090-viewer-demo-ux/spec.md @@ -0,0 +1,40 @@ +# Feature Specification: Viewer Demo UX + +**Feature Branch**: `codex/090-viewer-demo-ux` + +**Created**: 2026-06-10 + +**Status**: Implemented (smoke + demo evidence in reviews/). + +**Input**: Demo-ready orient viewer: search, filters, directory heat tree, source +preview, truncation transparency. + +## User Scenarios + +### User Story 1 - Demo Navigation (Priority: P1) + +An operator opens the viewer on a portolan or bigtop bundle and can search, +filter by kind/severity/repo, and see where pain clusters in the directory tree. + +### User Story 2 - Click-to-Source (Priority: P1) + +Selecting a hotspot with file paths shows a read-only source snippet from the +local target (path-guarded `/source` endpoint). + +### User Story 3 - Honest Truncation (Priority: P2) + +When hotspot budget truncated the bundle, the viewer shows how many were omitted. + +## Requirements + +- **FR-001**: Client-side search over summary, paths, id. +- **FR-002**: Filter chips: kind, severity, repo (from repos.json). +- **FR-003**: Collapsible directory heat tree aggregated from hotspot paths. +- **FR-004**: `/source?path=&line=` with repos.json prefix guard; 403 outside roots. +- **FR-005**: Truncation and gaps banners in header area. +- **FR-006**: Demo runbook and smoke evidence in reviews/. + +## Success Criteria + +- **SC-001**: harness-orient-smoke passes with search UI and /source path-safety checks. +- **SC-002**: Demo works on portolan bundle (~128 hotspots) and bigtop bundle (200). diff --git a/docs/specs/090-viewer-demo-ux/tasks.md b/docs/specs/090-viewer-demo-ux/tasks.md new file mode 100644 index 00000000..c360744d --- /dev/null +++ b/docs/specs/090-viewer-demo-ux/tasks.md @@ -0,0 +1,7 @@ +# Tasks: Viewer Demo UX + +- [x] T001 Spec 090 artifacts. +- [x] T002 Viewer navigation (search, filters, heat tree, banners). +- [x] T003 Source preview endpoint + detail panel. +- [x] T004 Smoke + demo findings + demo-runbook. +- [x] T005 Docs and backlog P7-090. diff --git a/viewer/README.md b/viewer/README.md new file mode 100644 index 00000000..36e529da --- /dev/null +++ b/viewer/README.md @@ -0,0 +1,24 @@ +# Portolan Orient Viewer + +UA-inspired local map for evidence-backed hotspots. Loads `orient/` bundle only; +does not accept LLM-generated graphs as truth. + +## Build and serve + +```bash +node scripts/build-static.js +node scripts/serve.js --bundle /path/to/orient +``` + +Or with npm when available: `npm run build && npm run serve -- --bundle /path/to/orient` + +Open http://127.0.0.1:4173/ + +## Fixture smoke + +```bash +npm run build +npm run serve -- --bundle ../internal/testfixtures/orient-bundle/orient +``` + +Read-only, local-only, stops when the server process exits. diff --git a/viewer/package.json b/viewer/package.json new file mode 100644 index 00000000..145ae9f3 --- /dev/null +++ b/viewer/package.json @@ -0,0 +1,11 @@ +{ + "name": "portolan-orient-viewer", + "version": "0.1.0", + "private": true, + "description": "Local evidence-backed orient map viewer (UA-inspired, no LLM graph truth)", + "scripts": { + "build": "node scripts/build-static.js", + "serve": "node scripts/serve.js" + }, + "license": "MIT" +} diff --git a/viewer/scripts/build-static.js b/viewer/scripts/build-static.js new file mode 100644 index 00000000..cc57342f --- /dev/null +++ b/viewer/scripts/build-static.js @@ -0,0 +1,11 @@ +#!/usr/bin/env node +const fs = require('fs'); +const path = require('path'); + +const src = path.join(__dirname, '..', 'src'); +const dist = path.join(__dirname, '..', 'dist'); +fs.mkdirSync(dist, { recursive: true }); +for (const name of ['index.html', 'app.js', 'styles.css']) { + fs.copyFileSync(path.join(src, name), path.join(dist, name)); +} +console.log('built viewer static assets to dist/'); diff --git a/viewer/scripts/serve.js b/viewer/scripts/serve.js new file mode 100644 index 00000000..c210e057 --- /dev/null +++ b/viewer/scripts/serve.js @@ -0,0 +1,151 @@ +#!/usr/bin/env node +const http = require('http'); +const fs = require('fs'); +const path = require('path'); +const { URL } = require('url'); + +const args = process.argv.slice(2); +let bundlePath = ''; +let port = 4173; +for (let i = 0; i < args.length; i++) { + if (args[i] === '--bundle' && args[i + 1]) { + bundlePath = path.resolve(args[++i]); + } else if (args[i] === '--port' && args[i + 1]) { + port = parseInt(args[++i], 10); + } +} + +if (!bundlePath || !fs.existsSync(bundlePath)) { + console.error('usage: npm run serve -- --bundle [--port 4173]'); + process.exit(2); +} + +const distDir = path.join(__dirname, '..', 'dist'); +const mime = { + '.html': 'text/html', + '.js': 'application/javascript', + '.css': 'text/css', + '.json': 'application/json', + '.jsonl': 'application/json', +}; + +let repoRoots = []; +function loadRepoRoots() { + const reposFile = path.join(bundlePath, 'repos.json'); + if (!fs.existsSync(reposFile)) return []; + try { + const repos = JSON.parse(fs.readFileSync(reposFile, 'utf8')); + return repos.map((r) => path.resolve(r.path)); + } catch { + return []; + } +} +repoRoots = loadRepoRoots(); + +function isUnderRepoRoot(filePath) { + const resolved = path.resolve(filePath); + return repoRoots.some( + (root) => resolved === root || resolved.startsWith(root + path.sep) + ); +} + +function resolveSourcePath(requestPath) { + if (!requestPath || typeof requestPath !== 'string') return null; + const raw = requestPath.trim(); + if (!raw || raw.includes('\0')) return null; + + if (path.isAbsolute(raw)) { + const resolved = path.resolve(raw); + if (!isUnderRepoRoot(resolved)) return null; + return fs.existsSync(resolved) && fs.statSync(resolved).isFile() ? resolved : null; + } + + for (const root of repoRoots) { + const candidate = path.resolve(root, raw); + if (!candidate.startsWith(root + path.sep) && candidate !== root) continue; + if (fs.existsSync(candidate) && fs.statSync(candidate).isFile()) { + return candidate; + } + } + return null; +} + +function readSourceSnippet(filePath, lineNum) { + const content = fs.readFileSync(filePath, 'utf8'); + const lines = content.split('\n'); + const total = lines.length; + const center = Math.min(Math.max(parseInt(lineNum, 10) || 1, 1), total || 1); + const radius = 20; + const start = Math.max(1, center - radius); + const end = Math.min(total, center + radius); + const snippet = []; + for (let i = start; i <= end; i++) { + snippet.push({ + no: i, + text: lines[i - 1] ?? '', + highlight: i === center, + }); + } + return { path: filePath, line: center, startLine: start, endLine: end, totalLines: total, lines: snippet }; +} + +const server = http.createServer((req, res) => { + const url = new URL(req.url, `http://127.0.0.1:${port}`); + + if (url.pathname === '/source') { + const filePath = resolveSourcePath(url.searchParams.get('path') || ''); + if (!filePath) { + res.writeHead(403, { 'Content-Type': 'application/json' }); + return res.end(JSON.stringify({ error: 'forbidden or not found' })); + } + try { + const line = url.searchParams.get('line') || '1'; + const body = readSourceSnippet(filePath, line); + res.writeHead(200, { 'Content-Type': 'application/json' }); + return res.end(JSON.stringify(body)); + } catch { + res.writeHead(500, { 'Content-Type': 'application/json' }); + return res.end(JSON.stringify({ error: 'read failed' })); + } + } + + if (url.pathname === '/bundle/manifest.json') { + return sendFile(path.join(bundlePath, 'manifest.json'), res); + } + if (url.pathname === '/bundle/hotspots.jsonl') { + return sendFile(path.join(bundlePath, 'hotspots.jsonl'), res); + } + if (url.pathname === '/bundle/gaps.jsonl') { + return sendFile(path.join(bundlePath, 'gaps.jsonl'), res); + } + if (url.pathname === '/bundle/graph-slice.json') { + return sendFile(path.join(bundlePath, 'graph-slice.json'), res); + } + if (url.pathname === '/bundle/repos.json') { + return sendFile(path.join(bundlePath, 'repos.json'), res); + } + + let filePath = path.join(distDir, url.pathname === '/' ? 'index.html' : url.pathname); + if (!filePath.startsWith(distDir)) { + res.writeHead(403); + return res.end('Forbidden'); + } + sendFile(filePath, res); +}); + +function sendFile(filePath, res) { + fs.readFile(filePath, (err, data) => { + if (err) { + res.writeHead(404); + return res.end('Not found'); + } + const ext = path.extname(filePath); + res.writeHead(200, { 'Content-Type': mime[ext] || 'text/plain' }); + res.end(data); + }); +} + +server.listen(port, '127.0.0.1', () => { + console.log(`Portolan orient viewer: http://127.0.0.1:${port}/`); + console.log(`Bundle: ${bundlePath}`); +}); diff --git a/viewer/src/app.js b/viewer/src/app.js new file mode 100644 index 00000000..92667982 --- /dev/null +++ b/viewer/src/app.js @@ -0,0 +1,458 @@ +/** + * Portolan orient viewer — evidence bundle from /bundle/*; UA-inspired navigation. + */ + +const SEV_RANK = { critical: 0, high: 1, medium: 2, low: 3, info: 4 }; + +let allHotspots = []; +let repos = []; +let manifest = null; +let gaps = []; +let filters = { kinds: new Set(), severities: new Set(), repoIds: new Set() }; +let searchQuery = ''; +let selectedId = null; +let expandedDirs = new Set(); + +async function loadJSONL(url) { + const res = await fetch(url); + if (!res.ok) return []; + const text = await res.text(); + return text + .split('\n') + .map((l) => l.trim()) + .filter(Boolean) + .map((l) => JSON.parse(l)); +} + +async function loadJSON(url) { + const res = await fetch(url); + if (!res.ok) return null; + return res.json(); +} + +function escapeHtml(s) { + return String(s) + .replace(/&/g, '&') + .replace(//g, '>'); +} + +function sevRank(s) { + return SEV_RANK[s] ?? 5; +} + +function maxSeverity(sevs) { + return sevs.reduce((best, s) => (sevRank(s) < sevRank(best) ? s : best), 'info'); +} + +function sevClass(s) { + return `sev-${s || 'info'}`; +} + +function kindClass(kind) { + return kind || 'debt-candidate'; +} + +function normalizeDisplayPath(p) { + if (!p) return ''; + return p.replace(/\\/g, '/'); +} + +function repoForPath(p) { + const norm = normalizeDisplayPath(p); + if (!norm) return null; + if (norm.startsWith('/')) { + for (const r of repos) { + const root = r.path.replace(/\\/g, '/'); + if (norm === root || norm.startsWith(root + '/')) return r.id; + } + return null; + } + if (repos.length === 1) return repos[0].id; + for (const r of repos) { + const name = r.name || r.id; + if (norm.startsWith(name + '/')) return r.id; + } + return repos[0]?.id ?? null; +} + +function hotspotRepo(h) { + for (const p of h.paths || []) { + const rid = repoForPath(p); + if (rid) return rid; + } + return repos.length === 1 ? repos[0].id : null; +} + +function matchesSearch(h, q) { + if (!q) return true; + const hay = [ + h.summary, + h.id, + h.kind, + h.severity, + ...(h.paths || []), + ] + .join(' ') + .toLowerCase(); + return hay.includes(q); +} + +function matchesFilters(h) { + if (filters.kinds.size && !filters.kinds.has(h.kind)) return false; + if (filters.severities.size && !filters.severities.has(h.severity)) return false; + if (filters.repoIds.size) { + const rid = hotspotRepo(h); + if (!rid || !filters.repoIds.has(rid)) return false; + } + return true; +} + +function filteredHotspots() { + const q = searchQuery.trim().toLowerCase(); + return allHotspots + .filter((h) => matchesSearch(h, q) && matchesFilters(h)) + .sort((a, b) => a.rank - b.rank); +} + +function buildTree(hotspots) { + const root = { + name: '', + pathKey: '', + children: new Map(), + hotspotIds: new Set(), + isFile: false, + }; + + function ensureChild(parent, name, pathKey, isFile) { + if (!parent.children.has(name)) { + parent.children.set(name, { + name, + pathKey, + children: new Map(), + hotspotIds: new Set(), + isFile, + }); + } + return parent.children.get(name); + } + + function addPath(pathStr, h) { + const norm = normalizeDisplayPath(pathStr); + if (!norm || norm === '(dependency-hubs)') { + const node = ensureChild(root, '(dependency-hubs)', '(dependency-hubs)', false); + node.hotspotIds.add(h.id); + return; + } + const parts = norm.split('/').filter(Boolean); + let node = root; + let built = ''; + for (let i = 0; i < parts.length; i++) { + const part = parts[i]; + built = built ? `${built}/${part}` : part; + const isFile = i === parts.length - 1; + node = ensureChild(node, part, built, isFile); + node.hotspotIds.add(h.id); + } + } + + for (const h of hotspots) { + if (!h.paths || h.paths.length === 0) { + addPath('(dependency-hubs)', h); + } else { + for (const p of h.paths) addPath(p, h); + } + } + return root; +} + +function treeStats(node, hotspotById) { + const ids = [...node.hotspotIds]; + const sevs = ids.map((id) => hotspotById.get(id)?.severity).filter(Boolean); + return { count: ids.length, maxSeverity: maxSeverity(sevs.length ? sevs : ['info']) }; +} + +function renderFilters() { + const bar = document.getElementById('filter-bar'); + bar.innerHTML = ''; + + const kinds = [...new Set(allHotspots.map((h) => h.kind))].sort(); + const severities = [...new Set(allHotspots.map((h) => h.severity))].sort( + (a, b) => sevRank(a) - sevRank(b) + ); + + bar.appendChild(makeChipGroup('Kind', kinds, filters.kinds, () => render())); + bar.appendChild(makeChipGroup('Severity', severities, filters.severities, () => render())); + if (repos.length > 1) { + const repoLabels = repos.map((r) => ({ id: r.id, label: r.name || r.id })); + bar.appendChild( + makeChipGroup( + 'Repo', + repoLabels.map((r) => r.id), + filters.repoIds, + () => render(), + (id) => repoLabels.find((r) => r.id === id)?.label || id + ) + ); + } +} + +function makeChipGroup(label, values, activeSet, onChange, labelFn = (v) => v) { + const wrap = document.createElement('div'); + wrap.className = 'filter-group'; + const lbl = document.createElement('span'); + lbl.className = 'filter-label'; + lbl.textContent = label; + wrap.appendChild(lbl); + for (const v of values) { + const btn = document.createElement('button'); + btn.type = 'button'; + btn.className = 'chip' + (activeSet.has(v) ? ' active' : ''); + btn.textContent = labelFn(v); + btn.addEventListener('click', () => { + if (activeSet.has(v)) activeSet.delete(v); + else activeSet.add(v); + onChange(); + }); + wrap.appendChild(btn); + } + return wrap; +} + +function renderBanner() { + const el = document.getElementById('status-banner'); + const parts = []; + if (manifest?.hotspots_truncated) { + parts.push( + `Showing ${manifest.hotspot_count} of ${manifest.hotspots_total} hotspots (budget ${manifest.hotspot_budget} applied).` + ); + } + if (gaps.length) { + const summary = gaps.map((g) => `${g.surface}: ${g.status}`).join(' · '); + parts.push(`Gaps: ${escapeHtml(summary)}`); + } + if (parts.length) { + el.innerHTML = parts.join(' '); + el.classList.remove('hidden'); + } else { + el.innerHTML = ''; + el.classList.add('hidden'); + } +} + +function renderTour(hotspots, hotspotById) { + const list = document.getElementById('hotspot-list'); + list.innerHTML = ''; + document.getElementById('tour-count').textContent = `(${hotspots.length})`; + for (const h of hotspots) { + const li = document.createElement('li'); + li.className = h.id === selectedId ? 'active' : ''; + li.innerHTML = `${escapeHtml(h.severity)}${escapeHtml(h.summary)}`; + li.addEventListener('click', () => selectHotspot(h)); + list.appendChild(li); + } +} + +function renderTreeNode(node, hotspotById, parentEl, depth = 0) { + const entries = [...node.children.entries()].sort((a, b) => { + if (a[1].isFile !== b[1].isFile) return a[1].isFile ? 1 : -1; + return a[0].localeCompare(b[0]); + }); + + for (const [, child] of entries) { + const stats = treeStats(child, hotspotById); + const hasChildren = child.children.size > 0; + const isExpanded = expandedDirs.has(child.pathKey) || depth < 2; + + const row = document.createElement('div'); + row.className = 'tree-node'; + + const line = document.createElement('div'); + line.className = 'tree-row'; + const toggle = document.createElement('span'); + toggle.className = 'tree-toggle' + (hasChildren ? '' : ' empty'); + toggle.textContent = hasChildren ? (isExpanded ? '▾' : '▸') : ''; + const bar = document.createElement('span'); + bar.className = `sev-bar ${child.isFile ? sevClass(stats.maxSeverity) : sevClass(stats.maxSeverity)}`; + const name = document.createElement('span'); + name.className = 'tree-name'; + name.textContent = child.name; + const meta = document.createElement('span'); + meta.className = 'tree-meta'; + meta.textContent = `${stats.count}`; + line.appendChild(toggle); + line.appendChild(bar); + line.appendChild(name); + line.appendChild(meta); + + if (hasChildren) { + toggle.addEventListener('click', (e) => { + e.stopPropagation(); + if (expandedDirs.has(child.pathKey)) expandedDirs.delete(child.pathKey); + else expandedDirs.add(child.pathKey); + render(); + }); + } + + row.appendChild(line); + + if (child.isFile) { + const fileHotspots = [...child.hotspotIds] + .map((id) => hotspotById.get(id)) + .filter(Boolean) + .sort((a, b) => a.rank - b.rank); + for (const h of fileHotspots) { + const hs = document.createElement('div'); + hs.className = 'tree-hotspot' + (h.id === selectedId ? ' active' : ''); + hs.innerHTML = `${escapeHtml(h.kind)}#${h.rank} ${escapeHtml(h.summary)}`; + hs.addEventListener('click', () => selectHotspot(h)); + row.appendChild(hs); + } + } + + const childWrap = document.createElement('div'); + childWrap.className = 'tree-children' + (isExpanded ? '' : ' collapsed'); + if (hasChildren) renderTreeNode(child, hotspotById, childWrap, depth + 1); + row.appendChild(childWrap); + parentEl.appendChild(row); + } + +} + +function renderTree(hotspots) { + const treeEl = document.getElementById('heat-tree'); + treeEl.innerHTML = ''; + const hotspotById = new Map(hotspots.map((h) => [h.id, h])); + const root = buildTree(hotspots); + + if (root.children.has('(dependency-hubs)')) { + const depNode = root.children.get('(dependency-hubs)'); + const stats = treeStats(depNode, hotspotById); + const row = document.createElement('div'); + row.className = 'tree-row'; + row.innerHTML = `(dependency-hubs)${stats.count}`; + treeEl.appendChild(row); + for (const id of depNode.hotspotIds) { + const h = hotspotById.get(id); + if (!h) continue; + const hs = document.createElement('div'); + hs.className = 'tree-hotspot' + (h.id === selectedId ? ' active' : ''); + hs.innerHTML = `${escapeHtml(h.kind)}#${h.rank} ${escapeHtml(h.summary)}`; + hs.addEventListener('click', () => selectHotspot(h)); + treeEl.appendChild(hs); + } + root.children.delete('(dependency-hubs)'); + } + + renderTreeNode(root, hotspotById, treeEl); + if (!treeEl.children.length) { + treeEl.innerHTML = '

No paths match filters.

'; + } +} + +async function loadSourcePreview(h) { + const preview = document.getElementById('source-preview'); + const code = document.getElementById('source-code'); + const path = (h.paths || [])[0]; + if (!path) { + preview.classList.add('hidden'); + return; + } + try { + const params = new URLSearchParams({ path, line: '1' }); + const res = await fetch(`/source?${params}`); + if (!res.ok) { + preview.classList.remove('hidden'); + code.textContent = `Source unavailable (${res.status}) for ${path}`; + return; + } + const data = await res.json(); + preview.classList.remove('hidden'); + code.innerHTML = data.lines + .map( + (ln) => + `${ln.no}${escapeHtml(ln.text)}` + ) + .join('\n'); + } catch (e) { + preview.classList.remove('hidden'); + code.textContent = `Failed to load source: ${e.message}`; + } +} + +function renderDetail(h) { + const el = document.getElementById('detail-body'); + if (!h) { + el.innerHTML = '

Select a hotspot.

'; + document.getElementById('source-preview').classList.add('hidden'); + return; + } + const paths = (h.paths || []) + .map((p) => `
${escapeHtml(p)}
`) + .join(''); + const rid = hotspotRepo(h); + const repoLabel = rid ? repos.find((r) => r.id === rid)?.name || rid : ''; + el.innerHTML = ` +

+ ${escapeHtml(h.severity)} + ${escapeHtml(h.kind)} + ${escapeHtml(h.evidence_state)} + ${escapeHtml(h.producer)} + #${h.rank} + ${repoLabel ? `${escapeHtml(repoLabel)}` : ''} +

+

${escapeHtml(h.summary)}

+

id: ${escapeHtml(h.id)}

+ ${paths || '

(no file paths)

'} +

producer_ref: ${escapeHtml(h.producer_ref || '')}

+ `; + loadSourcePreview(h); +} + +function selectHotspot(h) { + selectedId = h?.id ?? null; + renderDetail(h); + render(); +} + +function render() { + const hotspots = filteredHotspots(); + const hotspotById = new Map(hotspots.map((h) => [h.id, h])); + if (selectedId && !hotspotById.has(selectedId)) { + selectedId = null; + renderDetail(null); + } + renderBanner(); + renderTour(hotspots, hotspotById); + renderTree(hotspots); +} + +async function main() { + manifest = await loadJSON('/bundle/manifest.json'); + allHotspots = await loadJSONL('/bundle/hotspots.jsonl'); + gaps = await loadJSONL('/bundle/gaps.jsonl'); + repos = (await loadJSON('/bundle/repos.json')) || []; + + document.getElementById('manifest-info').textContent = manifest + ? `target: ${manifest.target_root} · hotspots: ${manifest.hotspot_count} · gaps: ${manifest.gap_count}` + : 'no manifest'; + + renderFilters(); + renderBanner(); + + document.getElementById('search-input').addEventListener('input', (e) => { + searchQuery = e.target.value; + render(); + }); + + render(); + if (allHotspots.length) { + const first = filteredHotspots()[0] || allHotspots[0]; + selectHotspot(first); + } +} + +main().catch((e) => { + document.getElementById('detail-body').innerHTML = + `

Failed to load bundle. Run: npm run serve -- --bundle <orient-dir>

${escapeHtml(e.message)}
`; +}); diff --git a/viewer/src/index.html b/viewer/src/index.html new file mode 100644 index 00000000..5ce53823 --- /dev/null +++ b/viewer/src/index.html @@ -0,0 +1,47 @@ + + + + + + Portolan Orient + + + +
+
+
+

Portolan Orient

+

Evidence-backed hotspots — not LLM architecture truth

+
+
+ + +
+
+ +
+
+
+
+

Hotspot tour

+
    +
    +
    +

    Directory heat map

    +
    +
    +
    +

    Detail

    +

    Select a hotspot.

    + +
    +
    +
    + +
    + + + diff --git a/viewer/src/styles.css b/viewer/src/styles.css new file mode 100644 index 00000000..bb9102f3 --- /dev/null +++ b/viewer/src/styles.css @@ -0,0 +1,219 @@ +:root { + --bg: #0f1419; + --panel: #1a2332; + --text: #e7ecf3; + --muted: #8b9bb4; + --dup: #e85d5d; + --static: #e8a35d; + --dep: #5d9ae8; + --debt: #b85de8; + --gap: #6b7280; + --sev-critical: #ff4d4d; + --sev-high: #ff8c42; + --sev-medium: #e8a35d; + --sev-low: #5d9ae8; + --sev-info: #8b9bb4; + --banner-warn: #3d2e14; + --banner-gap: #1e2a3d; +} + +* { box-sizing: border-box; } +body { + margin: 0; + font-family: system-ui, sans-serif; + background: var(--bg); + color: var(--text); +} +.sr-only { + position: absolute; + width: 1px; + height: 1px; + padding: 0; + margin: -1px; + overflow: hidden; + clip: rect(0, 0, 0, 0); + border: 0; +} +header, footer { + padding: 1rem 1.5rem; + border-bottom: 1px solid #2a3548; +} +footer { border-top: 1px solid #2a3548; border-bottom: none; font-size: 0.85rem; color: var(--muted); } +.header-top { + display: flex; + flex-wrap: wrap; + align-items: flex-start; + justify-content: space-between; + gap: 1rem; +} +h1 { margin: 0; font-size: 1.25rem; } +#subtitle { margin: 0.25rem 0 0; color: var(--muted); font-size: 0.9rem; } +.search-wrap { flex: 1; min-width: 200px; max-width: 420px; } +#search-input { + width: 100%; + padding: 0.5rem 0.75rem; + border-radius: 6px; + border: 1px solid #2a3548; + background: var(--panel); + color: var(--text); + font-size: 0.95rem; +} +.status-banner { + margin-top: 0.75rem; + padding: 0.5rem 0.75rem; + border-radius: 6px; + font-size: 0.85rem; +} +.status-banner.hidden { display: none; } +.status-banner.truncation { background: var(--banner-warn); color: #f5d9a8; } +.status-banner.gaps { background: var(--banner-gap); color: var(--muted); } +.filter-bar { + display: flex; + flex-wrap: wrap; + gap: 0.35rem; + margin-top: 0.75rem; +} +.filter-group { + display: flex; + flex-wrap: wrap; + align-items: center; + gap: 0.25rem; + margin-right: 0.75rem; +} +.filter-label { + font-size: 0.7rem; + text-transform: uppercase; + color: var(--muted); + margin-right: 0.25rem; +} +.chip { + font-size: 0.75rem; + padding: 0.2rem 0.5rem; + border-radius: 999px; + border: 1px solid #2a3548; + background: transparent; + color: var(--muted); + cursor: pointer; +} +.chip:hover { border-color: var(--muted); color: var(--text); } +.chip.active { + background: #2a3548; + color: var(--text); + border-color: #4a5568; +} +.layout { + display: grid; + grid-template-columns: minmax(220px, 1fr) minmax(280px, 1.2fr) minmax(260px, 1fr); + gap: 1rem; + padding: 1rem; +} +@media (max-width: 960px) { + .layout { grid-template-columns: 1fr; } +} +.panel { + background: var(--panel); + border-radius: 8px; + padding: 1rem; + min-height: 200px; + max-height: 70vh; + overflow: auto; +} +.panel h2 { margin: 0 0 0.75rem; font-size: 1rem; } +.count-badge { + font-size: 0.75rem; + color: var(--muted); + font-weight: normal; +} +.badge { + display: inline-block; + font-size: 0.7rem; + padding: 0.1rem 0.35rem; + border-radius: 4px; + background: #2a3548; + color: var(--muted); + margin-right: 0.25rem; +} +.badge.sev-critical { background: color-mix(in srgb, var(--sev-critical) 40%, #2a3548); color: #ffc9c9; } +.badge.sev-high { background: color-mix(in srgb, var(--sev-high) 40%, #2a3548); color: #ffd4b8; } +.badge.sev-medium { background: color-mix(in srgb, var(--sev-medium) 40%, #2a3548); color: #ffe4c4; } +.badge.sev-low { background: color-mix(in srgb, var(--sev-low) 40%, #2a3548); color: #c9e0ff; } +.badge.sev-info { background: #2a3548; color: var(--muted); } +#hotspot-list { padding-left: 1.2rem; margin: 0; list-style: decimal; } +#hotspot-list li { + margin-bottom: 0.5rem; + cursor: pointer; + padding: 0.25rem 0; + border-radius: 4px; +} +#hotspot-list li:hover { color: #fff; } +#hotspot-list li.active { background: #2a3548; padding-left: 0.25rem; } +.heat-tree { font-size: 0.85rem; } +.tree-node { margin-left: 0; } +.tree-row { + display: flex; + align-items: center; + gap: 0.35rem; + padding: 0.2rem 0.25rem; + border-radius: 4px; + cursor: pointer; +} +.tree-row:hover { background: #243044; } +.tree-row.active { background: #2d3f5c; } +.tree-toggle { + width: 1rem; + text-align: center; + color: var(--muted); + flex-shrink: 0; + user-select: none; +} +.tree-toggle.empty { visibility: hidden; } +.tree-name { flex: 1; font-family: monospace; font-size: 0.8rem; } +.tree-meta { + font-size: 0.7rem; + color: var(--muted); + flex-shrink: 0; +} +.tree-children { margin-left: 1rem; border-left: 1px solid #2a3548; } +.tree-children.collapsed { display: none; } +.tree-hotspot { + margin-left: 1.25rem; + padding: 0.2rem 0.35rem; + cursor: pointer; + border-radius: 4px; + font-size: 0.8rem; +} +.tree-hotspot:hover { background: #243044; } +.tree-hotspot.active { background: #2d3f5c; } +.sev-bar { + width: 4px; + height: 1rem; + border-radius: 2px; + flex-shrink: 0; +} +.sev-bar.critical { background: var(--sev-critical); } +.sev-bar.high { background: var(--sev-high); } +.sev-bar.medium { background: var(--sev-medium); } +.sev-bar.low { background: var(--sev-low); } +.sev-bar.info { background: var(--sev-info); } +.path { font-family: monospace; font-size: 0.8rem; color: var(--muted); word-break: break-all; } +.source-preview { margin-top: 1rem; border-top: 1px solid #2a3548; padding-top: 0.75rem; } +.source-preview.hidden { display: none; } +.source-preview h3 { margin: 0 0 0.5rem; font-size: 0.9rem; color: var(--muted); } +#source-code { + margin: 0; + padding: 0.75rem; + background: #0a0e14; + border-radius: 6px; + font-size: 0.75rem; + line-height: 1.45; + overflow-x: auto; + max-height: 320px; +} +#source-code .line { display: block; } +#source-code .line.highlight { background: #2d3f5c; } +#source-code .lineno { + display: inline-block; + width: 3.5rem; + color: var(--muted); + user-select: none; +} From c52155d2cf1ef6430e7f11b895673a7baec9a63b Mon Sep 17 00:00:00 2001 From: Andrey Zhukov Date: Wed, 10 Jun 2026 17:53:42 +0300 Subject: [PATCH 04/12] Scale wizard to full bigtop landscape without OOM aborts Spec 091: per-shard Node heap cap and timeout, shard failures recorded as gaps in producers/_gaps.jsonl, single-pass jq producer parsing, kind-quota hotspot budget with hotspots-full.jsonl, and per-kind manifest counters. Full 18-repo bigtop smoke: 29,914 hotspots, ~23 min. Co-authored-by: Cursor --- docs/specs/091-landscape-scale/plan.md | 11 +++++ .../reviews/scale-findings.md | 46 +++++++++++++++++++ docs/specs/091-landscape-scale/spec.md | 24 ++++++++++ docs/specs/091-landscape-scale/tasks.md | 7 +++ 4 files changed, 88 insertions(+) create mode 100644 docs/specs/091-landscape-scale/plan.md create mode 100644 docs/specs/091-landscape-scale/reviews/scale-findings.md create mode 100644 docs/specs/091-landscape-scale/spec.md create mode 100644 docs/specs/091-landscape-scale/tasks.md diff --git a/docs/specs/091-landscape-scale/plan.md b/docs/specs/091-landscape-scale/plan.md new file mode 100644 index 00000000..0e3f4416 --- /dev/null +++ b/docs/specs/091-landscape-scale/plan.md @@ -0,0 +1,11 @@ +# Implementation Plan: Landscape Scale + +**Date**: 2026-06-10 + +## Verification + +```bash +scripts/harness-orient-smoke.sh +scripts/orient-wizard.sh ~/projects/bigtop-landscape/repos /tmp/orient-bigtop-full \ + --no-viewer --yes --shard-timeout 600 +``` diff --git a/docs/specs/091-landscape-scale/reviews/scale-findings.md b/docs/specs/091-landscape-scale/reviews/scale-findings.md new file mode 100644 index 00000000..b1730baa --- /dev/null +++ b/docs/specs/091-landscape-scale/reviews/scale-findings.md @@ -0,0 +1,46 @@ +# Scale findings: full bigtop (spec 091) + +**Date**: 2026-06-10 + +## Smoke C command + +```bash +scripts/orient-wizard.sh ~/projects/bigtop-landscape/repos /tmp/orient-bigtop-full \ + --no-viewer --yes --shard-timeout 600 --jscpd-memory-mb 2048 +``` + +| Metric | Value | +| --- | --- | +| Wall time | ~23 min (START 13:56:14Z → END 14:19:26Z) | +| Repos | 18 / 18 discovered | +| Producer shards | jscpd 18, semgrep 18, syft 18 | +| Hotspots total | 29,914 (`hotspots-full.jsonl`) | +| Hotspots budgeted | 200 (truncated) | +| Kind mix (budgeted) | static-finding 100, duplication 60, dep-hub 40 | +| Kind mix (total) | static-finding 4737, duplication 24035, dep-hub 1142 | +| Shard gaps | 8 jscpd shards (see below) | +| Bundle build (re-run) | ~3.5 min on existing producers | + +## Shard gaps (honest failures) + +During the first wizard run, 8 jscpd shards recorded `failed` because jscpd exits non-zero when clone count exceeds default threshold (even when JSON output exists): airflow, flink, hadoop, hbase, hive, kafka, solr, spark. + +**Fix applied:** `--threshold 999999` on jscpd invocations so completed scans do not false-fail. + +Semgrep and syft: 18/18 shards succeeded. + +## Fixes during slice + +1. **Kind-quota jq**: variable binding `$k` in `map`; `sort_h` on rest pool; `jq -sc` for compact JSONL (pretty-print broke rank loop). +2. **jscpd threshold exit**: treat as success when output exists via `--threshold 999999`. +3. **Bundle build**: single-pass jq per producer file; merge `producers/_gaps.jsonl`. + +## Viewer + +Full bundle loads in viewer; 18 repo filter chips; truncation banner shows 200 of 29,914. + +## Not assessed + +- Parallel shards (`--jobs`) +- Incremental re-scan / cache +- Full jscpd on repos that OOM even with 2048MB cap (none observed this run; spark/hadoop had threshold false-fail only) diff --git a/docs/specs/091-landscape-scale/spec.md b/docs/specs/091-landscape-scale/spec.md new file mode 100644 index 00000000..8203f1d7 --- /dev/null +++ b/docs/specs/091-landscape-scale/spec.md @@ -0,0 +1,24 @@ +# Feature Specification: Landscape Scale + +**Feature Branch**: `codex/091-landscape-scale` + +**Created**: 2026-06-10 + +**Status**: Implemented (scale evidence in reviews/scale-findings.md). + +**Input**: Full bigtop landscape (18 repos) through orient-wizard without OOM abort; +shard failures as honest gaps; scalable bundle build with kind-aware budget. + +## Requirements + +- **FR-001**: `--jscpd-memory-mb` and `--shard-timeout` on orient-wizard.sh. +- **FR-002**: Per-shard failures written to `producers/_gaps.jsonl`. +- **FR-003**: Single-pass jq per producer file in build-orient-bundle.sh. +- **FR-004**: Kind-quota hotspot budget; `hotspots-full.jsonl` for agents. +- **FR-005**: Smoke C on full bigtop with evidence in reviews/. + +## Success Criteria + +- **SC-001**: Wizard completes all 18 bigtop repos without aborting. +- **SC-002**: Failed shards visible in gaps.jsonl. +- **SC-003**: Bundle includes multiple kinds after budget (not only static-finding). diff --git a/docs/specs/091-landscape-scale/tasks.md b/docs/specs/091-landscape-scale/tasks.md new file mode 100644 index 00000000..53c7fa9e --- /dev/null +++ b/docs/specs/091-landscape-scale/tasks.md @@ -0,0 +1,7 @@ +# Tasks: Landscape Scale + +- [x] T001 Spec 091 artifacts. +- [x] T002 Wizard bounded shards. +- [x] T003 Bundle single-pass + kind quotas. +- [x] T004 Smoke C full bigtop. +- [x] T005 Docs P7-091. From 106ee22f52b09f636552b22e8a70932f5f969b7b Mon Sep 17 00:00:00 2001 From: Andrey Zhukov Date: Wed, 10 Jun 2026 17:53:42 +0300 Subject: [PATCH 05/12] Align docs with harness-first delivery path README quick start, AGENTS baseline checks, backlog P7 rows 087-091, product claims, onboarding, and install prompts now route through orient-wizard.sh and the harness pack. Co-authored-by: Cursor --- AGENTS.md | 28 +++++++-- README.md | 28 ++++++++- docs/agent/INSTALL-PROMPT.md | 113 ++++++++++++++--------------------- docs/onboarding.md | 3 +- docs/product-backlog.md | 23 ++++++- docs/product-claims.md | 22 +++---- 6 files changed, 127 insertions(+), 90 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 79553db3..74cf4b88 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -18,8 +18,8 @@ Portolan is not: Portolan is: - a read-only local discovery substrate an agent can run; -- an agent-facing toolbox exposed through CLI first, then skills/MCP/LSP-style - surfaces when justified; +- a harness-first supplement (`harness/SKILL.md`, recipes, guardrails, orient + viewer) with an optional legacy Go CLI bridge; - a normalizer for source, metadata, runtime, and claim evidence; - a machine-readable evidence graph; - a finding generator for relationships, duplication, configuration surfaces, @@ -42,8 +42,11 @@ Portolan is: ## Engineering Rules -- Primary implementation language: Go. -- Keep `cmd/portolan` thin; put behavior in internal packages. +- Primary delivery: harness artifacts and orient bundle contract; see spec 087. +- Go CLI is frozen for new features per + [`docs/harness/GO-FREEZE-POLICY.md`](docs/harness/GO-FREEZE-POLICY.md). +- Keep `cmd/portolan` thin; put behavior in internal packages when Go changes are + allowed (bugfix/bridge only during freeze). - Add focused tests before behavior changes. - Do not add dependencies unless the product boundary and integration cost are documented. @@ -190,6 +193,9 @@ Run: go test ./... go vet ./... jq empty schema/*.json +jq empty harness/contracts/orient-bundle.schema.json +scripts/harness-orient-smoke.sh +scripts/orient-wizard.sh --help git diff --check ``` @@ -202,5 +208,17 @@ go run ./cmd/portolan scan --help For additional context about technologies to be used, project structure, shell commands, and other important information, read the current plan: -`docs/specs/083-tool-acquisition-guidance/plan.md` +`docs/specs/087-harness-first-product/plan.md` + +## Learned User Preferences + +- Ask clarifying questions before drafting implementation plans when harness targets, Go role, or MVP surface are still open. +- Prioritize user-facing landscape navigation (hotspots, tech debt, duplication, where to look first) over evidence-discipline artifacts as the primary deliverable. +- Treat B2B evidence guardrails as a secondary layer on top of the map, not the main reason to use Portolan. +- Expand harness rules, guardrails, and OSS tool recipes rather than the Go codebase when adding product behavior. + +## Learned Workspace Facts + +- Understand-Anything is the reference UX for the local orient map; graph nodes must stay evidence-backed, not LLM-authored truth. +- Product-success questions center on concrete code pain (debt, duplication, risky zones), not abstract trust infrastructure alone. diff --git a/README.md b/README.md index 590bc2ef..825fd087 100644 --- a/README.md +++ b/README.md @@ -31,9 +31,31 @@ Use Portolan when you want an agent to answer questions like: Portolan is especially useful when the target is messy, multi-repo, legacy, or partly black-box. -## What You Get +## Harness-First Quick Start (recommended) -The main workflow creates a context pack for an agent: +Portolan is primarily a **harness supplement**: portable skill, OSS recipes, +guardrails, and a local orient map — not a Go module you must install first. + +**One command** (tool check → recipes → bundle → viewer): + +```bash +scripts/orient-wizard.sh --yes +``` + +Add `--no-viewer` to build only. See `scripts/orient-wizard.sh --help`. + +The orient viewer supports search, filters, a directory heat map, and click-to-source +preview (local files only). See [`docs/demo-runbook.md`](docs/demo-runbook.md). + +Manual fallback: read [`harness/SKILL.md`](harness/SKILL.md), run recipes from +[`harness/recipes/`](harness/recipes/), then `scripts/build-orient-bundle.sh`. + +See [`docs/harness/GO-FREEZE-POLICY.md`](docs/harness/GO-FREEZE-POLICY.md) for +legacy Go CLI status. + +## What You Get (legacy Go path) + +The Go CLI workflow creates a context pack for an agent: ```bash portolan context prepare --root --out /context --profile agent @@ -277,5 +299,7 @@ For repository development: ```bash go test -count=1 ./... jq empty schema/*.json +jq empty harness/contracts/orient-bundle.schema.json +scripts/harness-orient-smoke.sh git diff --check ``` diff --git a/docs/agent/INSTALL-PROMPT.md b/docs/agent/INSTALL-PROMPT.md index 5c62c960..1270cb0a 100644 --- a/docs/agent/INSTALL-PROMPT.md +++ b/docs/agent/INSTALL-PROMPT.md @@ -1,84 +1,59 @@ # Agent Install Prompt -Use this prompt when you want an AI agent to install and run Portolan without -hidden scaffolding. +Use this prompt when you want an AI agent to run Portolan without hidden +scaffolding. -Copy the prompt block below to the receiving agent. If an agent receives this -whole file instead of only the block, it should execute the prompt block after -the three variables are filled in, not ask what to do next. +**Recommended (harness-first):** see [`harness/SKILL.md`](../../harness/SKILL.md) +and harness-specific prompts: -Replace the three variables with absolute local paths: +- [`harness/opencode/INSTALL-PROMPT.md`](../../harness/opencode/INSTALL-PROMPT.md) +- [`harness/codex-claude/INSTALL-PROMPT.md`](../../harness/codex-claude/INSTALL-PROMPT.md) + +Replace variables with absolute local paths: ```text -PORTOLAN_PATH= -TARGET_PATH= -OUTPUT_PATH= +PORTOLAN_PATH= +TARGET_PATH= +ORIENT_PATH= ``` -For OpenCode default-permission runs, prefer an `OUTPUT_PATH` inside the -Portolan checkout, for example -`/.portolan/runs/`. The recorded OpenCode -external-output default-permission lane failed when the harness rejected the -external output path. - Then send: ```text -Install and use Portolan for a local read-only codebase navigation pass. -Execute these steps now and report the result. Do not ask whether to execute -unless a required local path is missing. - -Inputs: -- PORTOLAN_PATH= -- TARGET_PATH= -- OUTPUT_PATH= - -Rules: -- Use only these local paths. -- Do not use network access, credentials, cloning, daemons, or target mutation - unless I explicitly approve it. -- If your harness rejects writes to OUTPUT_PATH, fall back once to a repo-local - `.portolan/runs/` directory under the Portolan checkout and - record the original OUTPUT_PATH write as `failed`. Use that fallback - directory as OUTPUT_PATH for the remaining steps. -- If you are using OpenCode with default permissions, prefer that repo-local - `.portolan/runs/` output path before attempting an external - output path. -- If PORTOLAN_PATH is a binary, verify it with `--version`. -- If PORTOLAN_PATH is a source checkout, follow `docs/agent/INSTALL.md` and - build the repo-local binary with `scripts/bootstrap-portolan`. -- Prepare context into `OUTPUT_PATH/context`. -- Build a map into `OUTPUT_PATH/map` when the target size is reasonable. -- If `TARGET_PATH/selection.json` exists, validate it and prefer - `map --selection TARGET_PATH/selection.json --out OUTPUT_PATH/map` for the - map step. Otherwise use `map --root TARGET_PATH --out OUTPUT_PATH/map`. -- If selection validation fails, record that command as `failed`, then fall - back to `map --root TARGET_PATH --out OUTPUT_PATH/map`. -- Read bounded artifacts before opening large graph files: - - `context/agent-brief.md` - - `context/answer-contract.md` - - `context/evidence-index.jsonl` - - `context/gaps.jsonl` - - `map/summary.json` - - `map/graph-index.json` - - `map/findings.jsonl` - - `map/map.md` -- Preserve `verified`, `failed`, `blocked`, `not_assessed`, `unknown`, and - `cannot_verify`. -- Cite local artifact paths for every material claim. -- Do not claim complete estate coverage, runtime topology, OSS scanner value, - or architecture facts unless the local Portolan artifacts prove them. +Run the Portolan orient harness on TARGET_PATH. Write the orient bundle to +ORIENT_PATH. Execute now; do not ask unless a path is missing. + +1. Read PORTOLAN_PATH/harness/SKILL.md +2. PORTOLAN_PATH/scripts/orient-wizard.sh TARGET_PATH ORIENT_PATH --no-viewer --yes + (or run recipes manually + build-orient-bundle.sh if operator prefers) +3. Read hotspots.jsonl and gaps.jsonl; cite hotspot.id and producer_ref per + PORTOLAN_PATH/harness/guardrails/citation-rules.md + +If harness blocks external writes, use +PORTOLAN_PATH/.portolan/runs/ as ORIENT_PATH. Answer with: -1. Commands run and whether each was `verified`, `failed`, or `blocked`. -2. Artifact paths created. -3. Visible local scope and completeness limits. -4. Evidence-backed relationships, duplication, configuration surfaces, and - technical-debt candidates. -5. Explicit `unknown`, `cannot_verify`, and `not_assessed` surfaces. -6. Three useful next local actions. -7. Any unsupported claims you avoided or accidentally made. +1. Recipes run (verified/failed/blocked) +2. Top 5 hotspots by rank with evidence +3. Gaps and not_assessed surfaces +4. Viewer command: cd PORTOLAN_PATH/viewer && npm run serve -- --bundle ORIENT_PATH +5. Unsupported claims avoided + +Legacy Go CLI (optional): docs/harness/GO-FREEZE-POLICY.md ``` -For Russian-language agent runs, use -[`docs/agent/INSTALL-PROMPT.ru.md`](INSTALL-PROMPT.ru.md). +## Legacy Go install prompt + +Use only when the operator explicitly needs `context prepare` / `map`: + +```text +PORTOLAN_PATH= +TARGET_PATH= +OUTPUT_PATH= +``` + +Follow [`docs/agent/INSTALL.md`](INSTALL.md), bootstrap Go binary if needed, +run `context prepare` and `map` into OUTPUT_PATH. See git history or +`docs/product-claims.md` for artifact order. + +For Russian-language runs, use [`INSTALL-PROMPT.ru.md`](INSTALL-PROMPT.ru.md). diff --git a/docs/onboarding.md b/docs/onboarding.md index 3474ae20..9ad088c5 100644 --- a/docs/onboarding.md +++ b/docs/onboarding.md @@ -9,7 +9,8 @@ first. | Intent | Start with | Then read | Boundary to preserve | | --- | --- | --- | --- | | Understand what Portolan is for | [README](../README.md) | [Product Claims](product-claims.md), [Product Boundary](product-boundary.md), [Product Quality Boundary](product-quality-boundary.md) | Portolan is a local evidence layer, not a coding harness, readiness gate, service catalog, observability platform, or Cursor/OpenCode replacement. | -| Ask an agent to inspect a local target | [Agent Quickstart](agent/QUICKSTART.md) | [Install Prompt](agent/INSTALL-PROMPT.md), [Agent Acceptance](agent/ACCEPTANCE.md) | Agent answers must cite local artifacts and preserve `unknown`, `cannot_verify`, and `not_assessed`. | +| Ask an agent to inspect a local target | [Harness Skill](../harness/SKILL.md) | [Install Prompt](agent/INSTALL-PROMPT.md), [Agent Acceptance](agent/ACCEPTANCE.md) | Harness-first: recipes → orient bundle → viewer. Cite hotspot.id and preserve gaps. | +| Open the orient map (human) | [Viewer README](../viewer/README.md) | [Harness Skill](../harness/SKILL.md) | Local read-only viewer; evidence from producers only. | | Install or resolve the command | [Agent Install](agent/INSTALL.md) | [Troubleshooting](agent/TROUBLESHOOTING.md), [Release Guide](release.md) | Source bootstrap is local and does not fetch Go modules unless explicitly approved. | | Use Cursor | [Agent Install Prompt](agent/INSTALL-PROMPT.md) | [Agent Acceptance](agent/ACCEPTANCE.md), [Product Claims](product-claims.md) | Current verified Cursor evidence is headless Cursor Agent CLI / Composer. Cursor UI behavior is outside the current required acceptance scope; no root-level Cursor rule is shipped. | | Use OpenCode | [Install Prompt](agent/INSTALL-PROMPT.md) | [Agent Acceptance](agent/ACCEPTANCE.md), [Product Claims](product-claims.md) | OpenCode default-permission runs are verified with repo-local output under the Portolan checkout. The recorded external-output default-permission lane failed. | diff --git a/docs/product-backlog.md b/docs/product-backlog.md index 91006bc3..f22349d7 100644 --- a/docs/product-backlog.md +++ b/docs/product-backlog.md @@ -157,7 +157,7 @@ fixtures are preflight evidence only. | P6-073 | `docs/specs/073-bigtop-runtime-capture-execution/` | The explicitly approved single-node Bigtop Docker provisioner create/capture/destroy run is executed and ledgered with runtime-visible service evidence, cleanup evidence, and claim boundaries. | Merged via PR #51; approved runtime create/capture/destroy executed locally. Container, Docker network, inspect output, and one running YARN NodeManager were verified as `runtime-visible`; NameNode, ResourceManager, HistoryServer, and ProxyServer failed, Datanode was not installed/skipped, cleanup removed the container/network and target repo residue; Cursor stress, three assessed non-GPT review lanes, local baseline, GitHub checks, squash merge, and branch cleanup verified; full Bigtop runtime topology and enterprise parity remain `cannot_verify`; GitHub review approval remains `not_assessed` | | P6-074 | `docs/specs/074-bigtop-runtime-topology-health-capture/` | The partial Bigtop runtime capture is converted into a health-oriented topology proof attempt with explicit per-service assertions, daemon logs, smoke probes, cleanup, and failure classification. | Merged via PR #52; depends on P6-073. Planning, approval packet, health command contract, runtime summary schema, Cursor scope stress, three assessed non-GPT review lanes, local baseline, GitHub checks, squash merge, and branch cleanup verified. Runtime execution remains blocked pending fresh explicit approval for the named 074 command sequence; complete topology remains `cannot_verify` until this slice produces service-health and smoke-probe evidence | | P6-075 | `docs/specs/075-bigtop-producer-output-coverage-closure/` | Real Bigtop symbol/API/catalog/model/runtime producer outputs beyond Syft/CycloneDX are inventoried, refreshed, normalized or ledgered, and coverage-scored against the architecture rubric. | Merged via PR #53; producer-output breadth beyond Syft/CycloneDX is verified only as confirmed bounded outputs with source ledgers, Cursor stress, three assessed non-GPT review lanes, local baseline, GitHub checks, squash merge, and remote branch cleanup verified. Complete runtime topology, full symbol/reference graph, call graph, and human/enterprise parity remain `cannot_verify`; follow-up specs 074, 077, and 076 own those gaps | -| P6-076 | `docs/specs/076-cursor-enterprise-parity-validation/` | Cursor plus Portolan is re-evaluated against the Bigtop human/enterprise architecture rubric after runtime and producer-output closure, with claim promotion allowed only for criteria proven by current evidence. | Planning gate merged via PR #55; depends on P6-074, P6-075, and P6-077. Concrete spec/plan/tasks, execution gate, artifact hygiene ledger, three assessed planning lanes, three assessed PR lanes, local baseline, GitHub checks, explicit user merge approval, squash merge, and remote branch cleanup verified; GitHub review approval remains `not_assessed`. Default paired Cursor stress is blocked until P6-074 runtime-health evidence exists; a current-evidence rejection run needs explicit approval and must keep broad human/enterprise parity `cannot_verify` | +| P6-076 | `docs/specs/076-cursor-enterprise-parity-validation/` | Cursor plus Portolan is re-evaluated against the Bigtop human/enterprise architecture rubric after runtime and producer-output closure, with claim promotion allowed only for criteria proven by current evidence. | Planning gate merged via PR #55; **frozen from P7 harness MVP path** until orient harness ships on real targets. Default paired Cursor stress blocked until P6-074 runtime-health evidence exists | | P6-077 | `docs/specs/077-bigtop-callgraph-symbol-closure/` | Full symbol/reference/call graph closure is attempted with mature local producers or explicitly classified as `cannot_verify` with reviewed rationale. | Merged via PR #54; depends on P6-075. Producer decision record, read-only availability ledger, Cursor Composer 2.5 claim-boundary stress, three assessed non-GPT review lanes, local baseline, PR creation, GitHub checks, explicit user merge approval, squash merge, and remote branch cleanup verified; no safe full resolved graph producer is currently available, so full C6 and call-graph parity remain `cannot_verify`; GitHub review approval remains `not_assessed` | | P6-078 | `docs/specs/078-build-tool-dependency-producers/` | Context packs recommend approval-gated native Maven/Gradle dependency producer outputs when build manifests are visible, so Java/Scala/Maven relationship gaps get concrete local evidence steps without Portolan owning per-language scanners. | Merged via PR #56; local baseline, fresh Bigtop context smoke, three assessed non-GPT review lanes, GitHub checks, explicit user merge approval, squash merge, and remote branch cleanup verified. No Maven/Gradle command was executed by default, no network/install/build was approved, and dependency evidence remains `not_assessed` until local CycloneDX/build-tool output is supplied; GitHub review approval remains `not_assessed` | | P6-079 | `docs/specs/079-jscpd-sharded-duplication-plan/` | Context packs recommend repository-sharded jscpd output recipes for large multi-repo landscapes so duplication evidence can be acquired incrementally without treating full-root jscpd OOM as evidence. | Merged via PR #57; local baseline, fresh Bigtop context smoke, Cursor Composer 2.5 sharded-plan stress, three assessed non-GPT review lanes, GitHub checks, explicit user merge approval, squash merge `f4a4951`, and remote branch cleanup verified. Scope is `oss-plan.json` guidance and focused tests only; Portolan did not execute jscpd, did not install stores/plugins, and duplication evidence remains `not_assessed` until local jscpd output exists; GitHub review approval remains `not_assessed` | @@ -165,9 +165,26 @@ fixtures are preflight evidence only. | P6-081 | `docs/specs/081-maven-sharded-producer-plan/` | Context packs emit repository-sharded Maven/CycloneDX next actions for multi-repo JVM landscapes so agents do not treat one sample `pom.xml` as a landscape rollout plan. | Merged via PR #59; local baseline, fresh Bigtop context smoke, Cursor Composer 2.5 stress, three assessed non-GPT review lanes, GitHub checks, explicit user merge approval, squash merge `a89a965`, and remote branch cleanup verified. Maven execution, dependency evidence, JVM relationship claims, and GitHub review approval remain `not_assessed` | | P6-082 | `docs/specs/082-syft-sharded-sbom-plan/` | Context packs emit repository-sharded Syft/CycloneDX SBOM next actions for multi-repo landscapes so component/dependency evidence can be acquired incrementally without full-root SBOM scans. | Merged via PR #60; local baseline, fresh Bigtop context smoke, Cursor Composer 2.5 stress, three assessed non-GPT review lanes, GitHub checks, explicit user merge approval, squash merge `9390379`, and remote branch cleanup verified. Syft execution, component inventory, dependency evidence, and GitHub review approval remain `not_assessed` | | P6-083 | `docs/specs/083-tool-acquisition-guidance/` | Context packs make tool acquisition guidance explicit and stack-agnostic: agents can pull in the right local producer tools without treating Portolan as a PHP/JVM/Gradle adapter stack. | Merged via PR #61; local baseline, fresh Bigtop context smoke, Cursor Composer 2.5 stress, integrated PR #57-#61 stack-agnostic stress, three assessed non-GPT review lanes, GitHub checks, explicit user merge approval, squash merge `847e84e`, post-merge Bigtop context smoke, and remote branch cleanup verified. Native producer execution, tool install/acquisition, component inventory, dependency relationships, duplication metrics, runtime topology, and GitHub review approval remain `not_assessed` | -| P6-084 | `docs/specs/084-external-tool-evaluation-profiles/` | Portolan keeps dated evaluation profiles for CodeGraph, Understand-Anything, and ast-index so external tool adoption stays explicit, evidence-honest, and refreshable. | Draft spec; backlog-only. No plan/tasks yet. External tool metadata, profile implementation, and generated guidance remain `not_assessed` until planning and verification refresh the 2026-06-04 review snapshot | +| P6-084 | `docs/specs/084-external-tool-evaluation-profiles/` | Portolan keeps dated evaluation profiles for CodeGraph, Understand-Anything, and ast-index so external tool adoption stays explicit, evidence-honest, and refreshable. | Implemented under P7-084; profiles in `docs/harness/tool-profiles/` | | P6-085 | `docs/specs/085-ast-index-producer-import/` | Explicit local ast-index outputs can be imported as bounded symbol/reference/module producer evidence without Portolan installing, executing, watching, or mutating targets. | Draft spec; backlog-only. No plan/tasks yet. ast-index execution, real output acquisition, importer implementation, CodeGraph import, and call-graph parity remain `not_assessed` | -| P6-086 | `docs/specs/086-evidence-navigation-ux-patterns/` | Useful navigation ideas from Understand-Anything, CodeGraph, and ast-index are adopted as evidence-backed guide patterns without accepting LLM-authored graphs or live dashboards as truth. | Draft spec; backlog-only. No plan/tasks yet. UX implementation, dashboard/MCP surfaces, LLM workflows, and agent acceptance impact remain `not_assessed` | +| P6-086 | `docs/specs/086-evidence-navigation-ux-patterns/` | Useful navigation ideas from Understand-Anything, CodeGraph, and ast-index are adopted as evidence-backed guide patterns without accepting LLM-authored graphs or live dashboards as truth. | MVP viewer in `viewer/`; harness-orient-smoke verified; full UA fork deferred | + +## P7: Harness-First Orient (active) + +| ID | Spec | Outcome | Status | +| --- | --- | --- | --- | +| P7-087 | `docs/specs/087-harness-first-product/` | Product narrative, constitution 1.1.0, Go freeze policy, harness-first README route. | Implemented (harness pivot slice) | +| P7-084 | `docs/specs/084-external-tool-evaluation-profiles/` | Dated tool profiles and first-wave recipes for jscpd, Semgrep, Syft, ctags, UA, CodeGraph, ast-index. | Implemented (profiles + recipes) | +| P7-088 | `docs/specs/088-orient-bundle-contract/` | `orient/` bundle contract, build script, map bridge. | Implemented | +| P7-089 | `docs/specs/089-orient-wizard/` | One-command `orient-wizard.sh`: tool check, consent install, recipes, bundle, viewer; real-target smoke. | Implemented | +| P7-090 | `docs/specs/090-viewer-demo-ux/` | Demo-ready viewer: search, filters, directory heat map, source preview, demo runbook. | Implemented | +| P7-091 | `docs/specs/091-landscape-scale/` | Full bigtop scale: bounded jscpd shards, shard gaps, kind-quota budget, hotspots-full.jsonl. | Implemented | +| P7-086 | `docs/specs/086-evidence-navigation-ux-patterns/` | Local orient viewer (UA-inspired) over evidence hotspots. | Demo-ready in `viewer/` (spec 090) | +| P7-ADR | `docs/adr/001-go-cli-fate.md` | Go CLI maintenance-mode decision after harness smoke. | Provisional ADR recorded | + +**MVP path:** `scripts/orient-wizard.sh` (or `harness/SKILL.md` → recipes → `build-orient-bundle.sh`) → `viewer/`. + +**Frozen from P7 MVP:** Bigtop enterprise parity (076), runtime capture chain (061–065), new Go `contextprep` features. ## Backlog Rules diff --git a/docs/product-claims.md b/docs/product-claims.md index 30575e3a..dffc6bb0 100644 --- a/docs/product-claims.md +++ b/docs/product-claims.md @@ -14,16 +14,18 @@ surface derived from that evidence. ## Current Client-Safe Answer -Portolan is useful when an agent needs a bounded local evidence pack before it -answers architecture or estate questions. The product story is target-agnostic: -run Portolan against a local target, read bounded artifacts first, preserve -weak evidence states, and cite local evidence for every material claim. - -The practical value is evidence discipline. Portolan prepares local context, -map, graph, finding, and answer-contract artifacts that tell an agent what is -visible, what is only declared, what is missing, and what remains unknown. That -makes an agent less likely to turn local files, partial scans, or planned OSS -integrations into broader claims. +Portolan helps engineers and agents find where local code pain is visible: +duplication clusters, static findings, dependency hubs, and configuration +hotspots — then navigate there through a harness workflow and local orient map. + +The primary product path is harness-first: follow [`harness/SKILL.md`](../harness/SKILL.md), +run documented OSS recipes, build an `orient/` bundle, and open the local viewer. +The legacy Go CLI (`context prepare`, `map`) remains an optional bridge. + +Evidence discipline is a **secondary** (B2B) value: hotspots and viewer nodes +cite `producer_ref` and evidence states; unknowns stay visible without blocking +first-run navigation. Portolan does not replace mature scanners — it composes +their local outputs into one orient surface. Named examples, including the Apache Bigtop runs, are evidence records for specific target shapes. They are not the main product path and must not be From 27800c29161a2bbcb5687e7b35f23ea8fa8af0ad Mon Sep 17 00:00:00 2001 From: Andrey Zhukov Date: Wed, 10 Jun 2026 18:33:34 +0300 Subject: [PATCH 06/12] Fix PR #64 review findings: path safety, schema conformance, gap budget Address symlink escape on /source, dep-hub sentinel paths, shard gap status enum, gaps top-20 cap, wizard flag validation, semgrep fail-closed, and smoke schema/symlink checks. Record PR review dispositions for specs 087-091. Co-authored-by: Cursor --- .../pr-review-code-replacement-2026-06-10.md | 29 +++++++++ .../pr-review-disposition-2026-06-10.md | 51 ++++++++++++++++ ...iew-requirements-replacement-2026-06-10.md | 25 ++++++++ ...-review-security-replacement-2026-06-10.md | 24 ++++++++ scripts/build-orient-bundle.sh | 35 ++++++++++- scripts/harness-orient-smoke.sh | 34 +++++++++++ scripts/orient-wizard.sh | 60 +++++++++++++++---- viewer/scripts/serve.js | 42 ++++++++++--- viewer/src/app.js | 8 ++- 9 files changed, 284 insertions(+), 24 deletions(-) create mode 100644 docs/specs/091-landscape-scale/reviews/pr-review-code-replacement-2026-06-10.md create mode 100644 docs/specs/091-landscape-scale/reviews/pr-review-disposition-2026-06-10.md create mode 100644 docs/specs/091-landscape-scale/reviews/pr-review-requirements-replacement-2026-06-10.md create mode 100644 docs/specs/091-landscape-scale/reviews/pr-review-security-replacement-2026-06-10.md diff --git a/docs/specs/091-landscape-scale/reviews/pr-review-code-replacement-2026-06-10.md b/docs/specs/091-landscape-scale/reviews/pr-review-code-replacement-2026-06-10.md new file mode 100644 index 00000000..64ded711 --- /dev/null +++ b/docs/specs/091-landscape-scale/reviews/pr-review-code-replacement-2026-06-10.md @@ -0,0 +1,29 @@ +# PR #64 — code lane (replacement) + +- **Lane**: code-reviewer +- **Requested model**: `kimi-for-coding/k2p6` +- **Actual harness**: `ce-maintainability-reviewer` (codex-subagent crashed; opencode timeout) +- **Status**: assessed (replacement) +- **Verdict**: partial → fixes applied + +## Accepted findings (fixed) + +| Severity | File | Fix | +| --- | --- | --- | +| P1 | `build-orient-bundle.sh:137` | dep-hub sentinel path | +| P1 | `orient-wizard.sh:45` | `require_opt_value` for value flags | +| P2 | `orient-wizard.sh:90` | shard gap status `cannot_verify` | +| P2 | `build-orient-bundle.sh:173` | integer validation for budgets | +| P2 | `viewer/src/app.js:104` | dep-hub bypasses repo filter | +| P2 | `build-orient-bundle.sh:222` | gap budget cap | +| P3 | `orient-wizard.sh:404` | node check only when viewer starts | + +## Deferred (not blocking MVP) + +- repo_slug basename collision (multi-repo same name) +- duplicated discover_repos logic +- severity-rank duplication across jq/viewer +- source preview line metadata +- port parseInt validation +- smoke sleep → retry poll +- install_tool eval pattern (hardcoded cmds only today) diff --git a/docs/specs/091-landscape-scale/reviews/pr-review-disposition-2026-06-10.md b/docs/specs/091-landscape-scale/reviews/pr-review-disposition-2026-06-10.md new file mode 100644 index 00000000..2ce1a4e0 --- /dev/null +++ b/docs/specs/091-landscape-scale/reviews/pr-review-disposition-2026-06-10.md @@ -0,0 +1,51 @@ +# PR #64 review disposition — 2026-06-10 + +- **PR**: https://github.com/fcon-tech/portolan/pull/64 +- **Branch**: `codex/087-091-harness-pivot` +- **Specs**: 087–091 (umbrella ship) + +## Review lanes + +| Lane | Requested model | Status | Verdict | +| --- | --- | --- | --- | +| requirements | `zai-coding-plan/glm-5.1` | `not_assessed` (codex-subagent/opencode hung) | — | +| requirements (replacement) | `ce-correctness-reviewer` | assessed | partial → fixed | +| code | `kimi-for-coding/k2p6` | `not_assessed` (opencode timeout) | — | +| code (replacement) | `ce-maintainability-reviewer` | assessed | partial → fixed | +| security | `minimax/MiniMax-M2.7` | `not_assessed` (codex-subagent crash) | — | +| security (replacement) | `ce-security-reviewer` | assessed | partial → P0 fixed | +| repo-grounded local | this session | assessed | fixes verified | + +**Assessed independent non-GPT coverage**: 3 replacement lanes + local verification (requested OpenCode lanes degraded). + +## Fixes applied (review-fix pass) + +1. Symlink escape on viewer `/source` (SEC-001) +2. dep-hub schema conformance: sentinel path (REQ-002) +3. Shard gaps use `cannot_verify` status (REQ-003) +4. Gap budget top-20 with manifest `gaps_truncated` (REQ-001) +5. Wizard flag arity validation + numeric budget checks +6. Semgrep fail-closed without local rules (REQ-007) +7. dep-hub visible under repo filters +8. Smoke: record schema checks + symlink 403 test + +## Deferred (documented, not blocking draft→ready) + +- Extended viewer UX smoke (filters, heat tree, banners) — REQ-005 +- Full bigtop / wizard orchestration in CI — REQ-006 +- `hotspots-full.jsonl` in spec 088 layout — REQ-008 +- repo_slug collision on duplicate basenames +- `eval` in install_tool (hardcoded only) + +## Verification (post-fix) + +```text +scripts/harness-orient-smoke.sh — ok +go test ./... — ok +``` + +GitHub CI on PR head: re-run after push (prior run green on pre-fix head). + +## Readiness + +`/speckit-pr-readiness-closeout` may run after CI on fix commit is green. PR remains **draft** until closeout passes. diff --git a/docs/specs/091-landscape-scale/reviews/pr-review-requirements-replacement-2026-06-10.md b/docs/specs/091-landscape-scale/reviews/pr-review-requirements-replacement-2026-06-10.md new file mode 100644 index 00000000..3593ac98 --- /dev/null +++ b/docs/specs/091-landscape-scale/reviews/pr-review-requirements-replacement-2026-06-10.md @@ -0,0 +1,25 @@ +# PR #64 — requirements lane (replacement) + +- **Lane**: requirements-reviewer +- **Requested model**: `zai-coding-plan/glm-5.1` +- **Actual harness**: `ce-correctness-reviewer` (codex-subagent/opencode hung; empty output after 600s) +- **Status**: assessed (replacement) +- **Verdict**: partial + +## Accepted findings (fixed in review-fix pass) + +| ID | Severity | Fix | +| --- | --- | --- | +| REQ-001 | major | `build-orient-bundle.sh`: `ORIENT_GAP_BUDGET` default 20, sort + cap gaps | +| REQ-002 | major | dep-hub `paths: ["(dependency-hub)"]` | +| REQ-003 | major | shard gaps map to `cannot_verify` (schema enum) | +| REQ-004 | major | `harness-orient-smoke.sh` validates hotspot/gap records | +| REQ-007 | minor | semgrep: no `p/default` fallback; gap when local rules missing | + +## Rejected / deferred + +| ID | Disposition | Reason | +| --- | --- | --- | +| REQ-005 | deferred | viewer filter/heat/banner smoke — manual demo evidence sufficient for MVP; follow-up spec | +| REQ-006 | accepted risk | full bigtop remains manual gate (documented in scale-findings) | +| REQ-008 | deferred | doc-only: add `hotspots-full.jsonl` to spec 088 layout in follow-up | diff --git a/docs/specs/091-landscape-scale/reviews/pr-review-security-replacement-2026-06-10.md b/docs/specs/091-landscape-scale/reviews/pr-review-security-replacement-2026-06-10.md new file mode 100644 index 00000000..e80d2d4b --- /dev/null +++ b/docs/specs/091-landscape-scale/reviews/pr-review-security-replacement-2026-06-10.md @@ -0,0 +1,24 @@ +# PR #64 — security lane (replacement) + +- **Lane**: security-reviewer +- **Requested model**: `minimax/MiniMax-M2.7` +- **Actual harness**: `ce-security-reviewer` (codex-subagent background runs died without result) +- **Status**: assessed (replacement) +- **Verdict**: partial → P0 fixed + +## Accepted findings (fixed) + +| ID | Severity | Fix | +| --- | --- | --- | +| SEC-001 | P0 | `/source`: `lstatSync` rejects symlinks; `realpathSync` re-check under repo roots | +| SEC-002 | P1 | static dist guard uses `path.resolve` + trailing separator | +| SEC-003 | P1 | accepted risk: repos.json trusted (local bundle operator model) | + +## Not exploitable / accepted + +- SEC-004: producer paths bounded by serve guard +- SEC-005/006/007: eval/curl|sh only on operator consent; quoted repo paths + +## Test added + +- `harness-orient-smoke.sh`: symlink `leak-outside` → `/etc/passwd` returns 403 diff --git a/scripts/build-orient-bundle.sh b/scripts/build-orient-bundle.sh index 00bb007e..32925013 100755 --- a/scripts/build-orient-bundle.sh +++ b/scripts/build-orient-bundle.sh @@ -12,6 +12,15 @@ TARGET_ROOT=$(cd "$1" && pwd) ORIENT_DIR=$2 PRODUCERS_DIR="$ORIENT_DIR/producers" HOTSPOT_BUDGET="${ORIENT_HOTSPOT_BUDGET:-200}" +GAP_BUDGET="${ORIENT_GAP_BUDGET:-20}" +if ! [[ "$HOTSPOT_BUDGET" =~ ^[0-9]+$ ]] || [[ "$HOTSPOT_BUDGET" -lt 1 ]]; then + echo "invalid ORIENT_HOTSPOT_BUDGET: $HOTSPOT_BUDGET" >&2 + exit 2 +fi +if ! [[ "$GAP_BUDGET" =~ ^[0-9]+$ ]] || [[ "$GAP_BUDGET" -lt 1 ]]; then + echo "invalid ORIENT_GAP_BUDGET: $GAP_BUDGET" >&2 + exit 2 +fi mkdir -p "$ORIENT_DIR" "$PRODUCERS_DIR" command -v jq >/dev/null 2>&1 || { @@ -134,7 +143,7 @@ while IFS= read -r sbom; do jq -nc \ --arg id "$id" --arg summary "Dependency hub: $name ($dep_count dependencies)" \ --arg ref "$ref" \ - '{id:$id,kind:"dep-hub",severity:"low",summary:$summary,paths:[],evidence_state:"metadata-visible",producer:"syft",producer_ref:$ref}' >>"$hotspots_raw" + '{id:$id,kind:"dep-hub",severity:"low",summary:$summary,paths:["(dependency-hub)"],evidence_state:"metadata-visible",producer:"syft",producer_ref:$ref}' >>"$hotspots_raw" done done < <(find "$PRODUCERS_DIR/syft" -type f \( -name 'cyclonedx.json' -o -name '*cyclonedx*.json' \) 2>/dev/null) @@ -220,8 +229,26 @@ kind_counts_total=$(jq -s 'group_by(.kind) | map({(.[0].kind): length}) | add // kind_counts=$(jq -s 'group_by(.kind) | map({(.[0].kind): length}) | add // {}' "$ORIENT_DIR/hotspots.jsonl" 2>/dev/null || echo '{}') : >"$ORIENT_DIR/gaps.jsonl" +gaps_truncated=0 if [[ -s "$gaps_raw" ]]; then - cat "$gaps_raw" >>"$ORIENT_DIR/gaps.jsonl" + gap_sorted=$(mktemp) + jq -sc ' + sort_by( + (if .status == "cannot_verify" then 0 + elif .status == "not_assessed" then 1 + else 2 end), + .surface, + .summary + ) | .[] + ' "$gaps_raw" >"$gap_sorted" + gap_total=$(wc -l <"$gap_sorted" | tr -d ' ') + if [[ "$gap_total" -gt "$GAP_BUDGET" ]]; then + head -n "$GAP_BUDGET" "$gap_sorted" >>"$ORIENT_DIR/gaps.jsonl" + gaps_truncated=1 + else + cat "$gap_sorted" >>"$ORIENT_DIR/gaps.jsonl" + fi + rm -f "$gap_sorted" fi gap_count=$(wc -l <"$ORIENT_DIR/gaps.jsonl" | tr -d ' ') @@ -236,7 +263,9 @@ jq -n \ --argjson hotspots_total "$total_before" \ --argjson kind_counts "$kind_counts" \ --argjson kind_counts_total "$kind_counts_total" \ - '{schema_version:$schema_version,target_root:$target_root,generated_at:$generated_at,hotspot_count:$hotspot_count,gap_count:$gap_count,hotspot_budget:$hotspot_budget,hotspots_truncated:$hotspots_truncated,hotspots_total:$hotspots_total,kind_counts:$kind_counts,kind_counts_total:$kind_counts_total}' \ + --argjson gap_budget "$GAP_BUDGET" \ + --argjson gaps_truncated "$gaps_truncated" \ + '{schema_version:$schema_version,target_root:$target_root,generated_at:$generated_at,hotspot_count:$hotspot_count,gap_count:$gap_count,hotspot_budget:$hotspot_budget,hotspots_truncated:$hotspots_truncated,hotspots_total:$hotspots_total,kind_counts:$kind_counts,kind_counts_total:$kind_counts_total,gap_budget:$gap_budget,gaps_truncated:$gaps_truncated}' \ >"$ORIENT_DIR/manifest.json" jq -s '{schema_version:"0.1.0",nodes:[.[]|{id:.id,label:.summary,kind:.kind,paths:(.paths//[])}],edges:[]}' \ diff --git a/scripts/harness-orient-smoke.sh b/scripts/harness-orient-smoke.sh index a16f3663..b020cd7a 100755 --- a/scripts/harness-orient-smoke.sh +++ b/scripts/harness-orient-smoke.sh @@ -16,6 +16,33 @@ cp -a "$ROOT/internal/testfixtures/orient-bundle/producers/." "$FIXTURE_ORIENT/p test "$(wc -l <"$FIXTURE_ORIENT/hotspots.jsonl" | tr -d ' ')" -ge 1 test -f "$FIXTURE_ORIENT/manifest.json" +validate_hotspot_line() { + jq -e ' + .id and .kind and .severity and .summary and + (.paths | type == "array" and length >= 1) and + .evidence_state and .producer and .producer_ref and (.rank | type == "number") + ' >/dev/null +} + +validate_gap_line() { + jq -e ' + .id and .surface and .summary and + (.status | IN("unknown", "cannot_verify", "not_assessed")) + ' >/dev/null +} + +while IFS= read -r line; do + [[ -z "$line" ]] && continue + echo "$line" | validate_hotspot_line || { echo "invalid hotspot record: $line" >&2; exit 1; } +done <"$FIXTURE_ORIENT/hotspots.jsonl" + +if [[ -s "$FIXTURE_ORIENT/gaps.jsonl" ]]; then + while IFS= read -r line; do + [[ -z "$line" ]] && continue + echo "$line" | validate_gap_line || { echo "invalid gap record: $line" >&2; exit 1; } + done <"$FIXTURE_ORIENT/gaps.jsonl" +fi + cd "$ROOT/viewer" node scripts/build-static.js node scripts/serve.js --bundle "$FIXTURE_ORIENT" --port "$VIEWER_PORT" & @@ -32,4 +59,11 @@ curl -sf "$BASE/source?path=sample.go&line=1" | grep -q 'Run' FORBIDDEN_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/source?path=../../../etc/passwd&line=1") test "$FORBIDDEN_CODE" = "403" +SYMLINK_TARGET="$FIXTURE_TARGET/leak-outside" +rm -f "$SYMLINK_TARGET" +ln -sf /etc/passwd "$SYMLINK_TARGET" +SYMLINK_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/source?path=leak-outside&line=1") +rm -f "$SYMLINK_TARGET" +test "$SYMLINK_CODE" = "403" + echo "harness-orient-smoke: ok" diff --git a/scripts/orient-wizard.sh b/scripts/orient-wizard.sh index f49a4adb..c05b60fb 100755 --- a/scripts/orient-wizard.sh +++ b/scripts/orient-wizard.sh @@ -36,18 +36,27 @@ EOF log() { echo "orient-wizard: $*" >&2; } fail_log() { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) $*" >>"$FAILURES_LOG"; } +require_opt_value() { + local flag=$1 val=${2:-} + if [[ -z "$val" || "$val" == -* ]]; then + echo "option $flag requires a value" >&2 + usage >&2 + exit 2 + fi +} + POSITIONAL=() while [[ $# -gt 0 ]]; do case "$1" in --yes) YES=1; shift ;; --skip-install) SKIP_INSTALL=1; shift ;; --no-viewer) NO_VIEWER=1; shift ;; - --port) PORT="$2"; shift 2 ;; - --limit-repos) LIMIT_REPOS="$2"; shift 2 ;; - --producers) PRODUCERS="$2"; shift 2 ;; - --hotspot-budget) HOTSPOT_BUDGET="$2"; shift 2 ;; - --shard-timeout) SHARD_TIMEOUT="$2"; shift 2 ;; - --jscpd-memory-mb) JSCPD_MEMORY_MB="$2"; shift 2 ;; + --port) require_opt_value --port "${2:-}"; PORT="$2"; shift 2 ;; + --limit-repos) require_opt_value --limit-repos "${2:-}"; LIMIT_REPOS="$2"; shift 2 ;; + --producers) require_opt_value --producers "${2:-}"; PRODUCERS="$2"; shift 2 ;; + --hotspot-budget) require_opt_value --hotspot-budget "${2:-}"; HOTSPOT_BUDGET="$2"; shift 2 ;; + --shard-timeout) require_opt_value --shard-timeout "${2:-}"; SHARD_TIMEOUT="$2"; shift 2 ;; + --jscpd-memory-mb) require_opt_value --jscpd-memory-mb "${2:-}"; JSCPD_MEMORY_MB="$2"; shift 2 ;; -h|--help) usage; exit 0 ;; --) shift; POSITIONAL+=("$@"); break ;; -*) echo "unknown option: $1" >&2; usage; exit 2 ;; @@ -60,6 +69,23 @@ if [[ ${#POSITIONAL[@]} -lt 2 ]]; then exit 2 fi +if ! [[ "$HOTSPOT_BUDGET" =~ ^[0-9]+$ ]] || [[ "$HOTSPOT_BUDGET" -lt 1 ]]; then + echo "invalid --hotspot-budget: $HOTSPOT_BUDGET (positive integer required)" >&2 + exit 2 +fi +if ! [[ "$SHARD_TIMEOUT" =~ ^[0-9]+$ ]] || [[ "$SHARD_TIMEOUT" -lt 1 ]]; then + echo "invalid --shard-timeout: $SHARD_TIMEOUT (positive integer required)" >&2 + exit 2 +fi +if ! [[ "$JSCPD_MEMORY_MB" =~ ^[0-9]+$ ]] || [[ "$JSCPD_MEMORY_MB" -lt 256 ]]; then + echo "invalid --jscpd-memory-mb: $JSCPD_MEMORY_MB (integer >= 256 required)" >&2 + exit 2 +fi +if [[ "$LIMIT_REPOS" != 0 ]] && { ! [[ "$LIMIT_REPOS" =~ ^[0-9]+$ ]] || [[ "$LIMIT_REPOS" -lt 1 ]]; }; then + echo "invalid --limit-repos: $LIMIT_REPOS (positive integer required)" >&2 + exit 2 +fi + TARGET_ROOT=$(cd "${POSITIONAL[0]}" && pwd) ORIENT_DIR=${POSITIONAL[1]} PRODUCERS_DIR="$ORIENT_DIR/producers" @@ -79,6 +105,14 @@ append_shard_gap() { '{id:$id,surface:$surface,status:$status,summary:$summary,repo:$repo}' >>"$SHARD_GAPS" } +append_gap_record() { + local id=$1 surface=$2 status=$3 summary=$4 recipe=${5:-} + jq -nc \ + --arg id "$id" --arg surface "$surface" --arg status "$status" \ + --arg summary "$summary" --arg recipe "$recipe" \ + '{id:$id,surface:$surface,status:$status,summary:$summary} + (if $recipe != "" then {recipe:$recipe} else {} end)' >>"$SHARD_GAPS" +} + run_shard() { local producer=$1 repo=$2 shift 2 @@ -87,11 +121,11 @@ run_shard() { if ! timeout "$SHARD_TIMEOUT" "$@"; then local code=$? if [[ $code -eq 124 ]]; then - append_shard_gap "shard-${producer}-${slug}" "$producer" "failed" \ + append_shard_gap "shard-${producer}-${slug}" "$producer" "cannot_verify" \ "${producer} timed out after ${SHARD_TIMEOUT}s on ${slug}" "$repo" fail_log "${producer} timeout: $repo" else - append_shard_gap "shard-${producer}-${slug}" "$producer" "failed" \ + append_shard_gap "shard-${producer}-${slug}" "$producer" "cannot_verify" \ "${producer} failed (exit ${code}) on ${slug}" "$repo" fail_log "${producer} failed: $repo (exit $code)" fi @@ -101,7 +135,6 @@ run_shard() { } command -v jq >/dev/null || { log "jq is required"; exit 1; } -command -v node >/dev/null || { log "node is required for viewer"; exit 1; } has_producer() { local p=$1 @@ -214,8 +247,11 @@ run_semgrep() { command -v semgrep >/dev/null || { log "semgrep not available"; return 1; } local rules="$SEMGREP_RULES" if [[ ! -f "$rules" ]]; then - rules="p/default" - log "local semgrep rules missing; using p/default (needs approval for network rules)" + log "local semgrep rules missing at $rules; recording gap (no network fallback)" + append_gap_record "gap-semgrep-rules" "static-analysis" "not_assessed" \ + "Local semgrep rules missing; install harness/recipes/semgrep-rules/portolan-local.yaml" \ + "harness/recipes/static-semgrep-local.md" + return 1 fi mkdir -p "$PRODUCERS_DIR/semgrep" local repos @@ -298,6 +334,8 @@ if [[ "$NO_VIEWER" -eq 1 ]]; then exit 0 fi +command -v node >/dev/null || { log "node is required for viewer"; exit 1; } + cd "$ROOT/viewer" node scripts/build-static.js log "viewer: http://127.0.0.1:$PORT/ (Ctrl+C to stop)" diff --git a/viewer/scripts/serve.js b/viewer/scripts/serve.js index c210e057..0babee20 100644 --- a/viewer/scripts/serve.js +++ b/viewer/scripts/serve.js @@ -42,11 +42,33 @@ function loadRepoRoots() { } repoRoots = loadRepoRoots(); +function isPathUnderRoot(filePath, root) { + const resolvedRoot = path.resolve(root); + const resolved = path.resolve(filePath); + return resolved === resolvedRoot || resolved.startsWith(resolvedRoot + path.sep); +} + function isUnderRepoRoot(filePath) { const resolved = path.resolve(filePath); - return repoRoots.some( - (root) => resolved === root || resolved.startsWith(root + path.sep) - ); + return repoRoots.some((root) => isPathUnderRoot(resolved, root)); +} + +function isReadableRepoFile(filePath) { + let stats; + try { + stats = fs.lstatSync(filePath); + } catch { + return false; + } + if (stats.isSymbolicLink()) return false; + if (!stats.isFile()) return false; + let realPath; + try { + realPath = fs.realpathSync(filePath); + } catch { + return false; + } + return isUnderRepoRoot(realPath); } function resolveSourcePath(requestPath) { @@ -57,13 +79,13 @@ function resolveSourcePath(requestPath) { if (path.isAbsolute(raw)) { const resolved = path.resolve(raw); if (!isUnderRepoRoot(resolved)) return null; - return fs.existsSync(resolved) && fs.statSync(resolved).isFile() ? resolved : null; + return isReadableRepoFile(resolved) ? resolved : null; } for (const root of repoRoots) { const candidate = path.resolve(root, raw); - if (!candidate.startsWith(root + path.sep) && candidate !== root) continue; - if (fs.existsSync(candidate) && fs.statSync(candidate).isFile()) { + if (!isPathUnderRoot(candidate, root)) continue; + if (isReadableRepoFile(candidate)) { return candidate; } } @@ -125,8 +147,12 @@ const server = http.createServer((req, res) => { return sendFile(path.join(bundlePath, 'repos.json'), res); } - let filePath = path.join(distDir, url.pathname === '/' ? 'index.html' : url.pathname); - if (!filePath.startsWith(distDir)) { + const distResolved = path.resolve(distDir) + path.sep; + let filePath = path.resolve( + distDir, + url.pathname === '/' ? 'index.html' : url.pathname.replace(/^\/+/, '') + ); + if (!filePath.startsWith(distResolved)) { res.writeHead(403); return res.end('Forbidden'); } diff --git a/viewer/src/app.js b/viewer/src/app.js index 92667982..791dc431 100644 --- a/viewer/src/app.js +++ b/viewer/src/app.js @@ -102,8 +102,12 @@ function matchesFilters(h) { if (filters.kinds.size && !filters.kinds.has(h.kind)) return false; if (filters.severities.size && !filters.severities.has(h.severity)) return false; if (filters.repoIds.size) { - const rid = hotspotRepo(h); - if (!rid || !filters.repoIds.has(rid)) return false; + if (h.kind === 'dep-hub') { + // Dependency hubs are landscape-wide; repo chips should not hide them. + } else { + const rid = hotspotRepo(h); + if (!rid || !filters.repoIds.has(rid)) return false; + } } return true; } From 1bb10f90c8ac83c7d1d1af0cbdfb3080b4719afc Mon Sep 17 00:00:00 2001 From: Andrey Zhukov Date: Wed, 10 Jun 2026 18:54:57 +0300 Subject: [PATCH 07/12] Align Baseline CI with harness smoke and orient bundle schema Run harness-orient-smoke.sh and validate orient-bundle.schema.json in GitHub Baseline. Record PR #64 readiness closeout for specs 087-091. Co-authored-by: Cursor --- .github/workflows/ci.yml | 12 ++++++- .../pr-readiness-closeout-2026-06-10.md | 31 +++++++++++++++++++ 2 files changed, 42 insertions(+), 1 deletion(-) create mode 100644 docs/specs/091-landscape-scale/reviews/pr-readiness-closeout-2026-06-10.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 621c8c3f..3e047fe9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,6 +26,11 @@ jobs: go-version-file: go.mod cache: true + - name: Set up Node + uses: actions/setup-node@v4 + with: + node-version: "22" + - name: Go vet run: go vet ./... @@ -33,7 +38,12 @@ jobs: run: go test -count=1 ./... - name: Validate schemas - run: jq empty schema/*.json internal/testfixtures/oss-adapter-contract/*.json + run: | + jq empty schema/*.json internal/testfixtures/oss-adapter-contract/*.json + jq empty harness/contracts/orient-bundle.schema.json + + - name: Harness orient smoke + run: scripts/harness-orient-smoke.sh - name: Diff whitespace check run: git diff --check diff --git a/docs/specs/091-landscape-scale/reviews/pr-readiness-closeout-2026-06-10.md b/docs/specs/091-landscape-scale/reviews/pr-readiness-closeout-2026-06-10.md new file mode 100644 index 00000000..ca699746 --- /dev/null +++ b/docs/specs/091-landscape-scale/reviews/pr-readiness-closeout-2026-06-10.md @@ -0,0 +1,31 @@ +# PR #64 readiness closeout — 2026-06-10 + +- **PR**: https://github.com/fcon-tech/portolan/pull/64 +- **Branch**: `codex/087-091-harness-pivot` +- **HEAD**: post review-fix + CI alignment commit + +## Readiness Matrix + +| Surface | State | Evidence | +| --- | --- | --- | +| Local implementation | verified | specs 087–091 tasks complete; review-fix commit applied | +| Local verification | verified | `go test -count=1 ./...`, `go vet ./...`, `jq empty` schemas, `scripts/harness-orient-smoke.sh` | +| Review evidence | verified | 3 replacement lanes + disposition in `reviews/pr-review-disposition-2026-06-10.md` | +| PR state | draft → ready-for-review | after green checks on CI-alignment commit | +| GitHub checks | verified (pending re-run) | Baseline/CodeQL green on `27800c2`; duplicate CodeQL workflow flake on `Analyze (python)` | +| Merge approval | not_assessed | no human/GitHub approval recorded | +| Merge readiness | not-ready | explicit merge approval required | + +## Blockers resolved in review-fix + +- `/source` symlink escape (SEC-001) +- Bundle schema conformance (dep-hub paths, gap status, gap budget) +- Wizard flag validation; semgrep fail-closed + +## CI note + +Two dynamic CodeQL workflows (`PR #64` and `Code Quality: PR #64`) both run on the same push. One `Analyze (python)` job failed with token upload errors while the sibling workflow succeeded — infra flake, not a code defect. Baseline job updated to run harness orient smoke in-repo. + +## Stop reason + +Ready-for-review after latest push checks are green. Not ready-to-merge without explicit approval. From 4d79aeb84a5653788b4f0d134b141bd87351f111 Mon Sep 17 00:00:00 2001 From: Andrey Zhukov Date: Wed, 10 Jun 2026 18:58:10 +0300 Subject: [PATCH 08/12] Exclude fixture trees from CodeQL extraction Add codeql-config paths-ignore for internal/testfixtures and generated viewer assets so dynamic CodeQL autobuild does not choke on intentional broken-import fixture modules. Co-authored-by: Cursor --- .github/codeql/codeql-config.yml | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 .github/codeql/codeql-config.yml diff --git a/.github/codeql/codeql-config.yml b/.github/codeql/codeql-config.yml new file mode 100644 index 00000000..6dbcf43a --- /dev/null +++ b/.github/codeql/codeql-config.yml @@ -0,0 +1,8 @@ +name: Portolan CodeQL + +# Fixture trees use intentional broken imports for graph/debt tests; exclude from extraction. +paths-ignore: + - internal/testfixtures + - viewer/dist + - docs + - harness/recipes/semgrep-rules From b75689c177482782874db898a9753df50c99e762 Mon Sep 17 00:00:00 2001 From: Andrey Zhukov Date: Wed, 10 Jun 2026 19:01:27 +0300 Subject: [PATCH 09/12] Add repo-owned CodeQL workflow with security-events write Dynamic org CodeQL jobs fail on token auth during init; add an explicit workflow with required permissions and fixture paths-ignore config. Co-authored-by: Cursor --- .github/workflows/codeql.yml | 47 ++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 .github/workflows/codeql.yml diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml new file mode 100644 index 00000000..27cc8cdf --- /dev/null +++ b/.github/workflows/codeql.yml @@ -0,0 +1,47 @@ +name: CodeQL + +on: + push: + branches: [main] + pull_request: + branches: [main] + workflow_dispatch: + +permissions: + actions: read + contents: read + security-events: write + +jobs: + analyze: + name: Analyze (${{ matrix.language }}) + runs-on: ubuntu-latest + timeout-minutes: 30 + strategy: + fail-fast: false + matrix: + language: [go, javascript] + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Initialize CodeQL + uses: github/codeql-action/init@v3 + with: + languages: ${{ matrix.language }} + config-file: ./.github/codeql/codeql-config.yml + + - name: Set up Go + if: matrix.language == 'go' + uses: actions/setup-go@v5 + with: + go-version-file: go.mod + cache: true + + - name: Autobuild + uses: github/codeql-action/autobuild@v3 + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@v3 + with: + category: /language:${{ matrix.language }} From e2c76951c41c8dce9c07bf267d91a44b5342ee5d Mon Sep 17 00:00:00 2001 From: Andrey Zhukov Date: Wed, 10 Jun 2026 19:04:39 +0300 Subject: [PATCH 10/12] Remove repo CodeQL workflow; conflicts with org default setup Custom codeql.yml/codeql-config.yml is rejected when GitHub default CodeQL is enabled. Keep Baseline CI (harness smoke) as repo-controlled gate; document org CodeQL flake in PR readiness closeout. Co-authored-by: Cursor --- .github/codeql/codeql-config.yml | 8 ---- .github/workflows/codeql.yml | 47 ------------------- .../pr-readiness-closeout-2026-06-10.md | 4 +- 3 files changed, 2 insertions(+), 57 deletions(-) delete mode 100644 .github/codeql/codeql-config.yml delete mode 100644 .github/workflows/codeql.yml diff --git a/.github/codeql/codeql-config.yml b/.github/codeql/codeql-config.yml deleted file mode 100644 index 6dbcf43a..00000000 --- a/.github/codeql/codeql-config.yml +++ /dev/null @@ -1,8 +0,0 @@ -name: Portolan CodeQL - -# Fixture trees use intentional broken imports for graph/debt tests; exclude from extraction. -paths-ignore: - - internal/testfixtures - - viewer/dist - - docs - - harness/recipes/semgrep-rules diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml deleted file mode 100644 index 27cc8cdf..00000000 --- a/.github/workflows/codeql.yml +++ /dev/null @@ -1,47 +0,0 @@ -name: CodeQL - -on: - push: - branches: [main] - pull_request: - branches: [main] - workflow_dispatch: - -permissions: - actions: read - contents: read - security-events: write - -jobs: - analyze: - name: Analyze (${{ matrix.language }}) - runs-on: ubuntu-latest - timeout-minutes: 30 - strategy: - fail-fast: false - matrix: - language: [go, javascript] - steps: - - name: Checkout - uses: actions/checkout@v4 - - - name: Initialize CodeQL - uses: github/codeql-action/init@v3 - with: - languages: ${{ matrix.language }} - config-file: ./.github/codeql/codeql-config.yml - - - name: Set up Go - if: matrix.language == 'go' - uses: actions/setup-go@v5 - with: - go-version-file: go.mod - cache: true - - - name: Autobuild - uses: github/codeql-action/autobuild@v3 - - - name: Perform CodeQL Analysis - uses: github/codeql-action/analyze@v3 - with: - category: /language:${{ matrix.language }} diff --git a/docs/specs/091-landscape-scale/reviews/pr-readiness-closeout-2026-06-10.md b/docs/specs/091-landscape-scale/reviews/pr-readiness-closeout-2026-06-10.md index ca699746..f8a521dc 100644 --- a/docs/specs/091-landscape-scale/reviews/pr-readiness-closeout-2026-06-10.md +++ b/docs/specs/091-landscape-scale/reviews/pr-readiness-closeout-2026-06-10.md @@ -12,7 +12,7 @@ | Local verification | verified | `go test -count=1 ./...`, `go vet ./...`, `jq empty` schemas, `scripts/harness-orient-smoke.sh` | | Review evidence | verified | 3 replacement lanes + disposition in `reviews/pr-review-disposition-2026-06-10.md` | | PR state | draft → ready-for-review | after green checks on CI-alignment commit | -| GitHub checks | verified (pending re-run) | Baseline/CodeQL green on `27800c2`; duplicate CodeQL workflow flake on `Analyze (python)` | +| GitHub checks | verified (Baseline) / flaky (org CodeQL) | Repo `CI` Baseline green incl. harness smoke; org default CodeQL dynamic jobs intermittently fail on token auth — do not add repo `codeql.yml` while default setup is enabled | | Merge approval | not_assessed | no human/GitHub approval recorded | | Merge readiness | not-ready | explicit merge approval required | @@ -24,7 +24,7 @@ ## CI note -Two dynamic CodeQL workflows (`PR #64` and `Code Quality: PR #64`) both run on the same push. One `Analyze (python)` job failed with token upload errors while the sibling workflow succeeded — infra flake, not a code defect. Baseline job updated to run harness orient smoke in-repo. +Org **default CodeQL setup** is enabled. Adding repo `codeql.yml` or `codeql-config.yml` causes SARIF rejection (`advanced configurations cannot be processed when the default setup is enabled`). Dynamic CodeQL jobs occasionally fail on `Requires authentication` during init/upload — org infra, not PR code. Repo-controlled **Baseline** CI (go test/vet, schemas, harness smoke) is green. ## Stop reason From 139a005ead6b6447ef7680ad1f61917cf038e713 Mon Sep 17 00:00:00 2001 From: Andrey Zhukov Date: Wed, 10 Jun 2026 19:07:51 +0300 Subject: [PATCH 11/12] Record PR #64 ready-for-review closeout with green CI evidence All checks green on e2c7695 after removing conflicting repo CodeQL config. Co-authored-by: Cursor --- .../reviews/pr-readiness-closeout-2026-06-10.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/specs/091-landscape-scale/reviews/pr-readiness-closeout-2026-06-10.md b/docs/specs/091-landscape-scale/reviews/pr-readiness-closeout-2026-06-10.md index f8a521dc..d883a41c 100644 --- a/docs/specs/091-landscape-scale/reviews/pr-readiness-closeout-2026-06-10.md +++ b/docs/specs/091-landscape-scale/reviews/pr-readiness-closeout-2026-06-10.md @@ -11,8 +11,8 @@ | Local implementation | verified | specs 087–091 tasks complete; review-fix commit applied | | Local verification | verified | `go test -count=1 ./...`, `go vet ./...`, `jq empty` schemas, `scripts/harness-orient-smoke.sh` | | Review evidence | verified | 3 replacement lanes + disposition in `reviews/pr-review-disposition-2026-06-10.md` | -| PR state | draft → ready-for-review | after green checks on CI-alignment commit | -| GitHub checks | verified (Baseline) / flaky (org CodeQL) | Repo `CI` Baseline green incl. harness smoke; org default CodeQL dynamic jobs intermittently fail on token auth — do not add repo `codeql.yml` while default setup is enabled | +| PR state | ready-for-review | marked ready on `e2c7695` after all checks green | +| GitHub checks | verified | Baseline + Analyze (go/python/actions) + CodeQL success on `e2c7695` | | Merge approval | not_assessed | no human/GitHub approval recorded | | Merge readiness | not-ready | explicit merge approval required | @@ -28,4 +28,4 @@ Org **default CodeQL setup** is enabled. Adding repo `codeql.yml` or `codeql-con ## Stop reason -Ready-for-review after latest push checks are green. Not ready-to-merge without explicit approval. +Ready-for-review on `e2c7695` (all GitHub checks green). Not ready-to-merge without explicit approval. From dc82508b0bbcfbaad63706c7034ec77180466af7 Mon Sep 17 00:00:00 2001 From: Andrey Zhukov Date: Wed, 10 Jun 2026 19:10:58 +0300 Subject: [PATCH 12/12] ci: retrigger checks after closeout doc-only commit Org default CodeQL Analyze (python) flaked on docs-only push; no code change. Co-authored-by: Cursor