From 0d00a9bcb62d8d5f35adf9a35c6fc2f63c9e3308 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Wed, 10 Jun 2026 19:04:46 +0300 Subject: [PATCH 01/14] fix(demo): resolve F-01 reproducible failure path --- .github/workflows/demo-ci.yml | 14 ++++++++++- .../phases/01-enterprise-hardening/01-UAT.md | 24 +++++++++++++++++++ README.md | 5 ++-- .../20260610_f01_reproducible-failure.md | 8 +++++++ 4 files changed, 48 insertions(+), 3 deletions(-) create mode 100644 .planning/phases/01-enterprise-hardening/01-UAT.md create mode 100644 memory/examples/20260610_f01_reproducible-failure.md diff --git a/.github/workflows/demo-ci.yml b/.github/workflows/demo-ci.yml index 2c7758c..764374a 100644 --- a/.github/workflows/demo-ci.yml +++ b/.github/workflows/demo-ci.yml @@ -1,7 +1,13 @@ -name: Demo CI +name: Demo CI on: workflow_dispatch: + inputs: + simulate_failure: + description: "Fail the demo job to exercise Autopilot intake" + required: true + type: boolean + default: false push: branches: [main] @@ -12,3 +18,9 @@ jobs: - name: Demo sanity run: | echo "Simulated demo step (non-blocking)." + + - name: Simulate repairable failure + if: ${{ inputs.simulate_failure }} + run: | + echo "Intentional demo failure requested." + exit 1 diff --git a/.planning/phases/01-enterprise-hardening/01-UAT.md b/.planning/phases/01-enterprise-hardening/01-UAT.md new file mode 100644 index 0000000..8a582c5 --- /dev/null +++ b/.planning/phases/01-enterprise-hardening/01-UAT.md @@ -0,0 +1,24 @@ +# Enterprise Hardening UAT + +**Date:** 2026-06-10 +**Source:** `gsd-audit-fix --severity all --max 8` +**Scope:** demo reproducibility, security, tests, CI, user journey, failure modes, and docs + +## Classification + +| ID | Severity | Classification | Finding | Files | +|---|---|---|---|---| +| F-01 | high | auto-fixable | The documented demo cannot produce a failure because Demo CI always succeeds. | `.github/workflows/demo-ci.yml`, `README.md` | +| F-02 | high | auto-fixable | CI prints YAML parse warnings but still exits successfully. | `.github/workflows/ci.yml` | +| F-03 | high | auto-fixable | Third-party Actions use mutable major-version tags instead of immutable commit SHAs. | `.github/workflows/ci.yml`, `.github/workflows/autopilot-create-issue.yml` | +| F-04 | medium | auto-fixable | CI and Demo CI do not declare least-privilege token permissions or job timeouts. | `.github/workflows/ci.yml`, `.github/workflows/demo-ci.yml` | +| F-05 | medium | auto-fixable | No automated contract tests protect workflow behavior and security invariants. | `tests/test_workflows.py`, `.github/workflows/ci.yml` | +| F-06 | medium | auto-fixable | Intake processing has no concurrency guard or timeout, allowing duplicate/racing issue updates. | `.github/workflows/autopilot-create-issue.yml` | +| F-07 | low | auto-fixable | Intake issues omit run attempt, event, actor, and failure conclusion from the audit evidence. | `.github/workflows/autopilot-create-issue.yml` | +| F-08 | low | auto-fixable | The demo runbook lacks prerequisites, reset steps, and failure-mode troubleshooting. | `README.md` | + +## Manual-only Findings + +- M-01: Protect `main` and require the `CI` status check. The GitHub API reports that `main` is not protected. +- M-02: Configure the organization Actions policy to allow only approved actions and require immutable SHA pinning where supported. The current GitHub token cannot read or change this setting. + diff --git a/README.md b/README.md index b7ae683..3c38fb9 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# autopilot-demo +# autopilot-demo [![CI](https://github.com/Coding-Autopilot-System/autopilot-demo/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/Coding-Autopilot-System/autopilot-demo/actions/workflows/ci.yml) [![Demo CI](https://github.com/Coding-Autopilot-System/autopilot-demo/actions/workflows/demo-ci.yml/badge.svg?branch=main)](https://github.com/Coding-Autopilot-System/autopilot-demo/actions/workflows/demo-ci.yml) @@ -47,7 +47,7 @@ gh pr list -R Coding-Autopilot-System/autopilot-demo ## Demo runbook -1. Trigger [`.github/workflows/demo-ci.yml`](.github/workflows/demo-ci.yml) to produce a known failure signal. +1. Trigger [`.github/workflows/demo-ci.yml`](.github/workflows/demo-ci.yml) with `simulate_failure=true` to produce a known failure signal. Pushes and default dispatches remain green. 2. Confirm [`.github/workflows/autopilot-create-issue.yml`](.github/workflows/autopilot-create-issue.yml) creates an `autofix + queued` issue. 3. Watch `autopilot-core` pick up the issue and open a PR back into this repo. 4. Use this repo's issue, branch, and PR history as the audit trail for the demo. @@ -72,3 +72,4 @@ gh pr list -R Coding-Autopilot-System/autopilot-demo - [autopilot-core](https://github.com/Coding-Autopilot-System/autopilot-core) - operator control plane - [ci-autopilot](https://github.com/Coding-Autopilot-System/ci-autopilot) - worker/runtime reference - [Coding-Autopilot-System org](https://github.com/Coding-Autopilot-System) + diff --git a/memory/examples/20260610_f01_reproducible-failure.md b/memory/examples/20260610_f01_reproducible-failure.md new file mode 100644 index 0000000..baca1f9 --- /dev/null +++ b/memory/examples/20260610_f01_reproducible-failure.md @@ -0,0 +1,8 @@ +# F-01 reproducible demo failure + +Issue Description: The documented demo could not produce a failure because Demo CI always succeeded. +State: Pushes and manual dispatches both ran only a non-blocking step. +Action: Added an explicit `simulate_failure` dispatch input and documented the required command. +Result: Operators can intentionally exercise intake while normal pushes and default dispatches stay green. +Diff Patch: Updated `.github/workflows/demo-ci.yml` and `README.md`. +Rationale: A demo repair pipeline must expose a safe, deliberate, and reproducible failure path. From f5df6b9ffdec83d081d863ab2fce120845fbd0a5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Wed, 10 Jun 2026 19:05:05 +0300 Subject: [PATCH 02/14] fix(ci): resolve F-02 fail-closed YAML validation --- .github/workflows/ci.yml | 32 +++++++++++-------- .../20260610_f02_strict-yaml-validation.md | 8 +++++ requirements-dev.txt | 2 ++ 3 files changed, 28 insertions(+), 14 deletions(-) create mode 100644 memory/examples/20260610_f02_strict-yaml-validation.md create mode 100644 requirements-dev.txt diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f3ecdbd..bd7dd42 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,4 +1,4 @@ -name: CI +name: CI on: push: @@ -12,18 +12,22 @@ jobs: steps: - uses: actions/checkout@v4 + - name: Install validation dependencies + run: python -m pip install --requirement requirements-dev.txt + - name: Validate workflow YAML + shell: python run: | - python -c " - import yaml, glob - files = glob.glob('.github/workflows/*.yml') - errors = [] - for f in files: - try: - with open(f) as fh: - yaml.safe_load(fh) - print(' OK:', f) - except yaml.YAMLError as e: - print(' WARN:', f, '-', str(e).split('\n')[0]) - print('Validated', len(files), 'workflow files') - " + import pathlib + import yaml + + files = sorted(pathlib.Path('.github/workflows').glob('*.yml')) + if not files: + raise SystemExit('No workflow files found') + + for path in files: + with path.open(encoding='utf-8') as stream: + yaml.safe_load(stream) + print(f'OK: {path}') + + print(f'Validated {len(files)} workflow files') diff --git a/memory/examples/20260610_f02_strict-yaml-validation.md b/memory/examples/20260610_f02_strict-yaml-validation.md new file mode 100644 index 0000000..63887cb --- /dev/null +++ b/memory/examples/20260610_f02_strict-yaml-validation.md @@ -0,0 +1,8 @@ +# F-02 strict YAML validation + +Issue Description: CI printed YAML parsing warnings but always returned success. +State: Invalid workflow YAML could pass the portfolio CI check. +Action: Made parsing errors fatal and pinned the YAML parser dependency. +Result: CI now fails closed when workflow YAML is missing or invalid. +Diff Patch: Updated `.github/workflows/ci.yml` and added `requirements-dev.txt`. +Rationale: Validation jobs must fail on the condition they claim to validate. diff --git a/requirements-dev.txt b/requirements-dev.txt new file mode 100644 index 0000000..ad9efe1 --- /dev/null +++ b/requirements-dev.txt @@ -0,0 +1,2 @@ +PyYAML==6.0.3 + From 8a6508700979c06623b0264cd8d33f57a634f665 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Wed, 10 Jun 2026 19:05:20 +0300 Subject: [PATCH 03/14] fix(security): resolve F-03 pin Actions by SHA --- .github/workflows/autopilot-create-issue.yml | 4 ++-- .github/workflows/ci.yml | 2 +- memory/examples/20260610_f03_immutable-actions.md | 8 ++++++++ 3 files changed, 11 insertions(+), 3 deletions(-) create mode 100644 memory/examples/20260610_f03_immutable-actions.md diff --git a/.github/workflows/autopilot-create-issue.yml b/.github/workflows/autopilot-create-issue.yml index b98f2ba..653bede 100644 --- a/.github/workflows/autopilot-create-issue.yml +++ b/.github/workflows/autopilot-create-issue.yml @@ -1,4 +1,4 @@ -name: Autopilot Issue Intake +name: Autopilot Issue Intake on: workflow_run: @@ -17,7 +17,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Create or update issue - uses: actions/github-script@v7 + uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1 with: script: | const run = context.payload.workflow_run; diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index bd7dd42..77e42e2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -10,7 +10,7 @@ jobs: validate: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - name: Install validation dependencies run: python -m pip install --requirement requirements-dev.txt diff --git a/memory/examples/20260610_f03_immutable-actions.md b/memory/examples/20260610_f03_immutable-actions.md new file mode 100644 index 0000000..b62c481 --- /dev/null +++ b/memory/examples/20260610_f03_immutable-actions.md @@ -0,0 +1,8 @@ +# F-03 immutable Action references + +Issue Description: Workflows trusted mutable major-version Action tags. +State: Upstream tag movement could change executed code without a repository diff. +Action: Pinned checkout v4.2.2 and github-script v7.0.1 to verified commit SHAs. +Result: Third-party workflow code is immutable and remains human-readable through version comments. +Diff Patch: Updated CI and intake workflow Action references. +Rationale: Immutable references reduce CI supply-chain risk. From c18cd4b7d6b0b4beb26631a8654425ca9a887665 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Wed, 10 Jun 2026 19:05:35 +0300 Subject: [PATCH 04/14] fix(security): resolve F-04 bound workflow permissions --- .github/workflows/ci.yml | 4 ++++ .github/workflows/demo-ci.yml | 4 ++++ memory/examples/20260610_f04_workflow-boundaries.md | 8 ++++++++ 3 files changed, 16 insertions(+) create mode 100644 memory/examples/20260610_f04_workflow-boundaries.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 77e42e2..4bef5d7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -6,9 +6,13 @@ on: pull_request: branches: [main] +permissions: + contents: read + jobs: validate: runs-on: ubuntu-latest + timeout-minutes: 5 steps: - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 diff --git a/.github/workflows/demo-ci.yml b/.github/workflows/demo-ci.yml index 764374a..9c624c9 100644 --- a/.github/workflows/demo-ci.yml +++ b/.github/workflows/demo-ci.yml @@ -11,9 +11,13 @@ on: push: branches: [main] +permissions: + contents: read + jobs: demo: runs-on: ubuntu-latest + timeout-minutes: 5 steps: - name: Demo sanity run: | diff --git a/memory/examples/20260610_f04_workflow-boundaries.md b/memory/examples/20260610_f04_workflow-boundaries.md new file mode 100644 index 0000000..fd8a1ac --- /dev/null +++ b/memory/examples/20260610_f04_workflow-boundaries.md @@ -0,0 +1,8 @@ +# F-04 least-privilege workflow boundaries + +Issue Description: CI and Demo CI relied on repository-default token permissions and had no execution timeout. +State: Jobs could inherit broader permissions and run without a repository-defined time bound. +Action: Declared read-only contents permission and five-minute job timeouts. +Result: Routine workflows now have explicit least privilege and bounded execution. +Diff Patch: Updated `.github/workflows/ci.yml` and `.github/workflows/demo-ci.yml`. +Rationale: Security boundaries should be visible and enforced in source control. From 605078134150bddb0e3f2a0374c3e9e42efded2a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Wed, 10 Jun 2026 19:06:37 +0300 Subject: [PATCH 05/14] fix(test): resolve F-05 add workflow contracts --- .github/workflows/ci.yml | 3 ++ .../20260610_f05_workflow-contract-tests.md | 8 ++++ tests/test_workflows.py | 46 +++++++++++++++++++ 3 files changed, 57 insertions(+) create mode 100644 memory/examples/20260610_f05_workflow-contract-tests.md create mode 100644 tests/test_workflows.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4bef5d7..657ec66 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -35,3 +35,6 @@ jobs: print(f'OK: {path}') print(f'Validated {len(files)} workflow files') + + - name: Run workflow contract tests + run: python -m unittest discover -s tests -v diff --git a/memory/examples/20260610_f05_workflow-contract-tests.md b/memory/examples/20260610_f05_workflow-contract-tests.md new file mode 100644 index 0000000..1199ae9 --- /dev/null +++ b/memory/examples/20260610_f05_workflow-contract-tests.md @@ -0,0 +1,8 @@ +# F-05 workflow contract tests + +Issue Description: No automated tests protected workflow behavior and security invariants. +State: Regressions in YAML validity, Action pinning, permissions, timeouts, or demo failure behavior depended on review alone. +Action: Added standard-library workflow contract tests and executed them from CI. +Result: Core demo and security invariants now fail visibly in pull requests. +Diff Patch: Added `tests/test_workflows.py` and updated `.github/workflows/ci.yml`. +Rationale: Workflow code needs executable contracts like application code. diff --git a/tests/test_workflows.py b/tests/test_workflows.py new file mode 100644 index 0000000..3bb946c --- /dev/null +++ b/tests/test_workflows.py @@ -0,0 +1,46 @@ +import pathlib +import re +import unittest + +import yaml + + +ROOT = pathlib.Path(__file__).resolve().parents[1] +WORKFLOWS = ROOT / ".github" / "workflows" + + +class WorkflowContractTests(unittest.TestCase): + def workflow_text(self, name: str) -> str: + return (WORKFLOWS / name).read_text(encoding="utf-8-sig") + + def test_all_workflows_are_valid_yaml(self) -> None: + paths = sorted(WORKFLOWS.glob("*.yml")) + self.assertTrue(paths, "expected at least one workflow") + for path in paths: + with self.subTest(path=path.name): + self.assertIsInstance(yaml.safe_load(path.read_text(encoding="utf-8-sig")), dict) + + def test_third_party_actions_are_pinned_to_commits(self) -> None: + action_ref = re.compile(r"uses:\s+[^./\s][^@\s]*@([^\s#]+)") + for path in WORKFLOWS.glob("*.yml"): + for ref in action_ref.findall(path.read_text(encoding="utf-8-sig")): + with self.subTest(path=path.name, ref=ref): + self.assertRegex(ref, r"^[0-9a-f]{40}$") + + def test_routine_workflows_are_read_only_and_bounded(self) -> None: + for name in ("ci.yml", "demo-ci.yml"): + text = self.workflow_text(name) + with self.subTest(name=name): + self.assertIn("permissions:\n contents: read", text.replace("\r\n", "\n")) + self.assertIn("timeout-minutes: 5", text) + + def test_demo_failure_is_explicitly_opt_in(self) -> None: + text = self.workflow_text("demo-ci.yml") + self.assertIn("simulate_failure:", text) + self.assertIn("default: false", text) + self.assertIn("if: ${{ inputs.simulate_failure }}", text) + self.assertIn("exit 1", text) + + +if __name__ == "__main__": + unittest.main() From 5b2146770dcdd052e738c8e1e36fed2a31ee38c6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Wed, 10 Jun 2026 19:06:53 +0300 Subject: [PATCH 06/14] fix(intake): resolve F-06 serialize issue updates --- .github/workflows/autopilot-create-issue.yml | 5 +++++ memory/examples/20260610_f06_serialized-intake.md | 8 ++++++++ tests/test_workflows.py | 7 ++++++- 3 files changed, 19 insertions(+), 1 deletion(-) create mode 100644 memory/examples/20260610_f06_serialized-intake.md diff --git a/.github/workflows/autopilot-create-issue.yml b/.github/workflows/autopilot-create-issue.yml index 653bede..7564939 100644 --- a/.github/workflows/autopilot-create-issue.yml +++ b/.github/workflows/autopilot-create-issue.yml @@ -6,6 +6,10 @@ on: workflows: ["Demo CI"] types: [completed] +concurrency: + group: autopilot-intake-${{ github.event.workflow_run.head_sha }} + cancel-in-progress: false + permissions: actions: read issues: write @@ -15,6 +19,7 @@ jobs: create-issue: if: ${{ github.event.workflow_run.conclusion == 'failure' }} runs-on: ubuntu-latest + timeout-minutes: 5 steps: - name: Create or update issue uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1 diff --git a/memory/examples/20260610_f06_serialized-intake.md b/memory/examples/20260610_f06_serialized-intake.md new file mode 100644 index 0000000..c4ad48e --- /dev/null +++ b/memory/examples/20260610_f06_serialized-intake.md @@ -0,0 +1,8 @@ +# F-06 serialized intake processing + +Issue Description: Intake processing had no concurrency guard or execution timeout. +State: Multiple failures for the same commit could race while updating the same signature issue. +Action: Serialized intake by workflow head SHA, preserved queued runs, and bounded the job to five minutes. +Result: Intake updates are deterministic and cannot run indefinitely. +Diff Patch: Updated the intake workflow and its contract tests. +Rationale: Idempotent issue handling still needs race protection at the workflow boundary. diff --git a/tests/test_workflows.py b/tests/test_workflows.py index 3bb946c..896fecf 100644 --- a/tests/test_workflows.py +++ b/tests/test_workflows.py @@ -1,4 +1,4 @@ -import pathlib +import pathlib import re import unittest @@ -34,6 +34,11 @@ def test_routine_workflows_are_read_only_and_bounded(self) -> None: self.assertIn("permissions:\n contents: read", text.replace("\r\n", "\n")) self.assertIn("timeout-minutes: 5", text) + def test_intake_is_serialized_and_bounded(self) -> None: + text = self.workflow_text("autopilot-create-issue.yml") + self.assertIn("group: autopilot-intake-${{ github.event.workflow_run.head_sha }}", text) + self.assertIn("cancel-in-progress: false", text) + self.assertIn("timeout-minutes: 5", text) def test_demo_failure_is_explicitly_opt_in(self) -> None: text = self.workflow_text("demo-ci.yml") self.assertIn("simulate_failure:", text) From 8676864f5b085758d6e5d0bdd168671ed899c988 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Wed, 10 Jun 2026 19:07:08 +0300 Subject: [PATCH 07/14] fix(intake): resolve F-07 enrich audit evidence --- .github/workflows/autopilot-create-issue.yml | 4 ++++ memory/examples/20260610_f07_intake-audit-context.md | 8 ++++++++ tests/test_workflows.py | 5 +++++ 3 files changed, 17 insertions(+) create mode 100644 memory/examples/20260610_f07_intake-audit-context.md diff --git a/.github/workflows/autopilot-create-issue.yml b/.github/workflows/autopilot-create-issue.yml index 7564939..ba5e3a8 100644 --- a/.github/workflows/autopilot-create-issue.yml +++ b/.github/workflows/autopilot-create-issue.yml @@ -62,6 +62,10 @@ jobs: `Branch: ${run.head_branch}`, `SHA: ${run.head_sha}`, `Run: ${run.html_url}`, + `Attempt: ${run.run_attempt}`, + `Event: ${run.event}`, + `Actor: ${run.actor?.login || 'unknown'}`, + `Conclusion: ${run.conclusion}`, '', 'Failed steps:', stepSummary, diff --git a/memory/examples/20260610_f07_intake-audit-context.md b/memory/examples/20260610_f07_intake-audit-context.md new file mode 100644 index 0000000..9e180dd --- /dev/null +++ b/memory/examples/20260610_f07_intake-audit-context.md @@ -0,0 +1,8 @@ +# F-07 richer intake audit evidence + +Issue Description: Intake issues omitted operational context needed to reconstruct a failure. +State: Issues recorded workflow, branch, SHA, run URL, and failed steps only. +Action: Added run attempt, triggering event, actor, and conclusion to every intake issue body. +Result: Each issue now carries a stronger self-contained audit trail. +Diff Patch: Updated the intake workflow and its contract tests. +Rationale: Operational evidence should support diagnosis without requiring immediate API access. diff --git a/tests/test_workflows.py b/tests/test_workflows.py index 896fecf..2ca2e39 100644 --- a/tests/test_workflows.py +++ b/tests/test_workflows.py @@ -39,6 +39,11 @@ def test_intake_is_serialized_and_bounded(self) -> None: self.assertIn("group: autopilot-intake-${{ github.event.workflow_run.head_sha }}", text) self.assertIn("cancel-in-progress: false", text) self.assertIn("timeout-minutes: 5", text) + def test_intake_records_audit_context(self) -> None: + text = self.workflow_text("autopilot-create-issue.yml") + for field in ("Attempt:", "Event:", "Actor:", "Conclusion:"): + with self.subTest(field=field): + self.assertIn(field, text) def test_demo_failure_is_explicitly_opt_in(self) -> None: text = self.workflow_text("demo-ci.yml") self.assertIn("simulate_failure:", text) From 5d92fde8e6d801a2b6cc6158ea008611a8c37354 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Wed, 10 Jun 2026 19:07:37 +0300 Subject: [PATCH 08/14] fix(docs): resolve F-08 repeatable demo runbook --- .planning/phases/01-enterprise-hardening/01-UAT.md | 7 +++---- README.md | 12 ++++++++++++ memory/examples/20260610_f08_repeatable-runbook.md | 8 ++++++++ tests/test_workflows.py | 3 +++ 4 files changed, 26 insertions(+), 4 deletions(-) create mode 100644 memory/examples/20260610_f08_repeatable-runbook.md diff --git a/.planning/phases/01-enterprise-hardening/01-UAT.md b/.planning/phases/01-enterprise-hardening/01-UAT.md index 8a582c5..7c3e83f 100644 --- a/.planning/phases/01-enterprise-hardening/01-UAT.md +++ b/.planning/phases/01-enterprise-hardening/01-UAT.md @@ -1,4 +1,4 @@ -# Enterprise Hardening UAT +# Enterprise Hardening UAT **Date:** 2026-06-10 **Source:** `gsd-audit-fix --severity all --max 8` @@ -8,17 +8,16 @@ | ID | Severity | Classification | Finding | Files | |---|---|---|---|---| -| F-01 | high | auto-fixable | The documented demo cannot produce a failure because Demo CI always succeeds. | `.github/workflows/demo-ci.yml`, `README.md` | +| F-01 | high | auto-fixable | The documented demo cannot produce a failure because Demo CI always succeeds. | `.github/workflows/demo-ci.yml`, `.github/workflows/autopilot-create-issue.yml`, `README.md` | | F-02 | high | auto-fixable | CI prints YAML parse warnings but still exits successfully. | `.github/workflows/ci.yml` | | F-03 | high | auto-fixable | Third-party Actions use mutable major-version tags instead of immutable commit SHAs. | `.github/workflows/ci.yml`, `.github/workflows/autopilot-create-issue.yml` | | F-04 | medium | auto-fixable | CI and Demo CI do not declare least-privilege token permissions or job timeouts. | `.github/workflows/ci.yml`, `.github/workflows/demo-ci.yml` | | F-05 | medium | auto-fixable | No automated contract tests protect workflow behavior and security invariants. | `tests/test_workflows.py`, `.github/workflows/ci.yml` | | F-06 | medium | auto-fixable | Intake processing has no concurrency guard or timeout, allowing duplicate/racing issue updates. | `.github/workflows/autopilot-create-issue.yml` | | F-07 | low | auto-fixable | Intake issues omit run attempt, event, actor, and failure conclusion from the audit evidence. | `.github/workflows/autopilot-create-issue.yml` | -| F-08 | low | auto-fixable | The demo runbook lacks prerequisites, reset steps, and failure-mode troubleshooting. | `README.md` | +| F-08 | low | auto-fixable | The demo runbook lacks prerequisites, reset steps, and failure-mode troubleshooting; repeat demonstrations can leave a matching intake issue closed. | `.github/workflows/autopilot-create-issue.yml`, `README.md` | ## Manual-only Findings - M-01: Protect `main` and require the `CI` status check. The GitHub API reports that `main` is not protected. - M-02: Configure the organization Actions policy to allow only approved actions and require immutable SHA pinning where supported. The current GitHub token cannot read or change this setting. - diff --git a/README.md b/README.md index 3c38fb9..464a14f 100644 --- a/README.md +++ b/README.md @@ -72,4 +72,16 @@ gh pr list -R Coding-Autopilot-System/autopilot-demo - [autopilot-core](https://github.com/Coding-Autopilot-System/autopilot-core) - operator control plane - [ci-autopilot](https://github.com/Coding-Autopilot-System/ci-autopilot) - worker/runtime reference - [Coding-Autopilot-System org](https://github.com/Coding-Autopilot-System) +## Prerequisites and expected result +- Authenticate GitHub CLI with `gh auth status` and confirm Actions are enabled for this repository. +- Run the `autopilot-core` operator with access to this repository before triggering the failure. +- Expect Demo CI to fail, Autopilot Issue Intake to create or reopen one `autofix + queued` issue, and the operator to propose a pull request. + +## Reset and troubleshooting + +1. Close the completed intake issue and merge or close its repair pull request before the next demonstration. +2. Re-run with `simulate_failure=true`; intake reopens the matching issue when the same commit is demonstrated again. +3. If no issue appears, inspect `gh run list -R Coding-Autopilot-System/autopilot-demo --workflow autopilot-create-issue.yml` and confirm the failed run was named `Demo CI`. +4. If the issue remains queued, verify the `autopilot-core` operator is running and can read issues and create branches and pull requests in this repository. +5. If CI fails before the demo step, run `python -m unittest discover -s tests -v` locally and repair the workflow contract violation first. diff --git a/memory/examples/20260610_f08_repeatable-runbook.md b/memory/examples/20260610_f08_repeatable-runbook.md new file mode 100644 index 0000000..ec7a526 --- /dev/null +++ b/memory/examples/20260610_f08_repeatable-runbook.md @@ -0,0 +1,8 @@ +# F-08 repeatable operator runbook + +Issue Description: The runbook lacked prerequisites, reset steps, and troubleshooting, while repeat demonstrations could leave the matching issue closed. +State: Operators had no defined recovery path and intake updated closed matching issues without reopening them. +Action: Documented prerequisites, expected results, reset and troubleshooting; reopen matching intake issues on repeat failures. +Result: The user journey is repeatable and common failure modes have concrete diagnostics. +Diff Patch: Updated the intake workflow, README, UAT record, and contract tests. +Rationale: A portfolio demo must be operable repeatedly by someone other than its author. diff --git a/tests/test_workflows.py b/tests/test_workflows.py index 2ca2e39..6bdd92e 100644 --- a/tests/test_workflows.py +++ b/tests/test_workflows.py @@ -44,6 +44,9 @@ def test_intake_records_audit_context(self) -> None: for field in ("Attempt:", "Event:", "Actor:", "Conclusion:"): with self.subTest(field=field): self.assertIn(field, text) + def test_existing_intake_issue_is_reopened(self) -> None: + text = self.workflow_text("autopilot-create-issue.yml") + self.assertIn("state: 'open'", text) def test_demo_failure_is_explicitly_opt_in(self) -> None: text = self.workflow_text("demo-ci.yml") self.assertIn("simulate_failure:", text) From 3033bfbee7c0ac6c0d338f8e84bd076988ff0252 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Wed, 10 Jun 2026 19:07:49 +0300 Subject: [PATCH 09/14] Revert "fix(docs): resolve F-08 repeatable demo runbook" This reverts commit 5d92fde8e6d801a2b6cc6158ea008611a8c37354. --- .planning/phases/01-enterprise-hardening/01-UAT.md | 7 ++++--- README.md | 12 ------------ memory/examples/20260610_f08_repeatable-runbook.md | 8 -------- tests/test_workflows.py | 3 --- 4 files changed, 4 insertions(+), 26 deletions(-) delete mode 100644 memory/examples/20260610_f08_repeatable-runbook.md diff --git a/.planning/phases/01-enterprise-hardening/01-UAT.md b/.planning/phases/01-enterprise-hardening/01-UAT.md index 7c3e83f..8a582c5 100644 --- a/.planning/phases/01-enterprise-hardening/01-UAT.md +++ b/.planning/phases/01-enterprise-hardening/01-UAT.md @@ -1,4 +1,4 @@ -# Enterprise Hardening UAT +# Enterprise Hardening UAT **Date:** 2026-06-10 **Source:** `gsd-audit-fix --severity all --max 8` @@ -8,16 +8,17 @@ | ID | Severity | Classification | Finding | Files | |---|---|---|---|---| -| F-01 | high | auto-fixable | The documented demo cannot produce a failure because Demo CI always succeeds. | `.github/workflows/demo-ci.yml`, `.github/workflows/autopilot-create-issue.yml`, `README.md` | +| F-01 | high | auto-fixable | The documented demo cannot produce a failure because Demo CI always succeeds. | `.github/workflows/demo-ci.yml`, `README.md` | | F-02 | high | auto-fixable | CI prints YAML parse warnings but still exits successfully. | `.github/workflows/ci.yml` | | F-03 | high | auto-fixable | Third-party Actions use mutable major-version tags instead of immutable commit SHAs. | `.github/workflows/ci.yml`, `.github/workflows/autopilot-create-issue.yml` | | F-04 | medium | auto-fixable | CI and Demo CI do not declare least-privilege token permissions or job timeouts. | `.github/workflows/ci.yml`, `.github/workflows/demo-ci.yml` | | F-05 | medium | auto-fixable | No automated contract tests protect workflow behavior and security invariants. | `tests/test_workflows.py`, `.github/workflows/ci.yml` | | F-06 | medium | auto-fixable | Intake processing has no concurrency guard or timeout, allowing duplicate/racing issue updates. | `.github/workflows/autopilot-create-issue.yml` | | F-07 | low | auto-fixable | Intake issues omit run attempt, event, actor, and failure conclusion from the audit evidence. | `.github/workflows/autopilot-create-issue.yml` | -| F-08 | low | auto-fixable | The demo runbook lacks prerequisites, reset steps, and failure-mode troubleshooting; repeat demonstrations can leave a matching intake issue closed. | `.github/workflows/autopilot-create-issue.yml`, `README.md` | +| F-08 | low | auto-fixable | The demo runbook lacks prerequisites, reset steps, and failure-mode troubleshooting. | `README.md` | ## Manual-only Findings - M-01: Protect `main` and require the `CI` status check. The GitHub API reports that `main` is not protected. - M-02: Configure the organization Actions policy to allow only approved actions and require immutable SHA pinning where supported. The current GitHub token cannot read or change this setting. + diff --git a/README.md b/README.md index 464a14f..3c38fb9 100644 --- a/README.md +++ b/README.md @@ -72,16 +72,4 @@ gh pr list -R Coding-Autopilot-System/autopilot-demo - [autopilot-core](https://github.com/Coding-Autopilot-System/autopilot-core) - operator control plane - [ci-autopilot](https://github.com/Coding-Autopilot-System/ci-autopilot) - worker/runtime reference - [Coding-Autopilot-System org](https://github.com/Coding-Autopilot-System) -## Prerequisites and expected result -- Authenticate GitHub CLI with `gh auth status` and confirm Actions are enabled for this repository. -- Run the `autopilot-core` operator with access to this repository before triggering the failure. -- Expect Demo CI to fail, Autopilot Issue Intake to create or reopen one `autofix + queued` issue, and the operator to propose a pull request. - -## Reset and troubleshooting - -1. Close the completed intake issue and merge or close its repair pull request before the next demonstration. -2. Re-run with `simulate_failure=true`; intake reopens the matching issue when the same commit is demonstrated again. -3. If no issue appears, inspect `gh run list -R Coding-Autopilot-System/autopilot-demo --workflow autopilot-create-issue.yml` and confirm the failed run was named `Demo CI`. -4. If the issue remains queued, verify the `autopilot-core` operator is running and can read issues and create branches and pull requests in this repository. -5. If CI fails before the demo step, run `python -m unittest discover -s tests -v` locally and repair the workflow contract violation first. diff --git a/memory/examples/20260610_f08_repeatable-runbook.md b/memory/examples/20260610_f08_repeatable-runbook.md deleted file mode 100644 index ec7a526..0000000 --- a/memory/examples/20260610_f08_repeatable-runbook.md +++ /dev/null @@ -1,8 +0,0 @@ -# F-08 repeatable operator runbook - -Issue Description: The runbook lacked prerequisites, reset steps, and troubleshooting, while repeat demonstrations could leave the matching issue closed. -State: Operators had no defined recovery path and intake updated closed matching issues without reopening them. -Action: Documented prerequisites, expected results, reset and troubleshooting; reopen matching intake issues on repeat failures. -Result: The user journey is repeatable and common failure modes have concrete diagnostics. -Diff Patch: Updated the intake workflow, README, UAT record, and contract tests. -Rationale: A portfolio demo must be operable repeatedly by someone other than its author. diff --git a/tests/test_workflows.py b/tests/test_workflows.py index 6bdd92e..2ca2e39 100644 --- a/tests/test_workflows.py +++ b/tests/test_workflows.py @@ -44,9 +44,6 @@ def test_intake_records_audit_context(self) -> None: for field in ("Attempt:", "Event:", "Actor:", "Conclusion:"): with self.subTest(field=field): self.assertIn(field, text) - def test_existing_intake_issue_is_reopened(self) -> None: - text = self.workflow_text("autopilot-create-issue.yml") - self.assertIn("state: 'open'", text) def test_demo_failure_is_explicitly_opt_in(self) -> None: text = self.workflow_text("demo-ci.yml") self.assertIn("simulate_failure:", text) From 7804583d186e6e0fa9647d14efad9860fd24b2ed Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Wed, 10 Jun 2026 19:09:20 +0300 Subject: [PATCH 10/14] fix(docs): resolve F-08 repeatable demo runbook --- .github/workflows/autopilot-create-issue.yml | 1 + .planning/phases/01-enterprise-hardening/01-UAT.md | 5 ++--- README.md | 12 ++++++++++++ memory/examples/20260610_f08_repeatable-runbook.md | 8 ++++++++ tests/test_workflows.py | 3 +++ 5 files changed, 26 insertions(+), 3 deletions(-) create mode 100644 memory/examples/20260610_f08_repeatable-runbook.md diff --git a/.github/workflows/autopilot-create-issue.yml b/.github/workflows/autopilot-create-issue.yml index ba5e3a8..e2d28ff 100644 --- a/.github/workflows/autopilot-create-issue.yml +++ b/.github/workflows/autopilot-create-issue.yml @@ -101,6 +101,7 @@ jobs: owner, repo, issue_number: issue.number, + state: 'open', body, }); await github.rest.issues.addLabels({ diff --git a/.planning/phases/01-enterprise-hardening/01-UAT.md b/.planning/phases/01-enterprise-hardening/01-UAT.md index 8a582c5..b100d0b 100644 --- a/.planning/phases/01-enterprise-hardening/01-UAT.md +++ b/.planning/phases/01-enterprise-hardening/01-UAT.md @@ -1,4 +1,4 @@ -# Enterprise Hardening UAT +# Enterprise Hardening UAT **Date:** 2026-06-10 **Source:** `gsd-audit-fix --severity all --max 8` @@ -15,10 +15,9 @@ | F-05 | medium | auto-fixable | No automated contract tests protect workflow behavior and security invariants. | `tests/test_workflows.py`, `.github/workflows/ci.yml` | | F-06 | medium | auto-fixable | Intake processing has no concurrency guard or timeout, allowing duplicate/racing issue updates. | `.github/workflows/autopilot-create-issue.yml` | | F-07 | low | auto-fixable | Intake issues omit run attempt, event, actor, and failure conclusion from the audit evidence. | `.github/workflows/autopilot-create-issue.yml` | -| F-08 | low | auto-fixable | The demo runbook lacks prerequisites, reset steps, and failure-mode troubleshooting. | `README.md` | +| F-08 | low | auto-fixable | The demo runbook lacks prerequisites, reset steps, and failure-mode troubleshooting; repeat demonstrations can leave a matching intake issue closed. | `.github/workflows/autopilot-create-issue.yml`, `README.md` | ## Manual-only Findings - M-01: Protect `main` and require the `CI` status check. The GitHub API reports that `main` is not protected. - M-02: Configure the organization Actions policy to allow only approved actions and require immutable SHA pinning where supported. The current GitHub token cannot read or change this setting. - diff --git a/README.md b/README.md index 3c38fb9..464a14f 100644 --- a/README.md +++ b/README.md @@ -72,4 +72,16 @@ gh pr list -R Coding-Autopilot-System/autopilot-demo - [autopilot-core](https://github.com/Coding-Autopilot-System/autopilot-core) - operator control plane - [ci-autopilot](https://github.com/Coding-Autopilot-System/ci-autopilot) - worker/runtime reference - [Coding-Autopilot-System org](https://github.com/Coding-Autopilot-System) +## Prerequisites and expected result +- Authenticate GitHub CLI with `gh auth status` and confirm Actions are enabled for this repository. +- Run the `autopilot-core` operator with access to this repository before triggering the failure. +- Expect Demo CI to fail, Autopilot Issue Intake to create or reopen one `autofix + queued` issue, and the operator to propose a pull request. + +## Reset and troubleshooting + +1. Close the completed intake issue and merge or close its repair pull request before the next demonstration. +2. Re-run with `simulate_failure=true`; intake reopens the matching issue when the same commit is demonstrated again. +3. If no issue appears, inspect `gh run list -R Coding-Autopilot-System/autopilot-demo --workflow autopilot-create-issue.yml` and confirm the failed run was named `Demo CI`. +4. If the issue remains queued, verify the `autopilot-core` operator is running and can read issues and create branches and pull requests in this repository. +5. If CI fails before the demo step, run `python -m unittest discover -s tests -v` locally and repair the workflow contract violation first. diff --git a/memory/examples/20260610_f08_repeatable-runbook.md b/memory/examples/20260610_f08_repeatable-runbook.md new file mode 100644 index 0000000..ec7a526 --- /dev/null +++ b/memory/examples/20260610_f08_repeatable-runbook.md @@ -0,0 +1,8 @@ +# F-08 repeatable operator runbook + +Issue Description: The runbook lacked prerequisites, reset steps, and troubleshooting, while repeat demonstrations could leave the matching issue closed. +State: Operators had no defined recovery path and intake updated closed matching issues without reopening them. +Action: Documented prerequisites, expected results, reset and troubleshooting; reopen matching intake issues on repeat failures. +Result: The user journey is repeatable and common failure modes have concrete diagnostics. +Diff Patch: Updated the intake workflow, README, UAT record, and contract tests. +Rationale: A portfolio demo must be operable repeatedly by someone other than its author. diff --git a/tests/test_workflows.py b/tests/test_workflows.py index 2ca2e39..6bdd92e 100644 --- a/tests/test_workflows.py +++ b/tests/test_workflows.py @@ -44,6 +44,9 @@ def test_intake_records_audit_context(self) -> None: for field in ("Attempt:", "Event:", "Actor:", "Conclusion:"): with self.subTest(field=field): self.assertIn(field, text) + def test_existing_intake_issue_is_reopened(self) -> None: + text = self.workflow_text("autopilot-create-issue.yml") + self.assertIn("state: 'open'", text) def test_demo_failure_is_explicitly_opt_in(self) -> None: text = self.workflow_text("demo-ci.yml") self.assertIn("simulate_failure:", text) From eca5a272eed645b3ca8861b4617b19a10b7a41b0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Wed, 10 Jun 2026 19:09:54 +0300 Subject: [PATCH 11/14] fix(ci): resolve F-02 dependency file hygiene --- requirements-dev.txt | 1 - 1 file changed, 1 deletion(-) diff --git a/requirements-dev.txt b/requirements-dev.txt index ad9efe1..f62ce0c 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -1,2 +1 @@ PyYAML==6.0.3 - From 868a53306bcd5faf439157851dc023fc89b6acb0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Wed, 10 Jun 2026 19:10:31 +0300 Subject: [PATCH 12/14] fix(test): resolve F-05 ignore generated bytecode --- .gitignore | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 .gitignore diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..43ae0e2 --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +__pycache__/ +*.py[cod] From 30bc5fecea871a73c8f9e1cccd41534ed29885ea Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Wed, 10 Jun 2026 19:25:58 +0300 Subject: [PATCH 13/14] docs(gsd): verify F-01 through F-08 hardening --- .../01-enterprise-hardening/01-SUMMARY.md | 8 +++++ .../01-VERIFICATION.md | 34 +++++++++++++++++++ 2 files changed, 42 insertions(+) create mode 100644 .planning/phases/01-enterprise-hardening/01-SUMMARY.md create mode 100644 .planning/phases/01-enterprise-hardening/01-VERIFICATION.md diff --git a/.planning/phases/01-enterprise-hardening/01-SUMMARY.md b/.planning/phases/01-enterprise-hardening/01-SUMMARY.md new file mode 100644 index 0000000..f8f003b --- /dev/null +++ b/.planning/phases/01-enterprise-hardening/01-SUMMARY.md @@ -0,0 +1,8 @@ +# Enterprise Hardening Summary + +The GSD audit classified eight findings as auto-fixable and two GitHub administration items as manual-only. All eight repository findings were fixed, tested, and recorded in `memory/examples/`. + +The demo now provides a deliberate failure switch, strict CI validation, immutable Action references, least-privilege workflow boundaries, executable workflow contracts, serialized intake, richer audit evidence, and a repeatable operator runbook. + +F-08 failed its first contract test because the intake update path did not reopen closed issues. That attempt was reverted, corrected, and verified on retry. + diff --git a/.planning/phases/01-enterprise-hardening/01-VERIFICATION.md b/.planning/phases/01-enterprise-hardening/01-VERIFICATION.md new file mode 100644 index 0000000..4bff207 --- /dev/null +++ b/.planning/phases/01-enterprise-hardening/01-VERIFICATION.md @@ -0,0 +1,34 @@ +# Enterprise Hardening Verification + +**Date:** 2026-06-10 +**Branch:** `hardening/enterprise-audit-20260610` +**Result:** passed with manual GitHub administration gaps + +## Finding Status + +| ID | Result | Evidence | +|---|---|---| +| F-01 | passed | `simulate_failure` is explicit and defaults to false | +| F-02 | passed | invalid YAML now fails CI; dependency is pinned | +| F-03 | passed | third-party Actions use verified 40-character commit SHAs | +| F-04 | passed | routine jobs declare read-only permissions and five-minute timeouts | +| F-05 | passed | seven workflow contract tests run in CI; bytecode is ignored | +| F-06 | passed | intake updates are serialized by head SHA and time-bounded | +| F-07 | passed | issues record attempt, event, actor, and conclusion | +| F-08 | passed after one reverted attempt | matching closed issues reopen; runbook covers prerequisites, reset, and troubleshooting | + +## Verification Commands + +- `python -m unittest discover -s tests -v`: 7 passed +- strict PyYAML parse of `.github/workflows/*.yml`: 3 validated +- PowerShell parser check of `scripts/record-fix.ps1`: passed +- `git diff --check origin/main...HEAD`: passed +- clean worktree after tests: passed + +## Manual Gaps + +- Protect `main` and require the `CI` status check. +- Configure the organization Actions allow-list and immutable pinning policy. +- Run the live failure-to-issue-to-repair journey after this branch is merged because `workflow_dispatch` uses the default-branch workflow definition. +- Install `actionlint` in the workstation bootstrap and add it to CI; it was not available for this verification. + From 060186beb200562f286d3c081884ff6f138a3172 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Wed, 10 Jun 2026 19:26:28 +0300 Subject: [PATCH 14/14] docs(gsd): finalize F-01 through F-08 evidence --- .planning/phases/01-enterprise-hardening/01-SUMMARY.md | 1 - .planning/phases/01-enterprise-hardening/01-VERIFICATION.md | 1 - 2 files changed, 2 deletions(-) diff --git a/.planning/phases/01-enterprise-hardening/01-SUMMARY.md b/.planning/phases/01-enterprise-hardening/01-SUMMARY.md index f8f003b..83f83c9 100644 --- a/.planning/phases/01-enterprise-hardening/01-SUMMARY.md +++ b/.planning/phases/01-enterprise-hardening/01-SUMMARY.md @@ -5,4 +5,3 @@ The GSD audit classified eight findings as auto-fixable and two GitHub administr The demo now provides a deliberate failure switch, strict CI validation, immutable Action references, least-privilege workflow boundaries, executable workflow contracts, serialized intake, richer audit evidence, and a repeatable operator runbook. F-08 failed its first contract test because the intake update path did not reopen closed issues. That attempt was reverted, corrected, and verified on retry. - diff --git a/.planning/phases/01-enterprise-hardening/01-VERIFICATION.md b/.planning/phases/01-enterprise-hardening/01-VERIFICATION.md index 4bff207..c0c42b5 100644 --- a/.planning/phases/01-enterprise-hardening/01-VERIFICATION.md +++ b/.planning/phases/01-enterprise-hardening/01-VERIFICATION.md @@ -31,4 +31,3 @@ - Configure the organization Actions allow-list and immutable pinning policy. - Run the live failure-to-issue-to-repair journey after this branch is merged because `workflow_dispatch` uses the default-branch workflow definition. - Install `actionlint` in the workstation bootstrap and add it to CI; it was not available for this verification. -