-
Notifications
You must be signed in to change notification settings - Fork 0
feat: harden deterministic CI repair demo #4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
0d00a9b
fix(demo): resolve F-01 reproducible failure path
Aitomates f5df6b9
fix(ci): resolve F-02 fail-closed YAML validation
Aitomates 8a65087
fix(security): resolve F-03 pin Actions by SHA
Aitomates c18cd4b
fix(security): resolve F-04 bound workflow permissions
Aitomates 6050781
fix(test): resolve F-05 add workflow contracts
Aitomates 5b21467
fix(intake): resolve F-06 serialize issue updates
Aitomates 8676864
fix(intake): resolve F-07 enrich audit evidence
Aitomates 5d92fde
fix(docs): resolve F-08 repeatable demo runbook
Aitomates 3033bfb
Revert "fix(docs): resolve F-08 repeatable demo runbook"
Aitomates 7804583
fix(docs): resolve F-08 repeatable demo runbook
Aitomates eca5a27
fix(ci): resolve F-02 dependency file hygiene
Aitomates 868a533
fix(test): resolve F-05 ignore generated bytecode
Aitomates 30bc5fe
docs(gsd): verify F-01 through F-08 hardening
Aitomates 060186b
docs(gsd): finalize F-01 through F-08 evidence
Aitomates File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,29 +1,40 @@ | ||
| name: CI | ||
| name: CI | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main] | ||
| pull_request: | ||
| branches: [main] | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| validate: | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 5 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
|
|
||
| - name: Install validation dependencies | ||
| run: python -m pip install --requirement requirements-dev.txt | ||
|
|
||
| - name: Validate workflow YAML | ||
| shell: python | ||
| run: | | ||
| python -c " | ||
| import yaml, glob | ||
| files = glob.glob('.github/workflows/*.yml') | ||
| errors = [] | ||
| for f in files: | ||
| try: | ||
| with open(f) as fh: | ||
| yaml.safe_load(fh) | ||
| print(' OK:', f) | ||
| except yaml.YAMLError as e: | ||
| print(' WARN:', f, '-', str(e).split('\n')[0]) | ||
| print('Validated', len(files), 'workflow files') | ||
| " | ||
| import pathlib | ||
| import yaml | ||
|
|
||
| files = sorted(pathlib.Path('.github/workflows').glob('*.yml')) | ||
| if not files: | ||
| raise SystemExit('No workflow files found') | ||
|
|
||
| for path in files: | ||
| with path.open(encoding='utf-8') as stream: | ||
| yaml.safe_load(stream) | ||
| print(f'OK: {path}') | ||
|
|
||
| print(f'Validated {len(files)} workflow files') | ||
|
|
||
| - name: Run workflow contract tests | ||
| run: python -m unittest discover -s tests -v |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,14 +1,30 @@ | ||
| name: Demo CI | ||
| name: Demo CI | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| simulate_failure: | ||
| description: "Fail the demo job to exercise Autopilot intake" | ||
| required: true | ||
| type: boolean | ||
| default: false | ||
| push: | ||
| branches: [main] | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| demo: | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 5 | ||
| steps: | ||
| - name: Demo sanity | ||
| run: | | ||
| echo "Simulated demo step (non-blocking)." | ||
|
|
||
| - name: Simulate repairable failure | ||
| if: ${{ inputs.simulate_failure }} | ||
| run: | | ||
| echo "Intentional demo failure requested." | ||
| exit 1 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| __pycache__/ | ||
| *.py[cod] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| # Enterprise Hardening Summary | ||
|
|
||
| The GSD audit classified eight findings as auto-fixable and two GitHub administration items as manual-only. All eight repository findings were fixed, tested, and recorded in `memory/examples/`. | ||
|
|
||
| The demo now provides a deliberate failure switch, strict CI validation, immutable Action references, least-privilege workflow boundaries, executable workflow contracts, serialized intake, richer audit evidence, and a repeatable operator runbook. | ||
|
|
||
| F-08 failed its first contract test because the intake update path did not reopen closed issues. That attempt was reverted, corrected, and verified on retry. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # Enterprise Hardening UAT | ||
|
|
||
| **Date:** 2026-06-10 | ||
| **Source:** `gsd-audit-fix --severity all --max 8` | ||
| **Scope:** demo reproducibility, security, tests, CI, user journey, failure modes, and docs | ||
|
|
||
| ## Classification | ||
|
|
||
| | ID | Severity | Classification | Finding | Files | | ||
| |---|---|---|---|---| | ||
| | F-01 | high | auto-fixable | The documented demo cannot produce a failure because Demo CI always succeeds. | `.github/workflows/demo-ci.yml`, `README.md` | | ||
| | F-02 | high | auto-fixable | CI prints YAML parse warnings but still exits successfully. | `.github/workflows/ci.yml` | | ||
| | F-03 | high | auto-fixable | Third-party Actions use mutable major-version tags instead of immutable commit SHAs. | `.github/workflows/ci.yml`, `.github/workflows/autopilot-create-issue.yml` | | ||
| | F-04 | medium | auto-fixable | CI and Demo CI do not declare least-privilege token permissions or job timeouts. | `.github/workflows/ci.yml`, `.github/workflows/demo-ci.yml` | | ||
| | F-05 | medium | auto-fixable | No automated contract tests protect workflow behavior and security invariants. | `tests/test_workflows.py`, `.github/workflows/ci.yml` | | ||
| | F-06 | medium | auto-fixable | Intake processing has no concurrency guard or timeout, allowing duplicate/racing issue updates. | `.github/workflows/autopilot-create-issue.yml` | | ||
| | F-07 | low | auto-fixable | Intake issues omit run attempt, event, actor, and failure conclusion from the audit evidence. | `.github/workflows/autopilot-create-issue.yml` | | ||
| | F-08 | low | auto-fixable | The demo runbook lacks prerequisites, reset steps, and failure-mode troubleshooting; repeat demonstrations can leave a matching intake issue closed. | `.github/workflows/autopilot-create-issue.yml`, `README.md` | | ||
|
|
||
| ## Manual-only Findings | ||
|
|
||
| - M-01: Protect `main` and require the `CI` status check. The GitHub API reports that `main` is not protected. | ||
| - M-02: Configure the organization Actions policy to allow only approved actions and require immutable SHA pinning where supported. The current GitHub token cannot read or change this setting. |
33 changes: 33 additions & 0 deletions
33
.planning/phases/01-enterprise-hardening/01-VERIFICATION.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| # Enterprise Hardening Verification | ||
|
|
||
| **Date:** 2026-06-10 | ||
| **Branch:** `hardening/enterprise-audit-20260610` | ||
| **Result:** passed with manual GitHub administration gaps | ||
|
|
||
| ## Finding Status | ||
|
|
||
| | ID | Result | Evidence | | ||
| |---|---|---| | ||
| | F-01 | passed | `simulate_failure` is explicit and defaults to false | | ||
| | F-02 | passed | invalid YAML now fails CI; dependency is pinned | | ||
| | F-03 | passed | third-party Actions use verified 40-character commit SHAs | | ||
| | F-04 | passed | routine jobs declare read-only permissions and five-minute timeouts | | ||
| | F-05 | passed | seven workflow contract tests run in CI; bytecode is ignored | | ||
| | F-06 | passed | intake updates are serialized by head SHA and time-bounded | | ||
| | F-07 | passed | issues record attempt, event, actor, and conclusion | | ||
| | F-08 | passed after one reverted attempt | matching closed issues reopen; runbook covers prerequisites, reset, and troubleshooting | | ||
|
|
||
| ## Verification Commands | ||
|
|
||
| - `python -m unittest discover -s tests -v`: 7 passed | ||
| - strict PyYAML parse of `.github/workflows/*.yml`: 3 validated | ||
| - PowerShell parser check of `scripts/record-fix.ps1`: passed | ||
| - `git diff --check origin/main...HEAD`: passed | ||
| - clean worktree after tests: passed | ||
|
|
||
| ## Manual Gaps | ||
|
|
||
| - Protect `main` and require the `CI` status check. | ||
| - Configure the organization Actions allow-list and immutable pinning policy. | ||
| - Run the live failure-to-issue-to-repair journey after this branch is merged because `workflow_dispatch` uses the default-branch workflow definition. | ||
| - Install `actionlint` in the workstation bootstrap and add it to CI; it was not available for this verification. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| # F-01 reproducible demo failure | ||
|
|
||
| Issue Description: The documented demo could not produce a failure because Demo CI always succeeded. | ||
| State: Pushes and manual dispatches both ran only a non-blocking step. | ||
| Action: Added an explicit `simulate_failure` dispatch input and documented the required command. | ||
| Result: Operators can intentionally exercise intake while normal pushes and default dispatches stay green. | ||
| Diff Patch: Updated `.github/workflows/demo-ci.yml` and `README.md`. | ||
| Rationale: A demo repair pipeline must expose a safe, deliberate, and reproducible failure path. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| # F-02 strict YAML validation | ||
|
|
||
| Issue Description: CI printed YAML parsing warnings but always returned success. | ||
| State: Invalid workflow YAML could pass the portfolio CI check. | ||
| Action: Made parsing errors fatal and pinned the YAML parser dependency. | ||
| Result: CI now fails closed when workflow YAML is missing or invalid. | ||
| Diff Patch: Updated `.github/workflows/ci.yml` and added `requirements-dev.txt`. | ||
| Rationale: Validation jobs must fail on the condition they claim to validate. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| # F-03 immutable Action references | ||
|
|
||
| Issue Description: Workflows trusted mutable major-version Action tags. | ||
| State: Upstream tag movement could change executed code without a repository diff. | ||
| Action: Pinned checkout v4.2.2 and github-script v7.0.1 to verified commit SHAs. | ||
| Result: Third-party workflow code is immutable and remains human-readable through version comments. | ||
| Diff Patch: Updated CI and intake workflow Action references. | ||
| Rationale: Immutable references reduce CI supply-chain risk. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| # F-04 least-privilege workflow boundaries | ||
|
|
||
| Issue Description: CI and Demo CI relied on repository-default token permissions and had no execution timeout. | ||
| State: Jobs could inherit broader permissions and run without a repository-defined time bound. | ||
| Action: Declared read-only contents permission and five-minute job timeouts. | ||
| Result: Routine workflows now have explicit least privilege and bounded execution. | ||
| Diff Patch: Updated `.github/workflows/ci.yml` and `.github/workflows/demo-ci.yml`. | ||
| Rationale: Security boundaries should be visible and enforced in source control. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| # F-05 workflow contract tests | ||
|
|
||
| Issue Description: No automated tests protected workflow behavior and security invariants. | ||
| State: Regressions in YAML validity, Action pinning, permissions, timeouts, or demo failure behavior depended on review alone. | ||
| Action: Added standard-library workflow contract tests and executed them from CI. | ||
| Result: Core demo and security invariants now fail visibly in pull requests. | ||
| Diff Patch: Added `tests/test_workflows.py` and updated `.github/workflows/ci.yml`. | ||
| Rationale: Workflow code needs executable contracts like application code. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| # F-06 serialized intake processing | ||
|
|
||
| Issue Description: Intake processing had no concurrency guard or execution timeout. | ||
| State: Multiple failures for the same commit could race while updating the same signature issue. | ||
| Action: Serialized intake by workflow head SHA, preserved queued runs, and bounded the job to five minutes. | ||
| Result: Intake updates are deterministic and cannot run indefinitely. | ||
| Diff Patch: Updated the intake workflow and its contract tests. | ||
| Rationale: Idempotent issue handling still needs race protection at the workflow boundary. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| # F-07 richer intake audit evidence | ||
|
|
||
| Issue Description: Intake issues omitted operational context needed to reconstruct a failure. | ||
| State: Issues recorded workflow, branch, SHA, run URL, and failed steps only. | ||
| Action: Added run attempt, triggering event, actor, and conclusion to every intake issue body. | ||
| Result: Each issue now carries a stronger self-contained audit trail. | ||
| Diff Patch: Updated the intake workflow and its contract tests. | ||
| Rationale: Operational evidence should support diagnosis without requiring immediate API access. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| # F-08 repeatable operator runbook | ||
|
|
||
| Issue Description: The runbook lacked prerequisites, reset steps, and troubleshooting, while repeat demonstrations could leave the matching issue closed. | ||
| State: Operators had no defined recovery path and intake updated closed matching issues without reopening them. | ||
| Action: Documented prerequisites, expected results, reset and troubleshooting; reopen matching intake issues on repeat failures. | ||
| Result: The user journey is repeatable and common failure modes have concrete diagnostics. | ||
| Diff Patch: Updated the intake workflow, README, UAT record, and contract tests. | ||
| Rationale: A portfolio demo must be operable repeatedly by someone other than its author. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| PyYAML==6.0.3 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| import pathlib | ||
| import re | ||
| import unittest | ||
|
|
||
| import yaml | ||
|
|
||
|
|
||
| ROOT = pathlib.Path(__file__).resolve().parents[1] | ||
| WORKFLOWS = ROOT / ".github" / "workflows" | ||
|
|
||
|
|
||
| class WorkflowContractTests(unittest.TestCase): | ||
| def workflow_text(self, name: str) -> str: | ||
| return (WORKFLOWS / name).read_text(encoding="utf-8-sig") | ||
|
|
||
| def test_all_workflows_are_valid_yaml(self) -> None: | ||
| paths = sorted(WORKFLOWS.glob("*.yml")) | ||
| self.assertTrue(paths, "expected at least one workflow") | ||
| for path in paths: | ||
| with self.subTest(path=path.name): | ||
| self.assertIsInstance(yaml.safe_load(path.read_text(encoding="utf-8-sig")), dict) | ||
|
|
||
| def test_third_party_actions_are_pinned_to_commits(self) -> None: | ||
| action_ref = re.compile(r"uses:\s+[^./\s][^@\s]*@([^\s#]+)") | ||
| for path in WORKFLOWS.glob("*.yml"): | ||
| for ref in action_ref.findall(path.read_text(encoding="utf-8-sig")): | ||
| with self.subTest(path=path.name, ref=ref): | ||
| self.assertRegex(ref, r"^[0-9a-f]{40}$") | ||
|
|
||
| def test_routine_workflows_are_read_only_and_bounded(self) -> None: | ||
| for name in ("ci.yml", "demo-ci.yml"): | ||
| text = self.workflow_text(name) | ||
| with self.subTest(name=name): | ||
| self.assertIn("permissions:\n contents: read", text.replace("\r\n", "\n")) | ||
| self.assertIn("timeout-minutes: 5", text) | ||
|
|
||
| def test_intake_is_serialized_and_bounded(self) -> None: | ||
| text = self.workflow_text("autopilot-create-issue.yml") | ||
| self.assertIn("group: autopilot-intake-${{ github.event.workflow_run.head_sha }}", text) | ||
| self.assertIn("cancel-in-progress: false", text) | ||
| self.assertIn("timeout-minutes: 5", text) | ||
| def test_intake_records_audit_context(self) -> None: | ||
| text = self.workflow_text("autopilot-create-issue.yml") | ||
| for field in ("Attempt:", "Event:", "Actor:", "Conclusion:"): | ||
| with self.subTest(field=field): | ||
| self.assertIn(field, text) | ||
| def test_existing_intake_issue_is_reopened(self) -> None: | ||
| text = self.workflow_text("autopilot-create-issue.yml") | ||
| self.assertIn("state: 'open'", text) | ||
| def test_demo_failure_is_explicitly_opt_in(self) -> None: | ||
| text = self.workflow_text("demo-ci.yml") | ||
| self.assertIn("simulate_failure:", text) | ||
| self.assertIn("default: false", text) | ||
| self.assertIn("if: ${{ inputs.simulate_failure }}", text) | ||
| self.assertIn("exit 1", text) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| unittest.main() |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With this input defaulting to
false, thegh workflow run demo-ci.yml -R Coding-Autopilot-System/autopilot-democommand under README.md's "Running the demo" section now produces a successful run, so the intake workflow does not create an issue and the documented end-to-end demonstration stalls. Update that command to passsimulate_failure=trueor otherwise ensure the primary demo instructions explicitly select the failure input.Useful? React with 👍 / 👎.