fix: PLTF-2955 skip PR description validation for Dependabot PRs#3685
Conversation
Dependabot-created PRs don't follow the custom PR template format (HUMAN:, AGENT:, etc.) and shouldn't be required to. This change adds a check to skip validation entirely for PRs created by the dependabot[bot] user. Fixes the PR description validation check failing on dependency update PRs.
Python API breakage checks — ✅ PASSEDResult: ✅ PASSED |
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
✅ QA Report: PASS
The PR description validator now skips Dependabot-created PR events while preserving validation failures for non-Dependabot PRs.
This review was created by an AI agent (OpenHands) on behalf of the user.
Does this PR achieve its stated goal?
Yes. The stated goal is to skip PR description validation for PRs created by dependabot[bot], because Dependabot PRs do not use the repository's custom HUMAN/AGENT template. I reproduced the old behavior on origin/main: a realistic Dependabot event with a plain dependency-update body failed validation with 7 template errors. After switching to PR head e5f449d, the same event exited 0 with Skipping PR description validation for Dependabot PR., while an equivalent non-Dependabot event still failed validation.
| Phase | Result |
|---|---|
| Environment Setup | ✅ Repo was already checked out at PR head; Python 3.13.14 was available and no project dependency install was required for this stdlib script. |
| CI Status | 🟡 At observation time, 5 checks had succeeded, 1 was skipped, and 22/28 checks were still in progress. I did not run tests or linters locally. |
| Functional Verification | ✅ Ran the actual validator CLI with realistic GitHub event payloads on base and PR branches. |
Functional Verification
Test 1: Dependabot PR events skip template validation
Step 1 — Reproduce / establish baseline without the fix:
Ran on origin/main using a GitHub event payload whose PR author is dependabot[bot] and whose body is a normal dependency-update summary, not the custom template:
git checkout --detach origin/main
python .github/scripts/check_pr_description.py --event-path /tmp/qa-prdesc/dependabot_event.jsonOutput:
BASE_HEAD=914579dd
BASE_DEPENDABOT_EXIT=1
::error::The first visible line of the PR description must be `HUMAN:`.
::error::Add a short human-written note between `HUMAN:` and the human-tested checkbox.
::error::Keep the `- [ ] A human has tested these changes.` checkbox in the PR description.
::error::Keep the `AGENT:` marker from the PR template.
::error::Keep the `## Why` section from the PR template.
::error::Keep the `## Summary` section from the PR template.
::error::Keep the `## How to Test` section from the PR template.
PR description validation failed with 7 error(s).
This confirms the bug exists on the base branch: Dependabot-style PR descriptions are treated like human PR descriptions and fail the custom template validator.
Step 2 — Apply the PR's changes:
Checked out fix/skip-pr-validation-for-dependabot at e5f449d.
Step 3 — Re-run with the fix in place:
Ran the same validator command against the same Dependabot event payload:
git checkout fix/skip-pr-validation-for-dependabot
python .github/scripts/check_pr_description.py --event-path /tmp/qa-prdesc/dependabot_event.jsonOutput:
PR_HEAD=e5f449da
PR_DEPENDABOT_EXIT=0
Skipping PR description validation for Dependabot PR.
This confirms the fix works for the stated goal: Dependabot-created PR events bypass validation and return success.
Test 2: Non-Dependabot PR events still validate normally
Step 1 — Baseline without the fix:
Ran on origin/main using the same plain dependency-update body, but with PR author octocat:
python .github/scripts/check_pr_description.py --event-path /tmp/qa-prdesc/human_event.jsonOutput:
BASE_HUMAN_EXIT=1
::error::The first visible line of the PR description must be `HUMAN:`.
::error::Add a short human-written note between `HUMAN:` and the human-tested checkbox.
::error::Keep the `- [ ] A human has tested these changes.` checkbox in the PR description.
::error::Keep the `AGENT:` marker from the PR template.
::error::Keep the `## Why` section from the PR template.
::error::Keep the `## Summary` section from the PR template.
::error::Keep the `## How to Test` section from the PR template.
PR description validation failed with 7 error(s).
This establishes the expected behavior for non-Dependabot PRs: invalid descriptions fail.
Step 2 — Apply the PR's changes:
Checked out PR head e5f449d.
Step 3 — Re-run with the fix in place:
Ran the same validator command against the non-Dependabot event payload:
python .github/scripts/check_pr_description.py --event-path /tmp/qa-prdesc/human_event.jsonOutput:
PR_HUMAN_EXIT=1
::error::The first visible line of the PR description must be `HUMAN:`.
::error::Add a short human-written note between `HUMAN:` and the human-tested checkbox.
::error::Keep the `- [ ] A human has tested these changes.` checkbox in the PR description.
::error::Keep the `AGENT:` marker from the PR template.
::error::Keep the `## Why` section from the PR template.
::error::Keep the `## Summary` section from the PR template.
::error::Keep the `## How to Test` section from the PR template.
PR description validation failed with 7 error(s).
This shows the change is scoped correctly: non-Dependabot PRs continue to be validated.
Issues Found
None.
|
@OpenHands fix failing PR checks |
|
I'm on it! aivong-openhands can track my progress at all-hands.dev |
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
|
OpenHands encountered an error: **Failed to send message to agent server: HTTP 503 error: no available server See the conversation for more information. |
enyst
left a comment
enyst
left a comment
There was a problem hiding this comment.
Thank you! LGTM. I think maybe the check could be simplified and applied on the job conditions because dependabot[bot] is the login? But up to you, it’s not a problem
|
Just fixed in main the REST API check that was failing, should work |
No problem! @OpenHands can we simplify the check based on the suggestion above? |
|
I'm on it! aivong-openhands can track my progress at all-hands.dev |
aivong-openhands
changed the title
aivong-openhands
changed the title
…ondition Simplify the Dependabot check by moving it to the GitHub Actions workflow condition instead of checking it in the Python script. This is more efficient since the job won't even start for Dependabot PRs. Co-authored-by: openhands <openhands@all-hands.dev>
|
Summary of Work Completed: I simplified the Dependabot PR validation check based on @enyst's suggestion in PR #3685. Changes made:
Result: Reduced code by ~50 lines. The Dependabot exclusion is now handled at the workflow level, alongside other PR exclusion conditions (draft PRs, release PRs), making the intent clearer and the implementation simpler. |
|
✅ Review complete. This review was performed through OpenHands Cloud Automation. You can log in and view the conversation here. |
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
Code Review
🟢 Good taste - Elegant, simple solution.
The change is a minimal, targeted one-line addition to the workflow's conditional that skips PR description validation for Dependabot PRs. This is exactly the right approach:
- Real problem solved: Dependabot PRs don't use the custom
HUMAN:/AGENT:template, so they should be excluded from validation. - No breaking changes: All existing behavior for human-authored PRs is preserved.
- Low complexity: Single condition added to existing boolean expression.
The existing comprehensive QA review (PR #3685 by all-hands-bot) already verified both:
- Dependabot PRs skip validation (exit 0)
- Non-Dependabot PRs still fail validation as expected (exit 1)
An optional simplification was suggested by @enyst (moving the check to job-level if since it only depends on the PR author), but that's purely stylistic — the current placement is fine and follows the existing pattern.
[RISK ASSESSMENT]
- [Overall PR]
⚠️ Risk Assessment: 🟢 LOW
This is a minimal workflow logic change with no code execution impact. The change is already thoroughly tested by the existing QA review.
VERDICT:
✅ Worth merging: Core logic is sound, change is minimal and well-tested.
KEY INSIGHT:
Adding the github.event.pull_request.user.login != 'dependabot[bot]' condition to the existing workflow expression is the correct, minimal fix for this issue.
This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation. View conversation
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
✅ QA Report: PASS
The PR achieves its goal: Dependabot PR descriptions without the custom template are skipped, while non-Dependabot PRs still run validation.
Does this PR achieve its stated goal?
Yes. I exercised the PR Description Check workflow locally with realistic pull_request_target events using act: on the base workflow, a Dependabot PR body without HUMAN:/AGENT: failed validation with 7 template errors; with this PR's workflow, the same Dependabot event exited successfully without running the validation job. I also ran a non-Dependabot event with the same invalid body and confirmed validation still ran and failed, so the skip is limited to Dependabot.
| Phase | Result |
|---|---|
| Environment Setup | ✅ make build completed and dependencies were installed. |
| CI Status | ✅ gh pr checks showed no failing or pending checks; standard checks passed and the QA workflow entry was skipped. |
| Functional Verification | ✅ Base failure reproduced, Dependabot skip verified, non-Dependabot validation preserved. |
Functional Verification
Test 1: Dependabot PRs without the custom template are skipped after the fix
Step 1 — Reproduce / establish baseline (without the fix):
Checked out origin/main, created a realistic pull_request_target event where pull_request.user.login was dependabot[bot] and the PR body omitted the project template, then ran:
git checkout --detach origin/main
/tmp/qa-bin/act pull_request_target -W .github/workflows/pr-description-check.yml -j validate-pr-description -e /tmp/qa-pr-description/dependabot_event.json -s GITHUB_TOKEN="$GITHUB_TOKEN" --container-architecture linux/amd64 -P ubuntu-24.04=ghcr.io/catthehacker/ubuntu:act-24.04Observed excerpt:
BASE_DEPENDABOT_RC=1
[PR Description Check/Validate PR description] ⭐ Run Main Validate HUMAN note and PR template
[PR Description Check/Validate PR description] ❗ ::error::The first visible line of the PR description must be `HUMAN:`.
[PR Description Check/Validate PR description] ❗ ::error::Keep the `AGENT:` marker from the PR template.
[PR Description Check/Validate PR description] | PR description validation failed with 7 error(s).
[PR Description Check/Validate PR description] 🏁 Job failed
This confirms the original behavior: a Dependabot-style PR description entered the validation job and failed because it did not use the custom template.
Step 2 — Apply the PR's changes:
Checked out PR commit 4fd36a416465cfaa60151e1e054f9ac48b719b4a.
Step 3 — Re-run with the fix in place:
Ran the same workflow command against the same Dependabot event:
PR_DEPENDABOT_RC=0
time="2026-06-15T02:40:20Z" level=info msg="Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock'"
The run exited successfully and produced no validation-job steps, which shows the job condition skipped the Dependabot PR before template validation.
Test 2: Non-Dependabot PR descriptions are still validated
With the PR commit checked out, I changed only the mock PR author to octocat and kept the same invalid body, then ran:
/tmp/qa-bin/act pull_request_target -W .github/workflows/pr-description-check.yml -j validate-pr-description -e /tmp/qa-pr-description/human_event.json -s GITHUB_TOKEN="$GITHUB_TOKEN" --container-architecture linux/amd64 -P ubuntu-24.04=ghcr.io/catthehacker/ubuntu:act-24.04Observed excerpt:
PR_HUMAN_RC=1
[PR Description Check/Validate PR description] ⭐ Run Main Validate HUMAN note and PR template
[PR Description Check/Validate PR description] ❗ ::error::The first visible line of the PR description must be `HUMAN:`.
[PR Description Check/Validate PR description] ❗ ::error::Keep the `AGENT:` marker from the PR template.
[PR Description Check/Validate PR description] | PR description validation failed with 7 error(s).
[PR Description Check/Validate PR description] 🏁 Job failed
This confirms the fix does not disable validation for ordinary PR authors.
Issues Found
None.
This QA review was created by an AI agent (OpenHands) on behalf of the user.
|
@enyst wow this is so much simpler! Thanks for the suggestion 🫡 |
|
Thanks @csmith49 ! |
5 participants
HUMAN:
Exclude dependabot from PR description validation. Found while reviewing #3682.
Why
Dependabot-created PRs don't follow the custom PR template format (HUMAN:, AGENT:, etc.) and shouldn't be required to. This change adds a check to skip validation entirely for PRs created by the dependabot[bot] user.
Summary
Fixes the PR description validation check failing on dependency update PRs.
How to Test
The validation script can be tested by creating a mock GitHub webhook event with dependabot[bot] as the PR author and verifying it returns 0. For manual testing, open a Dependabot PR and verify the PR description check passes.
Type
AGENT:
I verified this change by examining the existing validation logic and confirming that adding the dependabot check at the beginning of the main function will skip validation early without breaking existing behavior for other PRs.
Agent Server images for this PR
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.13-nodejs22-slimgolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:4fd36a4-pythonRun
All tags pushed for this build
About Multi-Architecture Support
4fd36a4-python) is a multi-arch manifest supporting both amd64 and arm644fd36a4-python-amd64) are also available if needed