-
Notifications
You must be signed in to change notification settings - Fork 105
RFC 877: Automated Issue Triage #880
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,125 @@ | ||
| # Automated Issue Triage | ||
|
|
||
| * **Original Author(s):**: @conroyka | ||
| * **Tracking Issue**: #877 | ||
| * **API Bar Raiser**: @mrgrain | ||
|
|
||
| > This RFC does not follow the typical RFC framework because it is not a technical design. | ||
| > I still want to work out in the open though! | ||
|
|
||
| ## Why | ||
|
|
||
| Contributing to the CDK has long been messy, tedious, convoluted, etc. Meanwhile agentic AI is | ||
| reshaping the landscape of open-source contributions as it lowers the barrier to contribute | ||
| PRs to aws-cdk. We are seeing (and encouraging) more AI-generated contributions but this comes with | ||
| significant challenges -- increasing throughput of PRs is not always a net-positive, and ignores the | ||
| bigger picture of the code lifecycle (reviewing and maintaining in perpetuity). | ||
|
|
||
| We also have 2300+ open issues. We don't actually _want_ contributions to solve all these issues; | ||
| many open issues are _unwanted_: some combination of low impact and high effort (here "effort" means | ||
| both cost to create the PR and cost to maintain the changes). The diagram below conveys what we want | ||
| to support in the pre-AI era: | ||
|
|
||
|  | ||
|
|
||
| However, some of our assumptions don't hold true anymore. | ||
| First of all, effort level triage encapsulates developer effort only (which is rapidly minimized | ||
| by agentic AI) and does not consider review and maintenance effort (which is what we | ||
| care about). Second of all, initial triaging of priority is just a best guess, so we hedge: | ||
| if you author a PR for a `p2` contribution, we wont promise a review but maybe you will get one | ||
| eventually. This is inefficient because the decision on whether an issue should be fixed at all | ||
| happens only after a PR is created. And this consideration is left subjective to the reviewer, | ||
| which causes friction when we decided an approach wasn't worth the effort in the review phase. | ||
|
|
||
| Ultimately, this proposal aims to increase efficiency: an automated triage process with a clear | ||
| set of rules provides clarity over what issues we want solved, and limits churn on community | ||
| contributed PRs -- we will have high confidence that all PRs that come in are solving an issue | ||
| that has [Demonstrated Customer Impact](#demonstrated-customer-impact). | ||
|
|
||
| ## Proposal | ||
|
|
||
| We will: | ||
|
|
||
| 1. Fully automate triaging with new labels, and an appeals process via community input. | ||
| 2. Be extremely clear on what subset of open issues welcome community contributions. | ||
| 3. Meaningfully lower the open issue count in the aws-cdk repo by closing unwanted issues. | ||
|
|
||
| ### Priority Labels | ||
|
|
||
| - `regression`: This is _actively_ worked on by a CDK maintainer at a fast pace. `regression` replaces the `p0` label exactly. | ||
| - `maintainer-response`: An issue from a lower priority tier that has been bubbled up to review. | ||
| - `maintainer-claimed`: Important issues (i.e. bugs without a workaround) that CDK Maintainers are | ||
| actively working on. After 14 days of inaction these issues will be upgraded to `maintainer-response`. | ||
| - `accepting-prs`: Bugs/features that have [Demonstrated Customer Impact](#demonstrated-customer-impact). Indicates that the CDK team | ||
| is happy to review and maintain. | ||
| - `incubating`: Bugs/features that we do not want to support without better signals from our customers. | ||
| All incoming issues with workarounds are triaged here to start. Issues that subsequently show | ||
| [Demonstrated Customer Impact](#demonstrated-customer-impact) are upgraded to `accepting-prs`. | ||
| - `unwanted`: `incubating` issues that have been human-reviewed and closed. | ||
|
|
||
| ### Effort Labels | ||
|
|
||
| `design-required`: Feature requests that were previously labelled `effort-large`. No one, not the | ||
| community, maintainers, or AI should action on these issues without an approved RFC or design artifact. | ||
| `automatable`: Issues that are straightforward and can be automated. This grouping will expand | ||
| over time. `automatable` issues will kick off a human-in-the-loop AI cycle to create a high-quality | ||
| PR with minimal lead time. At the start, `automatable` issues are _issues that do not require | ||
| testing_ -- think updates to [enums](https://github.com/aws/aws-cdk/pull/36798), [region info](https://github.com/aws/aws-cdk/issues/36880), docs, etc. | ||
| `good-first-issue`: This is an open-source standard that I would like to keep as an ad-hoc | ||
| curated list of what we think is the easiest selection of available and wanted issues. | ||
|
|
||
| ### Triage | ||
|
|
||
| Triaging is simple and well-defined, allowing for a fully automated triage process for CDK and | ||
| other AWS open-source projects that adopt this proposal. Issues can be triaged immediately and | ||
| maintainers can spend their time elsewhere. | ||
|
|
||
| Notably, our initial triage process results in only a subset of labels being used. Other labels | ||
| require additional time windows. See [Upgrades and Downgrades](#upgrades-and-downgrades) for details on | ||
| how issues get upgraded. | ||
|
|
||
|  | ||
|
|
||
| ### Upgrades and Downgrades | ||
|
|
||
| 1. All incoming issues start as `incubating`. Issues that [demonstrate customer impact](#demonstrated-customer-impact) | ||
| are upgraded to `accepting-prs`. | ||
|
|
||
| 2. A `maintainer-claimed` issue is awaiting maintainer action, a phrase which here means providing | ||
| a workaround or creating a PR. After 14 days of inaction, `maintainer-claimed` issues upgrade to | ||
| `maintainer-response`. | ||
|
Comment on lines
+88
to
+90
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does this mean community members should not submit PRs for |
||
|
|
||
| 3. `incubating` PRs are eligible to be closed as `unwanted` with a reason provided as a comment. | ||
| Closure comes with a human review but AI will assist the decision making. AI helps us be more efficient | ||
| when it comes to surfacing issues we may not want, based on subjective criteria (duplicate of previously | ||
| closed issue, feature request misidentified as a bug, not the path we want to go down). | ||
|
|
||
| ### Pull Requests | ||
|
|
||
| Pull requests will inherit the labels of the issue they address. Pull requests without a linked issue | ||
| are by default labeled `incubating`. We will have heavy-handed communication in the Contributing Guide | ||
|
Comment on lines
+99
to
+100
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For PRs that are not linked to an issue, would they be promoted to
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yes this should be the case. and i hear you that for this proposal to work, we need to make it very clear that reactions on issues are being used to determine impact in this manner. it will take some time for the community to adjust to this new weight that interacting with issues holds
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
@kaizencc I reckon the proposal should have a plan for that. We know the RFC process doesn't fit here, but for technical designs we always have a rollout plan. this seems similar. |
||
| and in the PR description template that `incubating` PRs will not be reviewed. After 28 days, | ||
| `incubating` PRs are auto-closed. Critically, `incubating` PRs are `incubating` because they have been | ||
| identified as not worth review effort and maintenance effort, so we will ignore them. | ||
|
|
||
| ### Demonstrated Customer Impact | ||
|
|
||
| An issue must demonstrate customer impact during an incubation period of 28 days to upgrade to `accepting-prs`. | ||
| To demonstrate customer impact, 5 unique individuals must react to, comment, or link the issue. This number | ||
| is a proposal but derived from data from 1000 recent issues: | ||
|
|
||
|  | ||
|
|
||
| ## Epilogue | ||
|
|
||
| There are clear expectations in this model. `maintainer-claimed` issues are claimed and actively worked on by | ||
| CDK maintainers. `accepting-prs` issues are eligible for community contributions. No one should contribute a | ||
| solution to `incubating` issues. | ||
|
|
||
|  | ||
|
|
||
| I want to be clear — these are intentionally rigid rules for triage automation to follow. Triage is inherently | ||
| a best-effort activity, and with the right guardrails is ripe for automation. Separate to the triage process | ||
| humans and additional tools should intervene where appropriate to move issues from one bucket to another. | ||
|
|
||
| The goal is to save triage time and be transparent with the community on what is a good idea to work on. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this mean that community reviews would essentially be phased out / no longer serve their role?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good point. i did not consider community reviews initially but you're right that this proposal disregards them.
in truth, i don't think community reviews has been a popular initiative besides a select few community members who are willing to spend some of their time reviewing. i want to still encourage and reward this behavior, but i'm not sure if the current community review setup is the best way to do that. @badmintoncryer if you have any ideas, i'm happy to hear them! otherwise i'll keep thinking on this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your response.
As you pointed out, there are not many reviewers who actively engage in community reviews, and I feel that we are not achieving much throughput at the moment. I believe one of the reasons for this is that the benefits and motivation for conducting reviews are difficult to feel.
If this RFC is formalized, I expect the bar for reviews to rise and the number of community-sourced PRs that get merged to decrease. In such a situation, I felt that if PRs approved by community reviewers were made eligible for review without requiring five reactions, it would make it easier for reviewers to get their desired features merged, which in turn could help boost motivation to participate in reviews.
This is just one idea, but I will also talk with community reviewers around me to see if there are any other good suggestions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
PRs approved by a community reviewer should be eligible for maintainer merge regardless of the issue's interaction count. This would serve as a practical mitigation for the strict 5-reaction threshold, reduce the maintainer review burden, and give community reviewers a stronger incentive to stay engaged.
On the motivation side, moving the community reviewer list from the wiki into a dedicated file in the repository (like CONTRIBUTORS.md) could help. Not many people check the wiki, so most contributors probably don't even know community reviewers exist. More visibility would naturally lead to more recognition and motivation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the point in reviewing code if its never going to end up in the library anyway. The fundemental issue is not how its labelled.